diff --git a/README.md b/README.md index ac022db..b47c5fa 100644 --- a/README.md +++ b/README.md @@ -4,322 +4,38 @@ Generate markdown documentation from directory structure. Optimized for LLM inge -## Features +## Install -- Recursive directory traversal -- ASCII tree visualization (LLM-friendly, minimal token overhead) -- Full tree in every generated file — navigate from anywhere, no matter how deep you are -- Filtered files and directories stay visible in the tree with annotations (`~ ignored`, `~ tests omitted`) -- Flags provenance block: every generated file records which filters were active -- Full file contents with explicit delimiters and metadata -- Filters dotfiles, binaries, and blacklisted extensions -- Vertical slices mode: index file + root slice + one markdown per top-level directory -- Pick mode: include specific individual files alongside the slices -- Append mode for single files or directories -- Force overwrite of existing output files -- Configurable depth, file size limits, ignore patterns -- Optional upload to Proton Drive after generating output -- Skip frontend files (`.html`, `.css`, `.js`, etc.) -- Frontend only mode (include only frontend files and `README.md`) -- Omit test files and inline test blocks (Go, Java, Rust) + go install git.flo.fo/FLO/dirmd@latest -## Installation -Only for linux. -Download binary from release page or install with go -```bash -go install git.flo.fo/FLO/dirmd@latest -``` +Linux only. Run `dirmd -h` for all flags. -## Use with Git Post Hook -combine with git post hook to sync with proton drive on each commit +## Why -add something like this in ./.git/hooks/post-commit +LLMs don't need 40k tokens of preamble. dirmd dumps a repo as markdown: tree + file contents, minus the noise. Everything filtered out stays visible in the tree (`~ tests omitted`, `~ ignored`) so nothing vanishes silently — the model knows what exists and can ask for it. -```bash -#!/bin/bash +## Three ways to use it -set -euo pipefail +**One file — small repos.** +For small projects that fit in a context window comfortably: -if ! command -v dirmd &>/dev/null; then - exit 0 -fi + dirmd -o out.md ~/repos/myproject -REPO_ROOT=$(git rev-parse --show-toplevel) -REPO_NAME=$(basename "$REPO_ROOT") +Everything lands in a single markdown file. Simplest, works fine until it doesn't. -dirmd -o "/tmp/${REPO_NAME}" "$REPO_ROOT" -f --vertical-slices --proton-drive -``` +**Vertical slices — large repos.** +Once a single file gets too big to paste repeatedly, split it: + dirmd -o docs ~/repos/myproject --vertical-slices -## Build +This produces an index file (metadata + full tree + slice table) and one file per top-level directory. The full tree exists only in the index; every other file carries a one-line breadcrumb pointing back to it. Workflow: paste the index, then paste only the slice you need. - go build -o dirmd . +**Slices + picks — you already know which files matter.** +Sometimes you need one specific file without its whole directory: -## Usage - - # Create new documentation - dirmd -o output.md ~/repos/myproject - - # Overwrite existing output file - dirmd -o output.md ~/repos/myproject -f - - # Append a single file - dirmd -a -i README.md -o existing.md - - # Append another directory - dirmd -a ~/repos/anotherproject -o existing.md - - # Generate with vertical slices - dirmd -o output.md ~/repos/myproject --vertical-slices - - # Skip frontend files (.html, .css, .js, etc.) - dirmd -o output.md ~/repos/myproject --skip-frontend - - # Frontend only (include only frontend files and README.md) - dirmd -o output.md ~/repos/myproject --frontend-only - - # Omit test files and inline test blocks - # Go: excludes *_test.go - # Java: excludes *Test.java, *Tests.java, src/test/java/* - # Rust: strips #[cfg(test)] mod tests { ... } blocks - dirmd -o output.md ~/repos/myproject --omit-test - - # Combine filters - dirmd -o output.md ~/repos/myproject --frontend-only --omit-test - - # Pick specific files alongside vertical slices - dirmd -o output.md ~/repos/myproject --vertical-slices \ + dirmd -o docs ~/repos/myproject --vertical-slices \ --pick common/src/config.rs --pick init/src/main.rs - # Same, with a compact breadcrumb instead of the full tree in pick files - dirmd -o output.md ~/repos/myproject --vertical-slices \ - --pick common/src/config.rs --small-tree +Picks are individual files written alongside the slices, in the same directory layout (`common/config.rs.md` next to `common/common_myproject.md`). Use this when you're mid-conversation, already know the relevant files, and want a single consistent snapshot containing exactly those. - # Generate and upload to Proton Drive (default: /my-files/md/) - dirmd -o output.md ~/repos/myproject --proton-drive - - # Generate and upload to a specific remote path - dirmd -o output.md ~/repos/myproject --proton-drive --drive-path /my-files/docs/api.md - -## Test File Omission - -The `--omit-test` flag removes test files and inline test blocks to reduce token overhead in LLM context windows. Behavior varies by language: - -| Language | Exclusion Strategy | What Gets Removed | -|----------|-------------------|-------------------| -| **Go** | File exclusion | All `*_test.go` files | -| **Java** | File exclusion | `*Test.java`, `*Tests.java`, files under `src/test/java/` | -| **Rust** | Content stripping | Entire `#[cfg(test)] mod tests { ... }` blocks removed from `.rs` files | - -The Rust stripping works by detecting the `#[cfg(test)]` attribute, then tracking brace depth until the module closes. Nested braces in string literals may occasionally throw off the count, but this is acceptable for RAG use cases. - -Omitted test files and directories remain visible in the directory tree with a `~ tests omitted (--omit-test)` annotation, so consumers of the snapshot know they exist and can request them. - -## Frontend Filtering - -Two complementary flags control frontend file inclusion: - -| Flag | Effect | -|------|--------| -| `--skip-frontend` | Exclude frontend files, keep everything else | -| `--frontend-only` | Include only frontend files and `README.md`, exclude everything else | - -Frontend extensions covered: - -| Type | Extensions | -|------|------------| -| Templates | `.html`, `.htm`, `.gohtml`, `.tmpl` | -| Stylesheets | `.css`, `.scss`, `.sass`, `.less` | -| JavaScript | `.js`, `.jsx`, `.mjs`, `.cjs` | -| TypeScript | `.ts`, `.tsx` | -| Frameworks | `.vue`, `.svelte` | - -`--skip-frontend` and `--frontend-only` are mutually exclusive. Both can be combined with `--omit-test`. - -## Vertical Slices Mode - -When `--vertical-slices` is passed, dirmd splits output into multiple files instead of one monolithic document: - -- **Index file** (`index_.md`): timestamp, commit, `.dirmd` instructions (if `--instructions`), flags block, full directory tree with annotations and cross-references to slice files, and the vertical slices index table. Deliberately thin — no file contents. This is the always-load document. -- **Root slice** (`root_.md`): contents of all root-level files (README, Cargo.toml, justfile, etc.). Selectable like any other slice. -- **Slice files** (`/_.md`): one per top-level directory, placed in a subdirectory named after it. Each carries the full project tree (not just its subtree), a back-reference to the root document, the flags block, and full recursive file contents for that directory. - -Every generated file carries the **full project tree**, so navigation works no matter which file is loaded. Directories excluded via `--ignore` appear as a single collapsed node annotated `~ ignored (--ignore)` — never recursed into — and files filtered by `--omit-test`, `--omit-md`, or frontend flags appear annotated instead of vanishing. The tree tells the consumer what exists but was left out. - -This mode cannot be combined with `--append` or `--input-file`. - -Example output with `dirmd -o docs ~/repos/myproject --vertical-slices`: - - docs/ - ├── index_myproject.md ← index: metadata + tree + slices table - ├── root_myproject.md ← root files (README, etc.) - ├── cmd/ - │ ├── cmd_myproject.md ← slice: full tree + contents of cmd/ - │ └── main.rs.md ← pick file (only if --pick was used) - └── internal/ - └── internal_myproject.md - -## Pick Mode - -`--pick ` selects individual files for inclusion in a sliced run. Repeatable. Requires `--vertical-slices`. - -- Picks are written to `//.md` — e.g. `--pick common/src/config.rs` produces `common/config.rs.md` beside the `common` slice. Root-level picks land directly in the output directory. -- Every pick run regenerates the full snapshot (slices + picks); a pick run costs one full generation. This keeps the output directory a consistent snapshot of one run. -- Pick files carry the same metadata block (timestamp, commit, flags) as slices, plus the full project tree by default. With `--small-tree`, the tree is replaced by a breadcrumb line (`Tree: myproject / common / src / config.rs`). -- Pick files never carry `.dirmd` instructions — those live in the index only. -- Pick paths must be files, must exist, and must not be excluded by active filters (`--ignore`, `--omit-test`, extension skips). Violations fail the run. -- Picks are listed in a dedicated `## Picks` section in the index file. - -## Instructions File (`.dirmd`) - -If a file named `.dirmd` exists in the root of the input directory, you can include its contents as metadata in the generated output by passing the `--instructions` flag. - -This is useful for injecting specific context, constraints, or instructions for LLMs processing the snapshot. - -- **File Name**: `.dirmd` -- **Location**: Root of the input directory. -- **Format**: Plain text or Markdown. -- **Output**: Rendered as a quoted block (`> Instructions: ...`) immediately after the timestamp and commit info. -- **Opt-in**: The file is **ignored** unless `--instructions` is explicitly passed. -- **Scoping**: In vertical slices mode, instructions appear **only in the index file** — never in slice or pick files. One authoritative copy. - -Example `.dirmd`: - -```text -Focus on the `internal/` package for business logic. -Ignore `cmd/` unless explicitly asked. -Prioritize Go files over Python. -``` - -## LLM-Optimized Output - -Output is designed for RAG ingestion and LLM context windows, not human presentation: - -- **ISO 8601 Timestamp**: Added as a metadata line (`> Generated: ...`) immediately after the H1 header for every new file created. -- **Git Commit Info**: If the source directory is a Git repository, a second metadata line (`> Commit: ()`) is added below the timestamp. -- **Flags provenance block**: `> Flags: --omit-test --skip-frontend` plus lines for extra ignore patterns and skipped extensions. Appears in every generated file so any file in isolation reveals how the snapshot was produced and what to assume exists but was excluded. -- **Annotated full tree**: same complete tree in every file, with filtered entries annotated (`~ ignored`, `~ tests omitted`) instead of silently dropped -- **ASCII tree format** (`+-` instead of Unicode box-drawing) to reduce token overhead -- **Explicit file delimiters**: `--- FILE: ./path (bytes, lines) ---` and `--- END FILE ---` -- **Document type markers**: `[ROOT]` and `[SLICE]` in H1 headers -- **Cross-references**: Directory entries in the tree show `-> see cmd/cmd_myproject.md`; slice and pick files reference their root document -- **Slices index table**: File counts and total bytes per slice for informed retrieval decisions -- **Tilde fences** (`~~~`) to avoid conflicts with backticks in source files - -## Flags - -| Flag | Description | Default | -|------|-------------|---------| -| `-o`, `--output` | Output markdown file path (required) | | -| `-a`, `--append` | Append to existing output file | false | -| `-f`, `--force` | Overwrite existing output file (non-append mode) | false | -| `-i`, `--input-file` | Single file input (append mode only) | | -| `--vertical-slices` | Split output into index + root slice + one markdown per top-level directory | false | -| `--pick` | Relative path to a file to include as pick (repeatable, requires `--vertical-slices`) | | -| `--small-tree` | Use breadcrumb instead of full tree in pick files | false | -| `--instructions` | Include `.dirmd` instructions file if present in root (index file only) | false | -| `--max-size` | Max file size in bytes | 524288 | -| `--max-depth` | Max directory recursion depth | 20 | -| `--ignore` | Additional ignore patterns (glob) | | -| `--extensions` | Additional file extensions to skip | | -| `--proton-drive` | Upload output to Proton Drive after writing locally | false | -| `--drive-path` | Full remote path on Proton Drive (e.g. /my-files/md/report.md) | /my-files/md/ | -| `--skip-frontend` | Omit frontend file types (.html, .css, .js, .ts, .vue, .svelte, etc.) | false | -| `--frontend-only` | Include only frontend file types and README.md | false | -| `--omit-test` | Exclude test files (Go *_test.go, Java *Test.java) and strip #[cfg(test)] blocks from Rust | false | -| `--omit-md` | Omit all .md files except README.md | false | - -## Proton Drive Integration - -dirmd can optionally upload the generated markdown file(s) to Proton Drive using the [Proton Drive CLI](https://proton.me/support/drive-cli). The upload is performed only when `--proton-drive` is passed. Without it, dirmd behaves exactly as before — no Proton Drive dependency is required. - -In vertical slices mode, the entire output directory (index + root slice + slices + picks) is uploaded to `/my-files/md/`. The `--drive-path` flag is ignored in slices mode ; each file is uploaded by its basename. - -### Prerequisites - -1. Install the Proton Drive CLI — see the [official guide](https://proton.me/support/drive-cli) for download and installation instructions. -2. Authenticate by running `proton-drive auth login`. -3. Ensure `proton-drive` is in your PATH. - -If the CLI is not installed or you are not logged in, dirmd will report the error to stderr and exit with a non-zero code. Local files are always written successfully before the upload is attempted. - -For full documentation on the Proton Drive CLI, including installation, authentication, and troubleshooting, refer to the [official Proton Drive CLI support page](https://proton.me/support/drive-cli). - -## Examples - - # Custom depth and size limit - dirmd -o output.md ~/repos/project --max-depth 5 --max-size 262144 - - # Ignore specific directories - dirmd -o output.md ~/repos/project --ignore __pycache__ --ignore .venv - - # Skip additional file extensions - dirmd -o output.md ~/repos/project --extensions .log --extensions .tmp - - # Vertical slices with overwrite - dirmd -o docs ~/repos/project -f --vertical-slices - - # Skip frontend and omit tests - dirmd -o docs ~/repos/project -f --skip-frontend --omit-test - - # Frontend only with test omission - dirmd -o docs ~/repos/project -f --frontend-only --omit-test - - # Full snapshot workflow with picks - dirmd -o docs ~/repos/project -f --vertical-slices --instructions \ - --omit-test --skip-frontend \ - --pick common/src/config.rs --pick mssql-extractor/src/processor.rs - - # Generate, then upload to default remote location - dirmd -o output.md ~/repos/project --proton-drive - - # Generate, then upload to a custom remote location - dirmd -o output.md ~/repos/project --proton-drive --drive-path /my-files/projects/docs.md - - # Overwrite existing local file and upload to Drive - dirmd -o output.md ~/repos/project -f --proton-drive - -## Output Format - -Produces markdown with: - -1. Absolute path as H1 header with `[ROOT]` or `[SLICE]` marker -2. **Timestamp metadata line** (ISO 8601) immediately after the H1 header (only for new files) -3. **Flags provenance block** (`> Flags: ...`, `> Ignored: ...`, `> Skipped extensions: ...`) when non-default filters are active -4. ASCII directory tree (indented with `+-` and `|` characters), full project tree with annotations for filtered entries, identical in every file of a run -5. Vertical slices index table and picks section (sliced mode only) -6. All text files with content in fenced code blocks -7. Explicit `--- FILE: ---` and `--- END FILE ---` delimiters with byte and line counts -8. Language detection from file extensions - -Example output structure (non-sliced mode): - - # /home/user/repos/myproject [ROOT] - > Generated: 2026-09-02T14:30:00Z - > Commit: a1b2c3d (Fix login bug) - > Flags: --omit-test - - ## Directory Tree - - ~~~ - myproject/ - +- cmd/ - | +- main.go - +- internal/ - | +- config.go - | +- config_test.go ~ tests omitted (--omit-test) - +- target/ ~ ignored (--ignore) - ~~~ - Legend: ~ ignored | ~ tests omitted | ~ omitted | ~ exceeded --max-depth - - ## Contents - - --- FILE: ./cmd/main.go (45 bytes, 3 lines) --- - ~~~go - package main - - func main() {} - ~~~ - --- END FILE --- - -Note: Uses tilde fences (~~~) internally to avoid conflicts with nested backticks in source files. +Rule of thumb: one file for tiny repos, slices when you need to browse, picks when you already know the answer.