feat(fs): auto-compress cat'd markdown via compress_md pipeline#88
Merged
claudioemmanuel merged 1 commit intomainfrom Apr 22, 2026
Merged
feat(fs): auto-compress cat'd markdown via compress_md pipeline#88claudioemmanuel merged 1 commit intomainfrom
claudioemmanuel merged 1 commit intomainfrom
Conversation
When auto_compress_md=true (default), viewer commands (cat/head/tail/less/more/bat) targeting .md files now route through the markdown compressor instead of the generic smart_filter+truncation path. Emits a [squeez: md-mode — compressed N→M tokens from path] marker header. Falls back to the original output if the integrity check fails. Adds bench/fixtures/cat_claude_md.txt (486 lines, 98% token reduction in bench/run.sh).
aebc461 to
380d16a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
extract_md_viewer_target()helper insrc/commands/fs.rsthat detects viewer commands (cat/head/tail/less/more/bat) targeting.mdfilesFsHandler::compressroutes.mdfiles throughcompress_md::compress_text_with_localewhenauto_compress_md=true(the default), bypassing smart_filter/dedup/truncation entirely[squeez: md-mode — compressed N→M tokens from path]header; falls back to raw text if integrity check failstests/test_fs.rscovering the happy path, disabled flag, and non-.md filesbench/fixtures/cat_claude_md.txt(486 lines) picked up bybench/run.shMotivation
Large plan files like
plan.mdat 1,754 lines were being read repeatedly with no compression — the existingFsHandleronly head-truncated to 50 lines. The markdown compressor was already proven on memory files but unused for ad-hoccatinvocations. This change closes that gap with zero new dependencies.Benchmark impact
98% token reduction on a 486-line representative CLAUDE.md fixture.
Test plan
cargo test --test test_fs— 5 tests pass (2 existing + 3 new)cargo test— full suite greenbash bench/run.sh— new fixture shows 98% reductioncargo build— zero warnings