Skip to content

fix(core,cli): enforce internal invariants in release builds (degrade non-boundary spans, assert skeleton/width/range/cursor invariants); path helpers fail closed on strip_prefix and non-UTF-8 paths; fmt non-UTF-8 exits 2 (#220, #217) - #388

Merged
dean0x merged 11 commits into
mainfrom
fix/c5-release-invariants-fail-closed-paths
Sep 15, 2026

Conversation

@dean0x

@dean0x dean0x commented Sep 15, 2026

Copy link
Copy Markdown
Owner

Summary

Two independent hardening passes at the same boundary — "what does this code do in a release build when an internal assumption is violated?" — plus one exit-code correction.

  1. Release-enforced invariants (debug_assert!-only invariants silently degrade in release builds (3 sites) — PF-005 #220). Ten debug_assert!/debug_assert_eq! sites were audited. Two are now degradations (a non-boundary import/@extends span offset yields a zero-length span instead of slicing source raw), and eight are now unconditional assert!/assert_eq! — the skeleton-block splice, the neutralize_source_for_render byte-length check, the three pub-API range checks, and the three evaluator source-map cursor checks. None of the ten can be tripped by template input; each is a compiler defect. The remaining 30 genuine debug_assert! invocations were each examined and deliberately kept.
  2. Fail-closed path helpers (output.rs path helpers fail-open on strip_prefix edge cases and non-UTF-8 paths #217). Four host-path surfaces that failed open now fail closed: mds lint <dir>'s wire/sort key, the two source-map sources[] anchors, and two degenerate output-path joins. Plus a new warning when mds build/mds watch writes a source flat instead of mirrored.
  3. mds fmt <file> on a non-UTF-8 path exits 2 (output.rs path helpers fail-open on strip_prefix edge cases and non-UTF-8 paths #217), not 1 — mds::io, matching mds lint and mds build. Directory mode is unchanged.

security/snyk (dean0x) is SCA-only for this PR — the Snyk MCP server failed to connect locally (ENOENT on a stale snyk-macos-arm64 npx cache path), so no snyk_code_scan ran; Rust is not scanned by Snyk Code here in any case.

What was wrong

  • Sites 1 and 2 did not "silently degrade" in release — they panicked. This is the measured refutation of the issue's premise. attach_import_span (resolver.rs) and check_child_only_blocks (resolver/inheritance.rs) each had a debug_assert! that the offset was a char boundary, followed one line below by a raw slice of source. In a release build the assert compiles away and the slice runs. The RED release run produced, verbatim:

    resolver.rs:2518  start byte index 1 is not a char boundary; it is inside 'é' (bytes 0..2 of string)
    resolver.rs:2518  start byte index 1 is out of bounds for string of length 0
    inheritance.rs:83 (the same two, same two inputs)
    

    So the pre-existing release behaviour was a panic with a message naming a byte of the user's source, not a degraded span.

  • Site 3 genuinely was a silent fallback, and the worst of the ten. The skeleton-splice walk had debug_assert!(false, …) in the "no effective-blocks entry for this @block" arm and then spliced the base default. In release that silently drops a child template's @block override and emits the parent's content — wrong output, no diagnostic, no exit code.

  • Three more sites were promoted because their release fallback was also unsound rather than merely cosmetic: the neutralize_source_for_render byte-length check (a width mismatch desynchronises every following span offset, so miette renders carets against the wrong columns), and the three pub-API range constructors (TextEdit::new, FixLineSpan::range_inclusive, range_exclusive), where a release caller passing a reversed range got a value the fix planner later skipped in silence.

  • relative_display failed open on both the wire key and the sort key. mds lint <dir> built each files[].file value with a strip_prefix(root).unwrap_or(path) plus to_string_lossy. An entry not under the lint root emitted an absolute key; an entry with a non-UTF-8 component emitted a lossy key (U+FFFD). The same string is the sort key, so the directory ordering was affected too.

  • Empty-root containment was vacuous. relativize_source's containment test is starts_with_comps(source_comps, root_comps). path_to_unified returned String, so an empty or unusable root became an empty component list — and starts_with_comps(x, []) is vacuously true for every x. Result: every source-map sources[] entry would be treated as "inside the project" and emitted as a host-absolute path minus its leading /.

  • A non-UTF-8 root became a lossy anchor. NativeFs::source_root() used display().to_string(), so a root directory that is not valid UTF-8 produced a string ending in U+FFFD and was then compared, byte-wise, against real path components. It never matched, but it was still being used as an anchor.

  • Two stem-less joins could escape the out-dir. Both Dir(_)-mode output oracles fell back to source.as_os_str() when file_stem() was None. d.join(<absolute>) re-roots: output_base_no_ext(Path::new("/"), "/root", Dir("/out")) returned "/", and output_path_for then built "/.md" — outside the out-dir entirely.

  • The watch ghost-prune collided on file name. Pruning a vanished out-of-root dependency called the full DirWatchState::forget, which probes and removes the output path. For an out-of-root ghost the probe resolved through the flatten arm to <out>/<name>.md — the same path an in-root source with that file name owns — so the prune dropped the in-root source's write-dedup entry.

  • mds fmt exited 1 on an undecodable path where mds lint and mds build both exited 2, and the fmt module doc already claimed 2.

Design

  • Degrade at sites 1 and 2; do not assert. Both compute an underline length for a diagnostic that is already being constructed — the import failure or the stray-node error is the real payload, and the span is cosmetic. Panicking would replace a useful error with a crash, which is strictly worse for the user than a zero-length span. An in-tree helper, line_len_at, already had exactly the char-boundary-safe semantics (it backs build_type_mismatch), so both sites now route through it. One degradation rule, one helper, one test group — rather than a second hand-rolled boundary check at each site.
  • A real predicate at site 3, not assert!(false). Site 3 has no safe fallback value: splicing the base default is wrong output. The promotion is an assertion on the map lookup itself, so the check reads as the invariant it enforces rather than as an unreachable marker in an else-branch.
  • No user data in any of the eight messages. The CLI installs no panic hook, so a tripped invariant prints Rust's default panic message straight to stderr — bypassing the WIRE sanitizers (safe_path, safe_inline) that every other CLI output path is forced through by print_discipline. Interpolating source text or a path into an assert message would route unsanitized bytes around that guard. All eight messages therefore carry static text and integers only.
  • Option, not a sentinel, for source_root / path_to_unified. The bug being closed is a sentinel bug: the empty string is a valid prefix of every path, so "" as "no anchor" makes containment vacuously true. Option<String> makes "not an anchor" unrepresentable as a prefix — the choke-point must decide what None means at step 6b, and the compiler forces both branches to be written. An unusable root means "not contained" (basename); an unusable base means "anchor on the root".
  • The flatten warning lives in the write oracle only. output_base_no_ext is a probe — watch calls it many times per batch to compute candidate paths for stale-output cleanup and dedup bookkeeping, for paths that may never be written; reporting there would emit the warning repeatedly for one logical event. output_path_for is called once per actual write. T-P16 pins both halves (present in the write oracle, absent in the probe oracle), so neither half is vacuous — and P-M7/P-M7b confirmed each half fails independently.
  • The lint pre-pass is not ?. run_lint_directory computes every display path into a Vec<(PathBuf, String)> before sorting, and on the first Err routes it through emit_analysis_failure_json_or_stderr and exits — rather than propagating. Two reasons: the emit-before-exit ordering rule means a bare ? would let the caller's blanket catch exit 2 without writing the JSON envelope; and failing before any file is linted is what keeps a partial files[] array from being emitted alongside the error.
  • Root / is deliberately unchanged. path_to_unified("/") is Some("/"), which normalizes to an empty component list legitimately/ really is a root and every absolute path really is under it. The fail-closed rule therefore keys on "could not produce an anchor" (None), never on "the component list is empty". Collapsing those two would change behaviour for a real, reachable root.
  • error.rs's at() canary stays debug-only, and that is load-bearing. It is the mechanism that lets sites 1 and 2 degrade: the tests expect the debug run to panic with MdsError::at(): cross-source offset mismatch and the release run to return span: Some((1,0)), src: None. Promoting it would make the release branch panic — undoing this PR's degradation and re-creating the debug_assert!-only invariants silently degrade in release builds (3 sites) — PF-005 #220 failure one layer up. Recorded as M11 (reasoned, deliberately not performed).

Behaviour changes

# Change Tests
(a) TextEdit::new, FixLineSpan::range_inclusive, FixLineSpan::range_exclusive panic on a reversed range in release as well as debug. Public API; rustdoc # Panics added. No in-tree caller constructs one. T-E1..E4
(b) A @block placeholder with no effective-blocks entry now panics in release instead of splicing the base default. T-C1
(c) A neutralize_source_for_render byte-width mismatch now panics in release instead of desynchronising span offsets. T-D1
(d) The three evaluator source-map cursor checks are enforced in release. T-F1
On the CLI (b)/(c)/(d) surface as a Rust panic, exit code 101. The napi, WASM and Python bindings catch it at the FFI boundary and convert it to mds::internal, unchanged.
(e) mds lint <dir> with an unnameable entry fails the whole run before any file is linted (was: per-file error with a lossy/absolute key, rest of the tree still linted). Exit code is unchanged at 2; the message text is unchanged. T-P4
(f) mds fmt <file> on a non-UTF-8 path exits 2 (mds::io), was 1. Directory mode unchanged (exit 1 via the N failed tally). T-P6
(g) mds build/mds watch --out-dir prints a warning when a source is written flat instead of mirrored. Not gated on --quiet. No walked source can reach it — it is a tripwire, not user-facing advice. T-P16
(h) Degenerate stem-less sources now produce <out>/output.<ext> instead of / and /.md. Unreachable from the CLI; a real change to the pure functions. T-P15
(i) mds watch no longer probes output paths when pruning a vanished out-of-root dependency, so it can no longer drop an in-root source's write-dedup entry. T-P19
(j) Source-map anchors: zero change for every UTF-8 root. Not reachable from the CLI or the public API today (non-UTF-8 entry paths are rejected before compilation) — closed as a latent hazard. T-P20; goldens 304/304, Python 250

Nothing moved in any golden. @mdscript/mds's source maps (U-SM), source maps — WASM backend (W-SM) and source maps — compileFile differential (CF-SM) suites are green against a freshly rebuilt native addon and a freshly rebuilt crates/mds-wasm/pkg — the addon must be rebuilt first or the harness loads a stale mds-napi.node and silently exercises the old binary. Python parity: pytest -m "not perf" → 250 passed. WASM is structurally unaffected in any case: VirtualFs::source_root is the trait default None, so the WASM path never reaches step 6b, and an empty base is unreachable from the CLI because compute_source_map_base absolutizes on every branch.

Evidence

Commits — RED → GREEN per pair

# SHA Scope RED observation
1 ad23019 #220 RED: 15 tests dev 6 passed / 9 failed; release --list 14, 6 passed / 8 failed
2 9e23488 #220 fix dev 15/15; release 14/14
3 12b3408 #217 lint RED T-P1 both arms returned Ok of an out-of-root key; T-P2 returned Ok of a U+FFFD key
4 dd54b82 #217 lint fix both Err; controls Ok
5 7e9a5fe #217 anchors RED T-P7/T-P8/T-P11 leaked the host path minus its leading /; T-P12 returned Some(…U+FFFD)
6 d73cf2f #217 anchors fix basename / None; controls unchanged
7 a10a7bc #217 CLI RED T-P15 output_base_no_ext("/", "/root", Dir("/out"))"/"; T-P16 no report in output_path_for; T-P19 last_written lost the in-root entry
8 0be5e56 #217 CLI fix /out/output, report present, in-root entry kept
9 aae35a9 #217 fmt RED forced-branch run: EXIT=1
10 f4707d7 #217 fmt fix forced-branch run: mds::io, EXIT=2; directory mode still EXIT=1
11 e1a69d0 docs (this commit)

Note on 9/10: macOS APFS rejects touch $'\xff\xfe.mds' with Illegal byte sequence (EILSEQ) — the kernel enforces valid UTF-8 in filenames; Linux ext4 does not. The on-disk hostile arms of T-P4, T-P5, T-P6 and T-P13 therefore return early here and first execute on Linux CI. Local evidence for those arms is the forced-branch technique: the success path of the decode was temporarily forced to None, the real binary rebuilt, run end to end, and reverted. Each test's control arm runs unconditionally on every unix platform, and on any non-macOS unix a failed hostile create panic!s rather than skipping, so a filesystem that should accept the name can never masquerade as a skip. No #[ignore], no A || B assertions.

G1 — release-profile verification (not run by CI)

ci.yml runs cargo test --workspace in the dev profile only, so nothing in CI would have caught any of the eight promotions or the four degradations. G1 is a local gate:

cargo nextest run -p mds-core --cargo-profile release --lib -E 'test(/…/)'

Plain form; no CARGO_PROFILE_RELEASE_LTO=false override and no [profile.release-test] were needed. At this commit: 14 listed, 14 passed, 1 154 skipped, 29.35 s wall (list + release compile + run).

G2 — evaluator cost gate (the three cursor asserts fire per output node)

Fixture big.mds: 180 000 × Hello {{name}} \{{ tail text = 5 220 020 B → 4 500 020 B output, 3 600 083 B map. /usr/bin/time -p <bin> build big.mds --source-map -o out.md, release binaries built at ad23019 (production code then byte-identical to main) vs 9e23488.

run before (s) after (s)
1 0.24 0.69 †
2 0.21 0.21
3 0.22 0.21
4 0.21 0.22
5 0.21 0.21
median 0.21 0.21

† first touch of a freshly copied binary; does not recur.

Round 2 (independent repeat), wall/user: before 0.21/0.16 ×5, after 0.21/0.16 ×5 — medians identical. Noise control ×3 each without --source-map (the asserts are unreachable): median 0.17 both. Higher-resolution supplement (320 000 lines, 9 280 020 B, 960 000 segments — just under MAX_SOURCEMAP_SEGMENTS = 1 000 000): median wall 0.290.30 (ratio 1.034), median user 0.220.22.

Threshold median wall(after) ≤ 1.05 × median wall(before)0.21 ≤ 0.2205. PASS on two independent rounds (ratio 1.000). No revert; the three evaluator assert_eq!s stay.

G3 — WASM size

Measured with CI's own commands (wasm-pack build crates/mds-wasm --target nodejs --out-dir pkg / --target web --out-dir pkg-web) — not npm run build -w @mdscript/mds-wasm, which writes packages/mds-wasm/dist/{node,web} and leaves crates/mds-wasm/pkg stale.

tree pkg/mds_wasm_bg.wasm pkg-web/mds_wasm_bg.wasm
main production code (33380d7) 850 313 850 313
after #220 (phase 1) 850 971 850 971
this branch (e1a69d0) 851 728 851 728
delta vs main +1 415 (+0.166 %) +1 415 (+0.166 %)

The #220 work accounts for +658 B and the #217 mds-core changes (source_path.rs's Option return and step 6b, fs.rs's source_root) for the remaining +757 B. The #217 CLI work contributes nothing — mds-cli is not compiled into the WASM artifact.

Local uses wasm-pack's cached wasm-opt v117, not CI's pinned Binaryen v129, so only the delta transfers; the CI byte count is the authoritative one. Local sits 6 865 B above the CI-measured 843 448 at 33380d7 — inside the known 3–7 KB local/CI offset. Projected CI ≈ 844 863 B, ≈ 5 137 B under the 850 000 guard. .github/workflows/ci.yml and the guard were not touched.

Flake screen

cargo nextest run -p mds-cli --test cli_watch --no-fail-fast, three sequential runs on the green tree: 80/80 each time (4.536 s / 4.440 s / 5.069 s). Zero failures, zero skips, zero leaky. No timeout, sleep, debounce, serialisation or ordering change was made anywhere in watch.rs.

Mutation testing — #220 (M1–M11)

id mutation observed
M1 restore debug_assert! + raw slice at site 1 T-A1, T-A2, T-L1 FAIL; T-A3 passes
M2 same at site 2 T-B1, T-B2, T-L1 FAIL; T-B3 passes
M3 degrade inline instead of via line_len_at at site 1 T-A1/A2/A3 all PASS — behaviour identical; only T-L1 catches it. WEAK, purely lexical — renaming the helper would break it with no behaviour change
M4 site 3 back to debug_assert!(false) + fallback T-C1 + T-L1 FAIL; T-C2 passes
M5 TextEdit::new message interpolates ({start} > {end}) T-E1 still PASSES (the expected string is a substring); only T-L1 catches it
M6 neutralize back to debug_assert_eq! T-D1 and the bidi test both still PASS (parity really does hold); T-L1 only. WEAK
M7 TextEdit::new back to debug_assert! dev: T-E1 PASSES — a demotion is invisible to a dev-only CI; release: T-E1 FAILS; T-L1 FAILS in both. The single best argument for keeping T-L1
M8 EscapedBrace cursor assert back to debug_assert_eq! T-F1 still PASSES; T-L1 only. WEAK
M9 reword the site-3 message T-C1 FAILS (substring) + T-L1 FAILS
M10 delete the #220 justification comment above the neutralize assert T-L1 only
M11 (reasoned, not performed) promote error.rs:175 would make T-A1/T-B1's release branch panic — i.e. undo the degradation. Deliberately not done

Three mutations (M3, M6, M8) are caught ONLY by the lexical shape test T-L1 (crates/mds-core/tests/assert_promotions.rs). That test is load-bearing, not a style guard: with CI running dev-only, it is the sole detection for a silent assert!debug_assert! demotion (M7).

Mutation testing — #217 (P-M1 … P-M12)

id mutation observed
P-M1 strip_prefix(root).unwrap_or(path) Survived the first attempt. With only the absolute arm, /other/x.mds is still rejected — by the Component::Normal guard, not by the strip. A relative out-of-root arm was added to the RED commit; the mutant then dies
P-M2 to_string_lossy reintroduced T-P2 fails, reporting a two-U+FFFD key
P-M3 pre-pass → per-file lossy key Not observable on macOS; the forced-error run is the local stand-in (envelope + exit code + absent summary line)
P-M4 path_to_unified(root).unwrap_or_default() T-P7, T-P8, T-P11 all fail with the host path minus its leading /; T-P9/T-P10 correctly stay green
P-M5 unusable base → empty component list instead of None T-P9 stays GREEN, as predicted. Behaviour-preserving for the tested paths — step 9 already asks whether the base is inside the root and an empty list fails that length check. Documented rather than pinned with an implementation-detail test
P-M6 source_root back to display().to_string() T-P12 fails with Some(…U+FFFD)
P-M7 report moved from output_path_for into output_base_no_ext T-P16 fails on the positive half
P-M7b report present in both oracles T-P16 fails on the negative half — so neither half is vacuous
P-M8 safe_path dropped, value hoisted into a local print_discipline fails: output.rs:334: eprint_warning(format!) interpolates unsanitized
P-M9 Err-arm unwrap_or(source.as_os_str()) reintroduced T-P15 fails, left "/" right "/out/output"
P-M10 watch gate forced to if true T-P19 fails, last_written reduced to the out-of-root key
P-M11 fmt back to miette::miette! EXIT=1 (the pre-fix forced-branch reading)
P-M12 warning gated on !quiet NOT PERFORMED — named weakness. T-P16 pins the report's lexical presence inside fn output_path_for(, so an if !quiet { … } wrapper would still pass. output_path_for has no quiet in scope, so the mutation is unreachable without a signature change (out of scope) — but the test's weakness is real and recorded rather than papered over

Every mutation above except P-M12 and M11 was applied, observed and reverted. Post-revert greps for if true {, src_display, filter(|_| false) and TEMP-OBSERVE returned zero hits before each commit.

The 30 debug_assert! sites that deliberately stay

A naive git grep -nE 'debug_assert(_eq|_ne)?!' over crates/*/src returns 71 lines / 60 macro-looking hits, but only 40 are real invocations — the other 20 are prose in doc and inline comments that mention the macro (several of them explaining why a debug_assert! was deliberately not added there). The 40 real invocations are exactly 2 degraded + 8 promoted + 30 kept; 0 were unlisted. 39 are production, 1 (lint/fix.rs:1206) is inside a #[cfg(test)] mod tests and never compiles into a release build at all.

site invariant why it stays
mds-cli/lint.rs:1488 clean+warn+error+limit counts sum to files.len() a miscount mis-renders one summary line; all four counters are derived in the pass directly above
mds-cli/output.rs:279 computed output path must not escape the out-dir its release fallback is already contained (base/<stem>.<ext>), so a release check would guard nothing
mds-cli/watch.rs:1476, :2498 merged last_mtimes equals the pre-compile snapshot mtime bookkeeping; a violation costs one redundant rebuild, never a wrong or lost write
mds-core/error.rs:175 at() cross-source span in bounds / on a boundary load-bearing debug-only — promoting it undoes this PR's degradation (M11)
evaluator.rs:960, :977 And/Or operands are leaves guaranteed by the parser's same-level flattening; per-node in the evaluation loop
evaluator.rs:1005, :1432 elseif_branches.len() <= MAX_ELSEIF_BRANCHES the bound is already enforced by the parser with a real error; these are redundant self-checks in a loop
formatter.rs:312, :359, :376 body-start / span-ordering / cursor char-boundary the formatter's compile-equivalence safety gate refuses to write on any bad output, so the release failure mode is a refused format, not a corrupt file
lexer.rs:181 scan_code_content requires code_fence.is_some() call-order precondition of a private fn with one caller directly above
lib.rs:720 RESERVED_OUTPUT_KEYS contains type and imports const-table self-check; can only fail if the table is edited, which a test covers
lint/fix.rs:618, :653, :660, :664 plan not overlap-rejected; edit in bounds; both offsets on boundaries apply_plan_unchecked is the explicit bypass; the production path is apply_fixes_incremental, which validates with real runtime checks
lint/fix.rs:1206 test-fixture offset points at @import inside #[cfg(test)] mod tests — not in any release build
options.rs:144 unknowns slice non-empty precondition on a slice the single caller builds from a non-empty iterator
parser.rs:76, :91 guard depth > 0 before decrement in Drop per-node in the hot parse loop; underflow would only affect a limit counter
parser_helpers.rs:641, :865 op.len() == 2; names.len() == name_offsets.len() both established by construction two lines above
resolver.rs:1654 skeleton entry's prompt_body is None self-check immediately after the binding it checks
sourcemap.rs:775, :786 display_names.len() == sources.len() parity established by construction at both insertion points
sourcemap.rs:827, :886 segments.len() <= MAX_SOURCEMAP_SEGMENTS the cap has real runtime handling (segments_dropped); these fire per segment in the compiler's hottest loop — G2 measured the cost of promoting three siblings in that same loop
mds-wasm/lib.rs:80 Reflect::set succeeded cannot fail on a fresh object; the release fallback skips one property of a diagnostic

Other counts

  • T-D1 real hit count: is_control_char claims exactly 78 codepoints (30 C0 + DEL + 32 C1 + U+061C + 14 three-byte format hazards). The planned pin of 78 was correct. T-D1 also asserts width parity and actual substitution for each of the 78.
  • line_len_at stays private and unchanged. parser.rs's directive_line_len was deliberately not unified with it (see limitations).

Gates at e1a69d0

gate result
cargo fmt --all --check clean
cargo clippy --workspace --all-targets -- -D warnings clean
cargo clippy -p mds-cli --all-targets --features startup-race-probe -- -D warnings clean
cargo nextest run -p mds-core -p mds-cli 2 352 passed, 0 skipped (mds-core 1 465, mds-cli 887, each also measured on its own). The combined run reports 1 leaky — a passing test that left a handle open, which nextest flags but does not fail; both single-crate runs report 0 leaky
G1 release-profile subset 14/14, 29.35 s
cargo test --workspace (CI's command) green — every target test result: ok, 0 failed
cargo test --doc -p mds-core 53 passed
cargo +1.88 check -p mds-core -p mds-cli clean
RUSTDOCFLAGS="-D warnings" cargo doc -p mds-core -p mds-cli --no-deps clean
WASM pkg/pkg-web mds_wasm_bg.wasm 851 728 B local; projected CI ≈ 844 863 B (guard 850 000)
npm test --workspaces on a rebuilt native addon 598 pass / 0 fail@mdscript/mds 304/304 (the source-map goldens), mds-napi 115, bundler-utils 91, rspack 26, vite 24, webpack 21, rollup 17
node scripts/verify-no-control-bytes.mjs clean — 563 files, 6 819 578 bytes
node scripts/verify-versions.mjs clean — 8 packages + 4 crates at 0.4.2
npm run test:gates 212 pass / 0 fail
decision-ledger ids in added lines none added (one pre-existing id rides a rewrapped doc line)
release-surface path diff empty

Known limitations / out of scope

  • No CLI panic hook / catch_unwind. A tripped invariant on the CLI is a raw Rust panic: exit 101, default panic message, and in mds watch it takes down the watcher. A hook that renders mds::internal and a catch_unwind around the watch rebuild loop are a follow-up; they were not added here because the panic message content is already constrained (no user data) and adding a hook changes the failure presentation of every existing panic site at once.
  • Exit-code asymmetry is intentional but unlovely: CLI 101 vs mds::internal (exit 1) through napi/WASM/Python. Same defect, two presentations.
  • SECURITY.md ~:83 is inaccurate and was left alone. It says mds-core exposes the debug-panics feature; in fact only the three binding crates do. Out of scope for a debug_assert!-only invariants silently degrade in release builds (3 sites) — PF-005 #220/output.rs path helpers fail-open on strip_prefix edge cases and non-UTF-8 paths #217 docs commit — recorded here, not fixed.
  • Lint granularity changed (not the exit code): a single unnameable entry now fails the whole mds lint <dir> run instead of being one per-file error among many. This is the fail-closed behaviour the issue asked for, but it does mean one bad filename hides the findings of every other file in the tree.
  • Remaining host-path sinks are a follow-up, deliberately not closed here: build.rs's SMv3 file label (lossy), fmt.rs's raw display() as a diagnostic file name, lib.rs's stem-less label, lint.rs's <file> sentinel, the two current_dir() fallbacks (canonicalize_out_dir still fails open — a relative --out-dir anchors at "."), and watch.rs's graph_key fallback. Turning an infallible helper fallible changes every caller, which is why canonicalize_out_dir was scoped out.
  • directive_line_len (parser.rs) was not unified with line_len_at. Two helpers with near-identical semantics remain.
  • The 850 000-byte WASM guard was not raised. The +658 B delta leaves ≈ 5 894 B of projected headroom; raising the guard would be a separate, deliberate decision.
  • O_NOFOLLOW on the write fd is still deferred (carried over from fix(cli): atomic mds build/watch outputs and .map sidecars via atomic_write_file (removes the spurious metadata warning); empty directory exits non-zero; atomic_write_file contract documented (#227, #225, #204, #226) #385).
  • P-M12 and the T-L1 lexical checks are named weak spots, documented above rather than silently accepted.
  • The on-disk hostile fixtures for T-P4/T-P5/T-P6/T-P13 have never executed. Their first real run is this branch's Linux CI job. If one fails there, the failure is in the assertion, not the implementation — the forced-branch runs already exercised each end to end.

Review observations (self-review, non-blocking)

  • The promoted TextEdit::new / FixLineSpan::range_* checks guard the constructor path only; in-tree lint rules build these types with struct literals (e.g. lint/rules/legacy_interpolation.rs, rules/empty_block.rs, rules/unreachable_branch.rs, rules/unused_function.rs, lint/fix.rs) and bypass the checks — a pre-existing gap the debug_assert! shared; the promotion is not broader than the constructors.
  • The flatten warning's text is never rendered end-to-end by a test (unreachable from any walked tree); its presence is pinned lexically (flatten_warning_lives_in_the_write_oracle_only) and its absence is asserted in dir_build.rs with a non-empty-stderr control.
  • Release-profile behaviour has no recurring CI gate: the degrade/promote tests branch on cfg!(debug_assertions), so CI executes the debug arm; the release arm rests on the local release-profile run (14/14) plus the lexical guard in tests/assert_promotions.rs.
  • source_path.rs's property table does not pin the relative-source × unusable-base combination (the case whose anchor changed from basename to root); behaviour is safe (apply_relative still rejects an escape above the root anchor) and unreachable from the CLI and bindings.
  • .devflow/features/mds-lint/KNOWLEDGE.md says "the other 11 format hazards" where is_three_byte_format_hazard covers 14 — pre-existing wording next to an edited line; the 78-codepoint non-vacuity pin uses the real count.
  • spliced_regions keeps an if let Some after the assert!; removing the assert later would silently reinstate a drop-the-region path — residual shape, deliberate under the pinned macro form.
  • The new Io messages interpolate the raw root and entry paths (path escapes lint root {root}: {path}), byte-identical in shape to the pre-existing read_source_file rejection; belongs with the path-sink follow-up inventory.
  • canonicalize_out_dir still falls back to . when current_dir() fails — documented in place, deferred to the path-sink follow-up.

Changes

crates/mds-core/src

  • resolver.rs — site 1 degrades via line_len_at.
  • resolver/inheritance.rs — site 2 degrades via super::line_len_at; site 3 unconditional assert! + # Panics rustdoc.
  • lint/diagnostic.rsneutralize_source_for_render assert_eq!; TextEdit::new / FixLineSpan::range_inclusive / range_exclusive assert! + # Panics.
  • evaluator.rs — three source-map cursor assert_eq!.
  • sourcemap.rsMapBuilder cursor-invariant rustdoc.
  • source_path.rspath_to_unifiedOption<String>; step 6b "Anchor validity"; module # Security invariant.
  • fs.rsNativeFs::source_root uses to_str; FileSystem::source_root # Contract.
  • resolver_tests.rsline_len_at test group (+ header note).
  • tests/assert_promotions.rsnew, the T-L1 shape test.

crates/mds-cli/src

  • lint.rsrelative_displayResult<String, MdsError>; display-path pre-pass before the sort; LintDirCtx.lint_root removed; per-file helpers take display_path: &str.
  • output.rs — new MirroredStem + mirror_stem; the flatten report in output_path_for only; two unwrap_or(source.as_os_str()) fallbacks → OsStr::new("output").
  • watch.rsforget split into forget_graph + forget; ghost-external-dep prune gated; three invariant comments at the output_path_for call sites.
  • fmt.rs — undecodable-path arm is MdsError::Io.

Testscrates/mds-cli/tests/{cli_lint.rs, dir_build.rs, cli_fmt.rs}.

DocsCHANGELOG.md, spec.md (§7.2, §7.5, §7.9), SECURITY.md, and four feature KBs under .devflow/features/.

Related Issues

Closes #220
Closes #217
Refs #69 #196

…nge checks, neutralize width parity and source-map cursor asserts pinned (#220)

Fifteen tests, added BEFORE the fix, that fail for the reason #220 exists.
Nothing in production code changes in this commit.

Behavioural tests (14, lib target):
  resolver_tests.rs  T-A1..A3  attach_import_span_*        (3)
  resolver_tests.rs  T-B1..B3  check_child_only_blocks_*   (3)
  resolver_tests.rs  T-C1,C2   spliced_regions_*           (2)
  lint/diagnostic.rs T-D1      neutralize_replacement_width_matches_for_every_hazard_codepoint
  lint/diagnostic.rs T-E1..E4  text_edit_new_reversed / fix_line_span_range_* / range_constructors_accept
  evaluator.rs       T-F1      evaluate_with_map_cursor_tracks_output_length_across_node_kinds

Shape test (1, integration target): tests/assert_promotions.rs T-L1
  promoted_asserts_are_unconditional_prose_and_justified — a fixed ten-row
  table (8 promoted asserts + 2 degrade sites), NOT a repo-wide sweep: every
  other debug_assert! in the workspace is deliberately debug-only. Per promoted
  row it pins the macro form (no debug_ prefix), the exact message literal
  (with backslash-continuations joined), the absence of any '{' in it, and a
  justification comment naming #220 within 12 lines above. Per degrade row it
  pins that line_len_at( is called and that neither debug_assert nor [offset..]
  survives in the function body (comments and string literals masked first).

DEV-profile RED run (cargo nextest, --no-fail-fast, 1445 skipped):
  Summary 15 tests run: 6 passed, 9 failed

  Failing, with the panic text that proves the premise:
    attach_import_span_non_boundary_offset_degrades_to_zero_len_span
      resolver.rs:2513 "attach_import_span: offset 1 is not a UTF-8 char
      boundary in source (len=3)" — the site's OWN debug assert fires, not the
      MdsError::at() canary the degraded path must reach.
    attach_import_span_out_of_range_offset_on_empty_source_degrades
      resolver.rs:2513 "... (len=0)" — fires even though MdsError::at()
      exempts an empty source.
    check_child_only_blocks_non_boundary_offset_degrades
      inheritance.rs:77 "check_child_only_blocks: offset 1 is not a UTF-8 char
      boundary in source (len=3)"
    check_child_only_blocks_out_of_range_offset_on_empty_source_degrades
      inheritance.rs:77, same shape.
    spliced_regions_missing_effective_block_panics_in_every_profile
      should_panic expected "override map was not built for this skeleton";
      the current debug_assert!(false, ...) panics with different text.
    text_edit_new_reversed_range_panics
      panic message "TextEdit::new: start (5) > end (4)"; expected substring
      "TextEdit::new: start is greater than end".
    fix_line_span_range_inclusive_reversed_panics
      "FixLineSpan::range_inclusive: from (5) > to (4)" vs expected
      "FixLineSpan::range_inclusive: from is greater than to".
    fix_line_span_range_exclusive_reversed_panics
      "FixLineSpan::range_exclusive: from (5) > to (4)" vs expected
      "FixLineSpan::range_exclusive: from is greater than to".
    promoted_asserts_are_unconditional_prose_and_justified
      assert_promotions.rs:153 left: "debug_assert!" right: "assert!" at
      resolver/inheritance.rs:175 — the scanner really reads the macro form.

  Passing controls (6): attach_import_span_valid_offset_spans_whole_directive_
  and_passes_other_errors, check_child_only_blocks_valid_offset_spans_stray_
  line_and_accepts_blocks, spliced_regions_pairs_every_skeleton_block_with_its_
  effective_entry, range_constructors_accept_equal_and_ordered_bounds,
  neutralize_replacement_width_matches_for_every_hazard_codepoint,
  evaluate_with_map_cursor_tracks_output_length_across_node_kinds.

G1 RELEASE-profile RED run (the proof CI cannot give — CI runs Rust tests in
dev only). Form: plain `cargo nextest ... --cargo-profile release --lib`, no
LTO override needed; 31 s wall for list + build + run (30.45 s of that is the
release compile).
  `cargo nextest list -p mds-core --cargo-profile release --lib -E
   'test(/attach_import_span_|check_child_only_blocks_|spliced_regions_|
   text_edit_new_reversed|fix_line_span_range_|range_constructors_accept|
   neutralize_replacement_width|evaluate_with_map_cursor/)'`
  listed exactly 14 tests.
  Summary 14 tests run: 6 passed, 8 failed, 1149 skipped

  The four degrade tests fail here for a DIFFERENT and worse reason than in
  debug — the raw slice, not a compiled-out assert:
    resolver.rs:2518        "start byte index 1 is not a char boundary; it is
                             inside 'e-acute' (bytes 0..2 of string)"
    resolver.rs:2518        "start byte index 1 is out of bounds for string of
                             length 0"
    inheritance.rs:83       both of the above, same two inputs.
  This is the measured refutation of the issue's premise for sites 1 and 2:
  they do not "silently degrade" in release, they panic one line below the
  assert that was supposed to catch the condition.

  The four promotion tests fail with "test did not panic as expected":
    diagnostic.rs:2606, :2612, :2618 (the three range constructors) and
    resolver_tests.rs:3531 (spliced_regions) — i.e. in release the invariant
    is not enforced at all.

T-D1 non-vacuity: the codepoint walk found exactly 78 members of
is_control_char (30 C0 + DEL + 32 C1 + U+061C + 14 three-byte format hazards);
the pin matches the real count, no adjustment needed.

Gates on this commit: cargo fmt --all --check clean;
cargo clippy --workspace --all-targets -- -D warnings clean (exit 0).
… enforce the skeleton-block, neutralize-width, range-check and source-map-cursor invariants in release (#220)

Two remedies, chosen per site by what a defect there actually costs.

DEGRADE (2 sites). Both computed a span length by slicing source[offset..]
one line below a debug_assert! on the very condition the slice needs. In
release the assert vanishes and the slice panics; in debug the site's own
assert fires first and hides the real one. Both now call the existing private
helper line_len_at, which returns 0 for a non-boundary or out-of-range offset.
MdsError::at then keeps the numeric offset and drops the snippet, so the
diagnostic degrades instead of mis-attributing or crashing.
  resolver.rs                attach_import_span
  resolver/inheritance.rs    check_child_only_blocks  (super::line_len_at)
line_len_at stays private and in place; parser.rs directive_line_len is
untouched.

ENFORCE (8 asserts). Each is a pure code-shape invariant that no template
input can reach, and each had a release behaviour strictly worse than a panic.
Every message is prose with no interpolation: a promoted assert is a release
panic whose text reaches the terminal through the CLI's default handler (the
CLI installs no panic hook), so no block name, offset, or source text may
appear in it.
  resolver/inheritance.rs  spliced_regions
      was debug_assert!(false, "... block '{name}' ...") + a release fallback
      that spliced the base default in SILENCE — i.e. dropped a child's @block
      override from the compiled output with no diagnostic at all. Now
      assert!(eff_block.is_some(), ...) on the real predicate; the fallback is
      gone. rustdoc gains a # Panics section; the pre-existing (PF-004) text is
      kept verbatim.
  lint/diagnostic.rs  neutralize_source_for_render
      byte-length parity: a width mismatch does not fail, it shifts every span
      offset and caret column after the first substitution.
  lint/diagnostic.rs  TextEdit::new, FixLineSpan::range_inclusive,
                      FixLineSpan::range_exclusive
      public API range checks; every applier in the fix pipeline skips a
      reversed range rather than failing, so the lint would report a fix it
      never made. rustdoc "# Panics (debug)" -> "# Panics"; TextEdit::new gains
      a # Panics section it did not have.
  evaluator.rs  evaluate_nodes, Text / EscapedBrace / Interpolation arms
      source-map cursor: a desynchronised cursor mis-attributes the whole rest
      of the map silently. sourcemap.rs MapBuilder rustdoc updated to say an
      unconditional assert_eq! checks it.

Everything else stays debug_assert! — deliberately, one line each:
  resolver.rs:1654       tautology pin for a planned refactor, not an invariant
  error.rs:175           debug-only canary BY DESIGN: promoting it would make
                         the degraded release path panic, undoing this fix
  sourcemap.rs 775/786/827/886   encoder-internal, hot per-segment
  evaluator.rs 960/977/1005/1432 evaluator-internal, hot per-node
  parser_helpers.rs 641/865, parser.rs 76/91, lexer.rs 181  parse-time internals
  lint/fix.rs 618/653/660/664    planner internals; :631 is already an
                                 unconditional assert and stays as it is
  formatter.rs 312/359/376       formatter internals behind a safety gate
  lib.rs:720, options.rs:144     API-surface internals
  mds-cli output.rs:279, lint.rs:1488, watch.rs 1476/2498  CLI internals
  mds-wasm lib.rs:80             binding internal

GATES (all three measured, numbers verbatim).

G1 — release-profile proof. CI runs Rust tests in dev only, so this is local
and mandatory. Form: plain `cargo nextest ... --cargo-profile release --lib`;
no CARGO_PROFILE_RELEASE_LTO override was needed and no [profile.release-test]
was added. Wall time 31 s for list + build + run.
  list -> exactly 14 tests
  run  -> Summary 14 tests run: 14 passed, 1149 skipped
(the same 14 were 6 passed / 8 failed at the RED commit).
Dev profile, same 14 plus the T-L1 shape test: 15 passed, 1445 skipped.
Whole workspace: 2335 tests run: 2335 passed (2320 at the RED commit's parent
+ 15 new). Doc-tests mds: 53 passed.

G2 — evaluator cost, the gate on the three source-map cursor asserts.
Fixture: 180,000 lines of "Hello {{name}} \{{ tail text" over a frontmatter
name: World (5,220,020 bytes; 4,500,020-byte output, 3,600,083-byte map).
Binaries: target/release/mds built at the RED commit (production code then
byte-identical to main) vs built from this commit. Command, x5 each,
back-to-back: /usr/bin/time -p <bin> build big.mds --source-map -o out.md

  round 1, wall seconds
    before  0.24  0.21  0.22  0.21  0.21   median 0.21
    after   0.69  0.21  0.21  0.22  0.21   median 0.21
    (the 0.69 is the first touch of a freshly copied binary; it does not recur)
  round 2, wall / user seconds
    before  0.21/0.16 x5                   median 0.21 / 0.16
    after   0.21/0.16 x5                   median 0.21 / 0.16
  noise control, x3 each, no --source-map (the asserts are unreachable)
    before  0.17  0.17  0.17               median 0.17
    after   0.17  0.17  0.16               median 0.17

  Threshold: median wall(after) <= 1.05 x median wall(before)
             0.21 <= 0.2205   PASS  (ratio 1.000)

  Supplementary, higher resolution (the specified fixture runs in ~0.21 s, so
  /usr/bin/time's 10 ms granularity is 4.8% of the measurement): 320,000 lines,
  9,280,020 bytes, 960,000 segments — just under MAX_SOURCEMAP_SEGMENTS.
    before  0.29 x5            median wall 0.29, median user 0.22
    after   0.29 0.29 0.30 0.30 0.31   median wall 0.30, median user 0.22
    ratio 1.034 <= 1.05   PASS   (user-CPU median unchanged)

  Verdict: PASS on two independent rounds. The three evaluator assert_eq!s are
  KEPT; no revert to debug_assert_eq! was needed, and the three evaluator rows
  stay in the shape-test table.

G3 — WASM size. Measured with CI's own command
(wasm-pack build crates/mds-wasm --target nodejs --out-dir pkg, and --target
web --out-dir pkg-web) against the same two trees as G2. Local wasm-opt is
wasm-pack's cached v117, not CI's pinned Binaryen v129, so only the DELTA
transfers.
    before  pkg 850,313   pkg-web 850,313
    after   pkg 850,971   pkg-web 850,971
    delta   +658 bytes (+0.077%) on both — the eight new message strings.
  Local sits 6,865 bytes above the CI-measured 843,448 at 33380d7, inside the
  known 3-7 KB local/CI offset. Projected CI: ~844,106 bytes, leaving ~5,894
  bytes under the 850,000 guard. Far below the 5 KB trigger that would have
  forced shorter messages. ci.yml and the guard are untouched.

Gates on this commit: cargo fmt --all --check clean;
cargo clippy --workspace --all-targets -- -D warnings clean.
`mds lint <dir>` builds every `files[].file` wire key (and the directory sort
key) with `relative_display`, which fails OPEN twice: `strip_prefix(root)
.unwrap_or(path)` rebuilds a key out of an out-of-root path's own components,
and `to_string_lossy` turns an entry name that is not valid UTF-8 into U+FFFD
replacement characters. Either way a key that names no file under the linted
directory reaches stdout.

Three tests pin the fail-closed contract; each pairs its rejections with a
positive control so it cannot pass by rejecting everything.

RED observations (this commit, macOS):

- relative_display_rejects_path_outside_root — two rejection arms. The absolute
  arm `/other/x.mds` under root `/lint-root` expected Err("escapes lint root")
  and got Ok("other/x.mds"): the leading `/` is dropped and the host path is
  re-emitted as if it were relative to the lint root. The relative arm
  `other/x.mds` under root `lint-root` is the arm that is sensitive to the
  `strip_prefix` result alone — every component there is `Normal`, so a
  component-shape guard cannot see it — and it likewise got Ok("other/x.mds").
  Controls: both roots must still yield Ok("a/b.mds") for an in-root path.
- relative_display_rejects_non_utf8_component — expected Err("not valid
  UTF-8"), got Ok of a key whose first two characters are U+FFFD replacement
  characters followed by ".mds".
- lint_directory_non_utf8_entry_is_an_io_error_exit_2 — the on-disk half is a
  Linux-CI gate: macOS APFS/HFS+ enforce valid UTF-8 filenames and reject the
  create with EILSEQ, so the hostile arm cannot be built on this machine. The
  test runs its two positive controls here (exit 1 and `files[0].file ==
  "ok.mds"` in JSON mode; the literal `clean,` summary needle in human mode)
  and then returns at an explicitly macOS-gated branch; on any other unix
  filesystem a rejected create panics rather than skipping silently. Expected
  RED on Linux: a `files` key present in the envelope and a summary line on
  stderr.

Interface scaffold so the RED tests compile and fail on BEHAVIOUR rather than
on a type error: `relative_display` already returns
`Result<String, MdsError>`, with the existing fail-open body wrapped in `Ok`
and the three production call sites carrying a temporary `.expect`. The GREEN
commit replaces the body, the call sites and that scaffold. The four existing
`relative_display` unit tests switch to `.expect("in-root path")` with their
assertions byte-identical.

Counts: mds-cli 878 tests (875 before), 876 passed, 2 failed (the two RED unit
tests above). fmt and clippy clean.

Refs #217
…fore the sort; run exits 2 with the analysis-failure envelope (#217)

`relative_display` no longer manufactures a key it cannot justify. It returns
`Err(MdsError::Io { .. })` when the path is not under `root`, when a component
that survives the strip is not `Normal`, or when a component is not valid
UTF-8 — instead of `strip_prefix(root).unwrap_or(path)` plus a `to_string_lossy`
join. For every in-root UTF-8 path the `Ok` value is byte-identical to the
previous output, so no existing assertion moved.

`run_lint_directory` now computes every display key in a pre-pass BEFORE the
sort and before the per-file loop, and `lint_one_file_accumulating` /
`lint_one_file_human` take `display_path: &str` instead of recomputing it. Two
orderings are load-bearing and are commented at the pre-pass:

- before the sort — the sort key IS the display key, so a lossy key that only
  failed later would already have been built and compared;
- before the per-file loop — nothing is linted, so no `files[]` entry can carry
  a key that names no file under the linted directory.

The pre-pass does NOT use `?`. The caller returns `miette::Result`, so a bare
`?` would render a human diagnostic and drop the JSON analysis-failure envelope
that `--format json` consumers parse. It follows the existing symlinked-root
shape instead: `emit_analysis_failure_json_or_stderr(&e, format, None)` then
`std::process::exit(mds_error_exit_code(&e))`. The non-UTF-8 message text is
the one `read_source_file` already produced for the same condition, so the
user-visible wording is unchanged; the exit code is unchanged at 2. What
changes is WHEN: the whole run now fails before any file is linted, instead of
the tree being linted with one entry keyed by U+FFFD replacement characters.

`LintDirCtx.lint_root` existed only to feed those two per-file calls and is
removed; the `debug_assert_eq!` partition invariant now counts `keyed`.

Verified locally by forcing `relative_display` to error and running the binary
over a real directory: `--format json` emits
`{"error":{"code":"mds::io","message":"path is not valid UTF-8: …",…},"version":1}`
with no `files` key and exit 2; human mode prints the same message with no
summary line and exit 2. That is the exact shape
`lint_directory_non_utf8_entry_is_an_io_error_exit_2` asserts on Linux, whose
hostile arm cannot be built on macOS — APFS/HFS+ reject a non-UTF-8 filename
with EILSEQ at create. The test handles that with an explicitly macOS-gated
early return placed AFTER its positive controls have run; on any other unix
filesystem a rejected create panics rather than skipping silently.

Mutation controls (applied, observed, reverted):

- `strip_prefix(root).unwrap_or(path)` reintroduced →
  relative_display_rejects_path_outside_root fails on its relative arm
  ("other/x.mds"). Recording the first run of this control honestly: with only
  the absolute arm the mutant SURVIVED, because `/other/x.mds` is then rejected
  by the `Component::Normal` guard rather than by the strip. The relative arm
  and its control were added to the RED commit for exactly that reason, and the
  mutant now dies.
- `to_string_lossy` reintroduced → relative_display_rejects_non_utf8_component
  fails, reporting a key of two U+FFFD replacement characters.
- pre-pass replaced by a per-file lossy key: not observable on macOS, where the
  hostile arm cannot be created. The forced-error run above is the local stand-in
  — it exercises the envelope + exit code + absent summary that this mutation
  would break.

Counts: mds-cli 878 tests, 878 passed, 0 failed. fmt, clippy and the
source-hygiene gate clean.

Refs #217
…acuous (#217)

`relativize_source` converts its `root` and `base` anchors with
`path_to_unified`, which is `p.to_str().unwrap_or("")`. An anchor that cannot
be decoded therefore becomes the empty string, `normalize_abs("")` becomes the
EMPTY component list, and `starts_with_comps(x, [])` is vacuously true — an
empty component list is a prefix of every path. So an unusable root passes
containment for every source, and `component_diff` then emits the full host
path minus its leading `/` into `sources[]`. That is precisely the filesystem
layout the guard exists to keep out.

`NativeFs::source_root` has the mirror-image problem: it reports
`p.display().to_string()`, so a root that is not valid UTF-8 is handed to the
guard as a lossy string that names no directory and cannot be compared
component-wise against a real source.

Six tests, each pairing its degradation with a positive control.

RED observations (this commit):

- empty_root_is_not_a_vacuous_anchor — root `""`, source
  `/Users/alice/proj/src/a.mds`: expected `"a.mds"`, got
  `"Users/alice/proj/src/a.mds"` — the host path with the leading `/` removed.
- non_utf8_root_degrades_to_basename — root `/Users/alice/proj` plus one
  invalid byte, source `/Users/alice/projX/src/a.mds`: expected `"a.mds"`, got
  `"Users/alice/projX/src/a.mds"`. Control root `/Users/alice/projX` already
  yields `"src/a.mds"`.
- property_outputs_never_absolute_or_drive_qualified — the new `root: Some("")`
  case leaks `"Users/alice/proj/src/a.mds"`. The matrix's existing round-trip
  invariant cannot catch this on its own: an empty root is a prefix of every
  path, so the check is satisfied vacuously. The added assertion is explicit
  about the leak.
- source_root_is_none_for_non_utf8_root — a root of `/tmp/` plus one invalid
  byte: expected `None`, got `Some` of a string ending in a U+FFFD replacement
  character.

Two of the six are PINS that are green before and after, recorded as such
rather than presented as RED:

- non_utf8_base_falls_back_to_root_anchor — an unusable BASE already lands on
  the root anchor today, because step 9 asks whether the base is inside the
  root and an empty component list is not (`starts_with_comps([], ["proj"])`
  is false on the length check). The test pins that the behaviour survives the
  change, with a control proving a usable base still shifts emission to
  `"../src/a.mds"`.
- root_slash_is_a_real_root_not_a_vacuous_one — `normalize_abs("/")` is also
  the empty component list, but `/` is a legitimate project root (a container
  with no WORKDIR resolves there) and must keep root-relative emission. This
  pins the deliberate non-change: the distinction to draw is "the unified
  string is empty or undecodable" versus "the unified string is `/`".

The invalid bytes are built at runtime from numeric values; no escape sequence
or raw byte appears in any test source.

Counts: mds-core 1465 tests (1460 before), 1461 passed, 4 failed (the four RED
observations above). fmt, clippy and the source-hygiene gate clean.

Refs #217
…ble base → root anchor; source_root is None for a non-UTF-8 root (#217)

`path_to_unified` now returns `Option<String>`, `None` for a path that is not
valid UTF-8 or that unifies to the empty string. The whole hazard was that its
old `to_str().unwrap_or("")` fed `normalize_abs("")`, which is the EMPTY
component list — and `starts_with_comps(x, [])` is vacuously true. An anchor
that could not be decoded therefore authorised containment for every source,
and `component_diff` emitted the full host path minus its leading `/` into
`sources[]`.

`relativize_source` gains step 6b, immediately after the existing `root = None`
branch:

- root that is `None` from `path_to_unified` → `basename_fallback(&norm_comps)`;
- base that is `None` from `path_to_unified` → the root anchor, at BOTH of its
  uses (the step-7 resolution anchor and the step-9 emission anchor `b`).

Root `/` is deliberately NOT affected. It unifies to `"/"`, which is not empty,
so it stays a real root and keeps root-relative emission even though
`normalize_abs("/")` is also the empty component list — a container with no
WORKDIR resolves there. The distinction the guard draws is "the unified string
is empty or undecodable", not "the component list came out empty".

`NativeFs::source_root` uses `to_str` instead of `display()`, so a root that is
not valid UTF-8 is reported as `None` — the documented "no containment concept"
value with a guarded branch at every consumer (never absolute, never
drive-qualified, `..`-escape → basename) — rather than a lossy string naming a
directory that does not exist. A sentinel string would be a fake path leaking
through the public trait. Containment itself is untouched:
`check_path_traversal` compares `Path`s from `root_dir` directly and never goes
through this string. The trait rustdoc states the obligation for third-party
implementations, and the `relativize_source` rustdoc and the module's security
invariant both gain the anchor-validity rule.

No new error variant and no sentinel: the change is entirely in the Option
routing.

Source-map goldens: NONE moved. The design guarantees zero change for any
UTF-8 non-empty root and any base, and that is what the surfaces show —
`packages/mds` 304/304 pass (including `source maps (U-SM)`, `source maps —
WASM backend (W-SM)` and `source maps — compileFile differential (CF-SM)`)
against a freshly rebuilt native addon, and the Python suite is 250 passed /
6 deselected against a freshly built extension. The CLI's own source-map
assertions in output.rs and watch.rs are part of the green mds-cli run. The
WASM path is structurally unaffected: `VirtualFs::source_root` is the trait
default `None`, so it never reaches step 6b. An empty base is likewise
unreachable from the CLI — `compute_source_map_base` absolutizes every branch.

Mutation controls (applied, observed, reverted):

- `path_to_unified(root).unwrap_or_default()` → empty_root_is_not_a_vacuous_anchor,
  non_utf8_root_degrades_to_basename and the property matrix all fail, each
  reporting the host path minus its leading `/`.
- unusable base mapped to the empty component list instead of `None` →
  non_utf8_base_falls_back_to_root_anchor stays GREEN, and that is correct:
  step 9 already asks whether the base is inside the root, and an empty
  component list is not (the length check fails). The mutation is
  behaviour-preserving for the tested paths; it is recorded here rather than
  papered over with a test that would only pin an implementation detail.
- `source_root` back to `display().to_string()` →
  source_root_is_none_for_non_utf8_root fails with `Some` of a string ending in
  a U+FFFD replacement character.

Counts: workspace 2343 tests (mds-core 1465, mds-cli 878), all passing;
doctests 53. fmt, clippy, the rustdoc gate
(`RUSTDOCFLAGS="-D warnings" cargo doc -p mds-core -p mds-cli --no-deps`) and
the source-hygiene gate are clean.

Refs #217
…absolute; watch ghost prune keeps in-root state (#217)

Six tests over the directory-mode output oracles and the watch bookkeeping that
probes them. `MirroredStem` + `mirror_stem` are added here as an INTERFACE
SCAFFOLD only: the Err arm reproduces today's fail-open fallback verbatim
(`d.join(source.file_stem().unwrap_or(source.as_os_str()))`), so the
classification can be pinned before the behaviour changes. The follow-up commit
replaces that body and wires the report.

RED observations (`cargo nextest run -p mds-cli --bins --test dir_build
--no-fail-fast`, 204 tests, 201 passed, 3 failed):

- stemless_source_never_escapes_out_dir — FAILED.
  `output_base_no_ext(Path::new("/"), "/root", Dir("/out"))` returned `"/"`,
  not `"/out/output"`: `"/".file_stem()` is None, the fallback handed
  `source.as_os_str()` — `"/"` — to `d.join(...)`, and joining an absolute
  argument re-roots. Suppressing that first assertion locally to reach the
  second showed the write oracle's half: `output_path_for` built `"/.md"` and
  tripped the containment `debug_assert!` at output.rs:279 —
  `output_path_for: AC-M7 violated — output "/.md" escaped out-dir "/out"`.
  Neither value is inside the out-dir; neither is a path a source compiles to.
- flatten_warning_lives_in_the_write_oracle_only — FAILED. The body of
  `fn output_path_for(` contains no `eprint_warning(` and no
  "is outside the build root": the flatten is currently silent.
- ghost_external_dep_prune_keeps_in_root_last_written — FAILED.
  After one batch, `last_written` held only `[<root>/other.md]`; the in-root
  source's `out/x.md` entry was gone. Pruning a vanished OUT-OF-ROOT dep probes
  `output_base_no_ext`, takes the flatten arm, and forgets `<out-dir>/x.md` —
  an entry an in-root `x.mds` owns. Its control (`errored` no longer holds the
  ghost) passed, so the batch did reach the branch.

Green in this commit (pins and controls, not regressions):

- mirror_stem_classifies_out_of_root_as_flattened — passes on the scaffold;
  both arms asserted so a classifier that always says Flattened would fail.
- build_non_utf8_path_exits_2_and_writes_no_map — control arm pins
  `--source-map` on a normal file: exit 0, sidecar present, `sources ==
  ["ok.mds"]`. The hostile arm is a Linux-CI gate: macOS (APFS) rejects
  creating a non-UTF-8 filename with EILSEQ, so the create-failure path takes
  an explicit macOS-only `return` with a sibling `panic!` on every other
  platform — no silent skip.
- dir_build_symlinked_ancestor_root_mirrors_without_warning and
  dir_build_dotdot_root_mirrors_without_warning — non-canonical roots (a
  symlinked ancestor; a `..` component) still mirror their subtree and must not
  emit the report. `run_build_directory` hands the same raw `dir` to the walker
  and to `output_path_for`, so `strip_prefix` succeeds; these fail if a future
  change canonicalizes one and not the other. Each carries the
  "2 built, 0 failed" needle so the negative assertion cannot pass on empty
  stderr.

`cargo fmt --all` clean; `cargo clippy --workspace --all-targets -- -D warnings`
and `cargo clippy -p mds-cli --all-targets --features startup-race-probe --
-D warnings` both clean.
…s are relative, watch external-ghost prune skips the output probe (#217)

Three changes plus four invariant comments. Nothing about the mirrored (normal)
path moves; the flatten stays CONTAINED and stays a flatten.

1. `mirror_stem`'s Err arm no longer joins an absolute value.
   `file_stem()` is `None` only for `/`, `..` and a bare drive prefix — never a
   `.mds` file — and the old fallback, `source.as_os_str()`, was exactly the
   absolute value that escapes: `Path::new("/out").join("/")` re-roots to `"/"`.
   The literal `"output"` is the only value guaranteed relative for every input.
   The same substitution is applied to the two `unwrap_or(source.as_os_str())`
   fallbacks in `output_path_for`. The containment `debug_assert!` is KEPT and
   stays debug-only on purpose: its release fallback is contained, so there is
   nothing a release-time check would prevent.

2. `output_path_for` reports the flattened arm — once, inline, with `safe_path`
   at each interpolation site, not gated on `--quiet` (the depth-limit and
   stale-unlink warnings in the same file are not either; this is an invariant
   report, not a user preference). `output_base_no_ext` computes the same stem
   for bookkeeping probes and stays SILENT: watch calls it several times per
   batch and for sources that are never written.

3. `DirWatchState::forget` splits into `forget_graph` (the three graph removals)
   and `forget`. The ghost-prune at the `!src.exists()` branch of
   `process_dir_batch_incremental` now calls `forget_graph` for an out-of-root
   source. This was a REACHABLE bookkeeping bug: a vanished external dep has no
   output, so probing `output_base_no_ext` for one took the flatten arm and
   forgot `<out-dir>/<file name>.md` — an entry owned by the in-root source with
   that file name, whose next rebuild then rewrote identical bytes.

Comment-only, no behaviour change: `canonicalize_out_dir`'s fail-open
`current_dir()` fallback is documented as such with the reason it is a follow-up
rather than part of this change; `is_within_default_excluded_dir`'s `Err => false`
is documented as the CLOSED answer (both callers evaluate it only alongside their
own `starts_with(root)` test); `is_partial`'s non-UTF-8 `false` is documented as
fail-closed downstream (mds-core's `path_to_str` rejects the path, so the file
becomes a per-file failure instead of a silent skip); and the three watch
`output_path_for` call sites each state why the flatten arm cannot fire there.

Exact warning text as emitted (one line, `{}` filled by `safe_path`):

  warning: <source> is outside the build root <root>; its output is written flat
  as <out> (another source outside the root with the same file name would
  overwrite it)

GREEN — `cargo nextest run -p mds-cli --no-fail-fast`: 885 tests run, 885
passed, 0 skipped (was 878 before this phase; +7 new tests). All four RED tests
now pass. `print_discipline` both tests green, registry untouched
(`every_allowlist_entry_is_live` passes, so no entry went dead).

`cargo nextest run -p mds-cli --test cli_watch --no-fail-fast` x3, sequential:
80/80 passed each run (4.54 s / 4.44 s / 5.07 s). No timeout, sleep, debounce or
ordering change was made.

Mutation controls, each observed then reverted:
- report deleted from the write oracle and moved into the probe oracle:
  flatten_warning_lives_in_the_write_oracle_only fails on the positive half
  ("the write oracle must report the flattened arm").
- report present in BOTH oracles: the same test fails on the negative half
  ("the probe oracle must stay silent"), so neither half is vacuous.
- `safe_path` dropped at the site and the value hoisted into a `let`:
  print_discipline reports `output.rs:334: eprint_warning(format!) interpolates
  unsanitized \`src_display\``.
- Err-arm `unwrap_or(source.as_os_str())` reintroduced:
  stemless_source_never_escapes_out_dir fails, left `"/"` right `"/out/output"`.
- watch gate forced to `if true`:
  ghost_external_dep_prune_keeps_in_root_last_written fails, `last_written` keys
  reduced to `[<root>/other.md]`.

Named weakness: T-P16 pins the report's PRESENCE in `output_path_for`
lexically, so it would still pass if the call were wrapped in a `!quiet` gate.
`output_path_for` has no `quiet` in scope, so that mutation is not reachable
without a signature change; recorded rather than performed.

`cargo fmt --all` clean; `cargo clippy --workspace --all-targets -- -D warnings`
and `cargo clippy -p mds-cli --all-targets --features startup-race-probe --
-D warnings` both clean.
Two tests. `fmt_non_utf8_path_exits_two` (cli_fmt.rs) is the RED one;
`lint_single_file_non_utf8_path_exits_2` (cli_lint.rs) is the PIN it must match —
`lint`'s single-file arm has always raised `MdsError::Io` here and `exit_code` has
always mapped that to 2. Its directory sibling
(`lint_directory_non_utf8_entry_is_an_io_error_exit_2`) was added earlier in this
branch; this is the file arm.

`mds fmt`'s module doc has promised "2: file not found / not `.mds` / I/O / bad
UTF-8" since the subcommand shipped, and `check_symlink` — two statements earlier
in the same `read_source_file` — already exits 2. The undecodable-path arm raised
a bare `miette::miette!`, which does not downcast to `MdsError`, so `exit_code`
fell through to 1. One input class, two exit codes, decided by which of two
adjacent checks caught it first.

RED observation. macOS (APFS) rejects creating a non-UTF-8 filename with EILSEQ,
so the on-disk arm cannot run here — it is a Linux-CI gate, taken via an explicit
`#[cfg(target_os = "macos")] return` with a sibling `panic!` on every other
platform, so no other platform can skip silently. To observe the arm locally the
`to_str()` success path was temporarily forced to `None`
(`.filter(|_| false)`), the real binary rebuilt, and run over a normally named
file:

    $ mds fmt <scratch>/ok.mds
      × path is not valid UTF-8: <scratch>/ok.mds
    EXIT=1                         # must be 2

The same forced build in DIRECTORY mode:

    $ mds fmt <scratch>
      × path is not valid UTF-8: <scratch>/ok.mds
    0 formatted, 0 unchanged, 1 failed
    EXIT=1                         # correct, and must STAY 1

Directory mode exits 1 through the `Failed` tally, not through `exit_code`, so it
is unaffected by the fix and is deliberately not asserted here. The forced branch
was reverted before committing.

Both tests pass on macOS today — the control arms run, the hostile arms return
early. The hostile assertions first execute on Linux CI.

`cargo nextest run -p mds-cli --test cli_fmt --test cli_lint --no-fail-fast`:
176 tests run, 176 passed, 0 skipped. `cargo fmt --all --check` clean;
`cargo clippy --workspace --all-targets -- -D warnings` and
`cargo clippy -p mds-cli --all-targets --features startup-race-probe -- -D warnings`
both clean.
)

`read_source_file` raised the undecodable-path failure as a bare
`miette::miette!`. That report does not downcast to `MdsError`, so
`build::exit_code` fell through to 1 — while `check_symlink`, called one
statement earlier on the same argument, already returned an `MdsError` and exited
2. The module doc has documented 2 for this class since the subcommand shipped.

The message text is byte-identical to `lint.rs`'s `read_source_file`
(`path is not valid UTF-8: {}` over `path.display()`), so the two subcommands now
report the same condition the same way. No new `MdsError` variant; no change to
`exit_code`.

GREEN observation. macOS (APFS) cannot create the on-disk fixture, so the arm was
exercised by temporarily forcing the `to_str()` success path to `None`
(`.filter(|_| false)`), rebuilding the real binary, and running it — the same
procedure that produced the RED reading in the previous commit:

    $ mds fmt <scratch>/ok.mds
    mds::io
      × path is not valid UTF-8: <scratch>/ok.mds
    EXIT=2                         # was 1

DIRECTORY mode is UNCHANGED, as intended:

    $ mds fmt <scratch>
    mds::io
      × path is not valid UTF-8: <scratch>/ok.mds
    0 formatted, 0 unchanged, 1 failed
    EXIT=1                         # was 1

Directory mode exits through the `Failed` tally, not through `exit_code`, so the
error class it reports does not reach the exit status. Nothing about it moves.
The forced branch was reverted before committing.

Full gate battery on the final tree:
- `cargo nextest run -p mds-core -p mds-cli --no-fail-fast`: 2352 tests run,
  2352 passed, 0 skipped (mds-core 1465 unchanged, mds-cli 887 — was 878 at the
  start of this phase, +9: 7 in the output/watch RED commit, 2 here).
- `cargo clippy --workspace --all-targets -- -D warnings` — clean.
- `cargo clippy -p mds-cli --all-targets --features startup-race-probe --
  -D warnings` — clean.
- `cargo fmt --all --check` — clean.
- `RUSTDOCFLAGS="-D warnings" cargo doc -p mds-core -p mds-cli --no-deps` — clean.
- `node scripts/verify-no-control-bytes.mjs` — 563 files, 6811840 bytes, clean.
…GELOG, spec §7.2/§7.5/§7.9, SECURITY.md, rustdoc, KBs (#220, #217)

Documents the behaviour landed by the ten preceding commits on this branch. No
production code changes; the only .rs edit is a test-group comment.

CHANGELOG [Unreleased]:
- Changed: the pub-API range-check promotion (TextEdit::new,
  FixLineSpan::range_inclusive/range_exclusive now panic in release).
- Fixed: the #220 release-enforcement entry (non-boundary import/extends spans
  degrade instead of panicking; skeleton-block, neutralize-width and source-map
  cursor invariants enforced in every profile), plus four #217 entries — lint
  directory-mode fails closed on an unnameable path, fmt non-UTF-8 exits 2,
  source-map anchors fail closed, and the build/watch flatten warning.

spec.md:
- §7.2 dir-mode --out-dir: the out-of-root flat write and its non-quiet warning.
- §7.5 dir-mode lint: entries are named relative to the lint root before any
  file is linted; an unnameable path fails the whole run.
- §7.9: both exit-2 rows name the non-UTF-8 path case.

SECURITY.md: new filesystem-boundary bullet — source-map anchors are
byte-faithful, and an empty or non-UTF-8 root can never make containment vacuous.

rustdoc: the B.6 checklist was verified site by site; every item was already in
place from the earlier commits except the line_len_at test-group header in
resolver_tests.rs, which now records that the helper also backs
attach_import_span and check_child_only_blocks.

Feature KBs: mds-lint (neutralize assert_eq! promotion x2, relative_display now
fallible and LintDirCtx.lint_root gone), source-map-security (evaluator cursor
assert, the empty/non-UTF-8 anchor gotcha, source_root None for a non-UTF-8
root), mds-cli (AC-M7 paragraph rewritten for MirroredStem/mirror_stem, the
write-oracle-only flatten warning, relative degenerate stems, the watch
forget_graph prune), mds-fmt (single exit-code correction).

Refs #220, #217
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant