fix(syntax): stop the formatter dropping reader prefixes

[takeokunn]

edit format silently dropped reader prefixes at exit 0, with output that reparses — the corrupted-Lisp-still-reparses failure mode. Found independently by three agents doing unrelated reader work.

Real damage in the corrupting set:

;; clojure/src/clj/clojure/core.clj
(let ~(subvec bindings 0 2) …)  →  (let (subvec bindings 0 2) …)

;; system-fare-utils/base/macros.lisp — lost its unquote, and the loop
;; was then re-laid-out as a binding list
(symbol-macrolet ,(loop …) ,@body)

The stated mechanism was wrong in every particular

BODY_FORMS lives in reindent.rs, whose only consumers are SyntaxTree::reindent and the inspect indentation report. edit format never calls it. The ("do", 2) entry can't affect this, and the theory didn't explain the Common Lisp let repro at all — no do is involved.

What actually happens: Formatter::format_node (formatter/core.rs:848-853) writes a node's reader prefixes before dispatching on its head, so every shape renderer may open its own subject's delimiter freely. Six renderers also open a child's delimiter, and nothing wrote that child's prefix:

• general.rs::format_sequence_list — the binding list of let/do
• general.rs alignment branch / alignable_binding_name — one binding entry
• bindings.rs::format_local_callable_bindings / format_local_callable_binding
• clauses.rs::format_clause / format_body_clause

The corruption is dialect-independent. Had I "fixed" the dialect table as briefed, the CL let case would still corrupt.

Fixed with a shared carries_reader_prefix predicate at those six sites, handing a prefixed child back to format_node. That's also the correct layout answer: `(a b) in a let's binding slot is a quasiquoted datum, not a binding list. It cannot touch prefix-free code. Opaque reader forms (#+sbcl (…)) needed no entry — the parser gives them NodeKind::Atom, so the existing kind != List guard already covers them, verified empirically before assuming it.

Second bug: a prefix before a closing delimiter

(a ') → (a), in every dialect. form() accumulated prefixes, found a close delimiter, and called close_list() without consuming them. It now returns MissingReaderForm, exactly as a prefix at EOF already did — so form and its scanning twin skip_form, which was already refusing this shape, finally agree.

PR #96 pinned this as a stated expectation precisely so a fix would have to update it deliberately rather than silently. That test now asserts the refusal, with an eight-dialect matrix and an (a 'b) control.

Differential: 5353 files, exactly the 27 that were corrupted

formatted output differs between binaries : 27
tree-inequality (BEFORE binary)           : 27
tree-inequality (AFTER  binary)           : 0
output unparseable (BEFORE / AFTER)       : 0 / 0
newly refused by the AFTER binary         : 0

The other 5326 files are byte-identical. The 852 non-zero exits are pre-existing parse failures, unchanged — the parser fix refused nothing new.

The oracle, and why this went unnoticed

Tree equality, added as invariant 6 in tests/corpus.rs (spans excluded — spans are the one thing formatting should change). The negative control is the important part: reverting one guard and re-running over three known-bad files gives

core.clj: formatting changed the tree (29107 in, 29107 out; first difference at
expression 13069: reader_prefixes: [Unquote] became reader_prefixes: [])

…while invariant 3 (idempotence) and the reparse check stay silent. A consistently wrong format is a fixed point.

Worth recording: that oracle cannot see the second bug, because the corruption happens in the parser — input and output parse identically wrong. Only the refusal catches it.

Janet's column-sensitive long strings: deferred, with the reason

Not effort. stringend (parse.c:355-383) sets indent_col = top.column - 1 and strips that many bytes after each newline — but only if every one of them is a space; otherwise reindenting is disabled wholesale. So moving the literal right doesn't shift the value, it flips a mode.

Both candidate fixes are real design decisions: freeze every form containing one (29 of 210 corpus files, and in docstring-heavy Janet that's most defns — also only sound when the form starts at column 0), or teach the strictly top-down column model that an atom can pin its column. Rewriting the literal's interior to compensate is a third option worth rejecting outright: it makes the formatter edit string contents.

Scope measured with Janet itself as oracle (parse-all + deep=): 24 of 210 files changed meaning before, 23 after — one was a bug-1 prefix drop. Of the 23, 22 are pure long-string cases; the 23rd is the separate bare-@ reader divergence already documented in #96.

Verification

cargo build --workspace, cargo test --workspace, cargo test --test cli (3085 passed), cargo fmt --all --check, cargo clippy --all-targets --all-features -- -D warnings — all exit 0, re-run after rebasing onto merged #96.

treefmt formats 6 tracked files (4 lint-scheme-idiom fixtures + 2 migrate/recipes/*.lisp); all 6 are byte-identical under both binaries, so no recipe regeneration is needed and treefmt-pr-check passes. No golden or pinned count moved.

New fixtures include a case for the non-compact path — it preserves prefixes for a different reason than compact_node, so a partial fix would have missed it — plus prefix-on-last-element, ('), nested, and two CLI --write tests through the real binary.