fix(syntax): implement LFE's hash-dispatch forms and quoted symbols - #98
Merged
Conversation
Four reader defects, three of them silent. The headline is not the
parser: **the pre-change formatter silently corrupted 283 of 2600 LFE
files at exit 0**, rewriting `#M()` as `#M ()`, which LFE cannot scan at
all (`{illegal_token,"#M"}`). Round-trip idempotence passed on the
broken binary -- 0 refusals, 0 non-reparsing outputs, 0 non-idempotent.
Only LFE's own token stream as an external oracle caught it:
LFE reads the formatted output as the same program
pre 2317 / 2600 post 2674 / 2674
pre 283 DIFFERENT post 0
The defects, measured over 2735 .lfe files from 236 repos:
- `#B(` / `#M(` / `#S(` orphaned from their list, inflating arity at
exit 0. `#M(` is 336 files (12.3%); `#B(` is only 27. The brief's
combined figure was the right order but the wrong composition.
- **`#"..."` binary strings, not in the brief at all, and the biggest
one** -- 307 files (11.2%), 139 with whitespace or a delimiter
inside. `#"text/plain; version=0.0.4"` split into three atoms;
`#" a"` desynchronised the rest of the file into giant garbage
atoms; `#"a)b"` closed its enclosing list early.
- `#\c` character literals -- 84 files, not the 1 reported.
- `|quoted atoms|` -- 44 files, only 29 actually splitting. The brief's
95 was ~3x too high.
LFE 2.2.0 has **no leex file**; `src/lfe_scan.erl` is hand-written. The
complete `#`-dispatch set is read from `scan_hash1`/`scan_hash2`. Three
findings that shaped the code:
- `#B(` is its own token, not a prefix -- `spell1` reads
`sexpr -> '#B(' proper_list ')'` -- but it is structurally identical
to `#(`, so it is modelled the same way.
- `#S(` is scanned and then **unparseable**: `spell1` declares the
terminal but has no production using it, and LFE answers
`{illegal,'#S('}`. Lexed anyway, because orphaning `#S` in an
already-broken file is worse.
- `|` opens a quoted symbol only at a token's first byte
(`start_symbol_char($|)` is false but `symbol_char` falls through to
the printable range), so `a|b c|d` is two symbols. Reusing the CL
"anywhere" rule would have fused them; hence `BarQuoting::{None,
Anywhere,TokenStart}`.
`$c` is **not** LFE -- `$` is an ordinary symbol char -- so the `$(`
concern in the brief does not apply.
`#B(` deliberately does not reuse `HashLiteral`, whose `as_source()` is
`"#"`: that would have made `inline-function`'s renderer turn a binary
into a tuple. New silent corruption in place of old.
Differential over the 2639 files both builds parse, aligned by byte span
rather than position: 2667 list forms in 394 files changed child count,
3241 phantom nodes gone. Nine forms *gained* children and all nine were
inspected -- each correct, e.g. a `#"..."` that had been closing its
`defun` early. Parse failures 96 -> 22, with 74 newly parsing and 0
newly failing; 6 of the 22 are files LFE's own parser also rejects.
Against LFE's scanner directly: 2113/2549 -> 2540/2549. Differential
fuzz 670/2500 -> 2500/2500, with zero "both reject" cases, so no
agreement is coincidental.
Other dialects: **76164 parses, zero differences** across CL, Elisp,
Scheme, Racket, Clojure, Fennel, Janet, Hy, Carp and the legacy reader.
A test pins `bar_quoting` and `character_literal_is_exactly_one_char`
per dialect against `Dialect::ALL.len()`, so adding a dialect forces a
decision.
`lfe-catch-swallows-exit` findings are unmoved (1542 = 1542, zero
pre-only, zero post-only). The +41 in the whole-corpus run comes
entirely from the 74 files that previously could not be linted at all.
No golden or pinned count moved, because the repo's LFE fixture contains
none of these constructs -- that fixture is thin, and left alone rather
than moving several pinned counts for coverage the unit tests give
precisely.
Deferred and reported: triple-quoted strings (`scan_tq_string`, LFE 2.2)
-- 1 file; `"` not being an atom boundary, so `%"%"` is one atom where
LFE sees two forms -- cross-dialect, 2 files; improper-list `.` read as
an ordinary atom child -- 140 files.
One normalisation documented on the enum rather than left to be
discovered: `as_source()` spells these upper case (`#B`), matching LFE's
own printer, so inlining a macro body containing `#b(...)` returns
`#B(...)`. Same literal to LFE's reader.
This was referenced Aug 3, 2026
takeokunn
added a commit
that referenced
this pull request
Aug 3, 2026
…ngs (#103) * fix(syntax): implement Hy's string prefixes, shebang and bracket strings Hy string prefixes were unimplemented across the board -- `f"..."`, `r"..."`, `b"..."`, `rb"..."`, `t"..."`. The atom scanner ran past the opening quote and stopped at a delimiter *inside* the literal, so `r"a)b"` closed its enclosing list early. f-strings are one instance of this, not the whole bug: it accounts for **417 of 469** baseline parse failures. Two more, both silent: `#!` shebangs were not stripped (393 files parsed at exit 0 with the shebang as two junk atoms), and bracket strings `#[[...]]` were parsed as *code* -- `#[[(defn evil [] 1)]]` yielded a real `defn` node inside a raw string, which is the shape where a lint rule fires on text that is not code. `#[delim[...]delim]` was silent too, not a loud failure as first reported. Measured against HEAD with the same code, over 2824 readable files: **493 failures -> 116**, with 378 newly parsing and 1 newly failing. Every failure class strictly decreased. Against **Hy 1.3.0's own reader** as external oracle, files we refuse that Hy accepts fell from **308 to 15**, with zero new over-refusals; all 15 already failed at HEAD with byte-identical errors. The one newly-failing file is a true positive -- real Hy refuses it too (`LexException: invalid string prefix`). It previously "passed" only because the f-string was never scanned. f-strings are modelled as **one opaque atom**, deliberately. `read_fcomponent` calls `parse_one_form`, so the interpolations are arbitrary code and finding the closing quote needs a sub-reader -- `f"{(str "}")}"` is legal. But `ExpressionKind` is only Root/List/Atom, so interleaved text-and-forms needs a new node kind rippling through the formatter, edit engine and every rule; and children are *editable*, so the formatter would reindent inside the literal segments between interpolations. Exposing them trades the `#[[...]]` silent-corruption class for a new one. Blind beats wrong, and today the whole file is invisible. A surprise worth recording: **format specs are nearly unreachable.** `:` is not in `NON_IDENT`, so `f"{x:>10}"` interpolates the *symbol* `x:>10`, and `f"{x:>{width}}"` is a LexException. You need `f"{x :>10}"`. `~` as a reader prefix is deliberately **not** included, though it was implemented and measured. Several formatter paths open a child list without writing that child's prefixes, so a prefixed list is emitted with its prefix deleted -- making `~` a prefix aimed that at the most common construct in Hy macro code, and `edit format` newly changed the meaning of 14 files. PR #100 has since fixed that family; `~` is the obvious next step, and `hy_unquote_is_not_yet_a_reader_prefix` pins the known-wrong reading so whoever does it finds the reasoning. Other dialects: **42142 file-parses, zero differences**, full-tree hash pre- vs post-change across all nine plus the permissive `Unknown` reader. A unit test pins `has_prefixed_strings`/`has_bracket_strings` false and `#[[a]]` as a *list* for all ten. Rebased across #96, #98 and #100, all of which edit the same files. The `classify_reader_macro` dispatch table was resolved arm by arm rather than by side: base had the group `Unknown | Lfe | Hy | Carp`, #98 pulled `Lfe` out and this change pulls `Hy` out, so each side's group line still named the other's dialect. Taking either side would have silently deleted a dialect's reader support. A 29-case probe confirms every recent PR's behaviour survives -- Janet's `'`, Clojure's spaced `#:ns {...}`, LFE's `#B(`/`#M(`/`#S(`/`#"`/`#\`/`|...|`, and #100's `MissingReaderForm` on `(a ')`. One pre-existing issue found and not touched: 13 of the 15 remaining over-refusals are the `,`-as-`Unquote` divergence colliding with #100. Hy's empty tuple is `(,)`, and a trailing `,` before `}` is common, so those are now hard refusals. `classify_hy`'s comment claims `,` "occurs at a token start essentially never" -- the corpus says 13 files, including `hylang/simalq`, `kanaka/mal` and Hy's own `contrib/walk.hy`. Live on main today, and the obvious next Hy item. * perf(syntax): keep the Hy string check off the other dialects' parse path CI's bench gate measured `parse-scaling/reader-conditional` +11.7% at 1MiB and +10.6% at 8MiB. Both sizes moving together by the same amount is the real-regression signature, not the `edit-all-loop` measurement artifact -- and unlike that one, this branch genuinely does touch the parser's inner loop. `at_hy_string` is called once per *form* from `parse_form`'s match and again from the discarded-form scanner. Its dialect test sat inside `hy_string_prefix_width`, one call deep, with a second `has_bracket_strings()` test after it -- so for the other nine dialects it was two guarded calls and two byte reads per form rather than one comparison against a loop-invariant field. Hoisted the dialect gate to the top and marked the function `#[inline]`. On an 8MiB Common Lisp document that is millions of calls that can only ever answer `false`. This is the ordering the Carp reader change adopted deliberately for the same reason: `self.dialect` is loop-invariant across the per-byte and per-form calls, so testing it first lets the rest fold away.
takeokunn
added a commit
that referenced
this pull request
Aug 3, 2026
`docs/LanguageGuide.md:170-173` defines `&` and `@` under a literal
"Reader Macros" heading -- `&x ;; same as (ref x)`, `@x ;; same as
(copy x)` -- but Carp routed through `classify_legacy`, which implements
neither. So `@(f x)` split into a bare `@` atom **plus a sibling**,
inflating the enclosing call's arity in **116 of 248 files (47%)**, 1493
atoms in all. Byte spans survived, so round-trips were lossless; what
broke was structure, which makes any argument-counting analysis unsound
for Carp.
Also fixed: `@"..."` silently split string literals, because `@` glued
to the next token and swallowed the opening quote; character literals
were unrecognized, so `\{ \} \[ \] \( \) \"` were read as real
delimiters; and `#"..."` Pattern literals were unrecognized.
Three more the original brief missed, all real: **`~` (deref) is also a
reader macro** (76 glued atoms in 15 files); **`$[...]` static arrays**
(55 bare `$` atoms, the same arity-inflating defect); and **`,` is
whitespace in Carp, not unquote** (39 sites, where the legacy reader
gave `max` and `val` phantom `Unquote` prefixes in `[min, max, val]`).
`src/Parsing.hs` settles the shape: `readerMacro` is `string macroStr`
then `expr <- sexpr`, recursing into `sexpr` rather than `atom`, so
prefixes bind any form, `@@x` stacks, and `@"..."` is a copy of a
genuine string literal. `validCharacters` excludes `& ~ @ % $ #`, so a
sigil can never be inside a symbol. `aChar` is `\` then one character
with no terminator, so `\{` and `\ ` are valid.
Corpus differential over all 248 files: parse failures **6 -> 0**, 171
trees changed, **1623 child slots removed** and reconciled
token-by-token, **0 files gained atoms**.
`%`/`%@` (Carp's real unquote) is deliberately deferred: it needs two
new `ReaderPrefix` variants threaded through 40+ sites and *adds*
findings inside macro templates, so it wants its own FP audit. Leaving
it keeps templates inert, which is the suppressing direction. `r"..."`
has zero corpus occurrences.
`Ref`/`Copy`/`Deref`/`StaticArray` are new `ReaderPrefix` variants
rather than reuses of `Function`, whose `as_source()` is `#'` and which
feeds three real re-emission paths -- reusing it would have been latent
corruption in place of the old kind.
No `carp` binary was obtainable (removed from nixpkgs 2026-02-05 as
broken; no ghc/stack/cabal), so the semantics are **grammar-derived from
`src/Parsing.hs`, not oracle-checked** -- 20 hand-verified cases. Said
plainly rather than implied.
Rebased across #96, #98, #100 and #102. The `classify_reader_macro`
dispatch table was resolved arm by arm: ours had `Unknown | Hy | Carp`
with `Lfe` pulled out by #98, theirs had `Unknown | Lfe | Hy` with
`Carp` pulled out -- each side's group line still named the other's
dialect, so taking either would have silently deleted a dialect's reader
support. A pristine `origin/main` control binary was built and diffed
over 36 repo files x 10 dialects plus a 28-case battery: **444
comparisons, 0 differences**. Nothing outside Carp changed.
Three things in `lint-carp-idiom` (#95) needed updating, because the
reader change was authored before that package existed and it documents
these defects as *current behaviour*. One was a real test failure:
`the_correct_corpus_exercises_the_readers_arity_inflation` pinned the
bug with `assert!(bare > 0)`. Inverted rather than deleted -- it now
asserts `bare == 0` *and* `prefixed_lists == 8`, keeping the
corpus-coverage pin it existed for, with the 8 derived from the corpus
text rather than read off the parser.
`edit format` on `@(unsafe-nth world i)` is confirmed fixed by #100 --
that was the original agent's top-ranked follow-up, and #100's
`carries_reader_prefix` fix is dialect-agnostic, so Carp's new variants
were covered automatically.
takeokunn
added a commit
that referenced
this pull request
Aug 3, 2026
`docs/LanguageGuide.md:170-173` defines `&` and `@` under a literal
"Reader Macros" heading -- `&x ;; same as (ref x)`, `@x ;; same as
(copy x)` -- but Carp routed through `classify_legacy`, which implements
neither. So `@(f x)` split into a bare `@` atom **plus a sibling**,
inflating the enclosing call's arity in **116 of 248 files (47%)**, 1493
atoms in all. Byte spans survived, so round-trips were lossless; what
broke was structure, which makes any argument-counting analysis unsound
for Carp.
Also fixed: `@"..."` silently split string literals, because `@` glued
to the next token and swallowed the opening quote; character literals
were unrecognized, so `\{ \} \[ \] \( \) \"` were read as real
delimiters; and `#"..."` Pattern literals were unrecognized.
Three more the original brief missed, all real: **`~` (deref) is also a
reader macro** (76 glued atoms in 15 files); **`$[...]` static arrays**
(55 bare `$` atoms, the same arity-inflating defect); and **`,` is
whitespace in Carp, not unquote** (39 sites, where the legacy reader
gave `max` and `val` phantom `Unquote` prefixes in `[min, max, val]`).
`src/Parsing.hs` settles the shape: `readerMacro` is `string macroStr`
then `expr <- sexpr`, recursing into `sexpr` rather than `atom`, so
prefixes bind any form, `@@x` stacks, and `@"..."` is a copy of a
genuine string literal. `validCharacters` excludes `& ~ @ % $ #`, so a
sigil can never be inside a symbol. `aChar` is `\` then one character
with no terminator, so `\{` and `\ ` are valid.
Corpus differential over all 248 files: parse failures **6 -> 0**, 171
trees changed, **1623 child slots removed** and reconciled
token-by-token, **0 files gained atoms**.
`%`/`%@` (Carp's real unquote) is deliberately deferred: it needs two
new `ReaderPrefix` variants threaded through 40+ sites and *adds*
findings inside macro templates, so it wants its own FP audit. Leaving
it keeps templates inert, which is the suppressing direction. `r"..."`
has zero corpus occurrences.
`Ref`/`Copy`/`Deref`/`StaticArray` are new `ReaderPrefix` variants
rather than reuses of `Function`, whose `as_source()` is `#'` and which
feeds three real re-emission paths -- reusing it would have been latent
corruption in place of the old kind.
No `carp` binary was obtainable (removed from nixpkgs 2026-02-05 as
broken; no ghc/stack/cabal), so the semantics are **grammar-derived from
`src/Parsing.hs`, not oracle-checked** -- 20 hand-verified cases. Said
plainly rather than implied.
Rebased across #96, #98, #100 and #102. The `classify_reader_macro`
dispatch table was resolved arm by arm: ours had `Unknown | Hy | Carp`
with `Lfe` pulled out by #98, theirs had `Unknown | Lfe | Hy` with
`Carp` pulled out -- each side's group line still named the other's
dialect, so taking either would have silently deleted a dialect's reader
support. A pristine `origin/main` control binary was built and diffed
over 36 repo files x 10 dialects plus a 28-case battery: **444
comparisons, 0 differences**. Nothing outside Carp changed.
Three things in `lint-carp-idiom` (#95) needed updating, because the
reader change was authored before that package existed and it documents
these defects as *current behaviour*. One was a real test failure:
`the_correct_corpus_exercises_the_readers_arity_inflation` pinned the
bug with `assert!(bare > 0)`. Inverted rather than deleted -- it now
asserts `bare == 0` *and* `prefixed_lists == 8`, keeping the
corpus-coverage pin it existed for, with the 8 derived from the corpus
text rather than read off the parser.
`edit format` on `@(unsafe-nth world i)` is confirmed fixed by #100 --
that was the original agent's top-ranked follow-up, and #100's
`carries_reader_prefix` fix is dialect-agnostic, so Carp's new variants
were covered automatically.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four reader defects, three of them silent. The headline is not the parser.
The pre-change formatter silently corrupted 283 of 2600 LFE files at exit 0
It rewrote
#M()as#M (), which LFE cannot scan at all ({illegal_token,"#M"}).Round-trip properties passed on the broken binary — 0 refusals, 0 non-reparsing outputs, 0 non-idempotent, 0 self-structure changes. Exactly the fixed point that makes idempotence worthless as an oracle. Using LFE's own token stream instead:
treefmtrunsparedit edit formatover a repo's Lisp files, so this is a corruption vector, not a cosmetic bug.The defects, measured over 2735
.lfefiles from 236 repos#M(#"…"binary string#\#B(#B+#M= 243 combined|…|Three corrections to my brief: the
#B/#Mfigure was the right order but almost entirely#M;#\was far too low;|…|was ~3× too high. And the biggest LFE reader defect wasn't in the brief at all —#"text/plain; version=0.0.4"split into three atoms,#" a"desynchronised the rest of the file into giant garbage atoms, and#"a)b"closed its enclosing list early.What LFE's scanner actually specifies
It is not
lfe_scan.xrl— LFE 2.2.0 has no leex file;src/lfe_scan.erlis hand-written. The complete#-dispatch set was read fromscan_hash1/scan_hash2:#(·#B(/#b(·#M(/#m(·#S(/#s(·#"…"·#\C·#'f/2·#.·#|·#*·#b/#o/#d/#x·#<digits>r·#`·#;·#,·#,@Three findings that shaped the code:
#B(is its own token, not a prefix —spell1readssexpr -> '#B(' proper_list ')'— but structurally identical to#(, so modelled the same way.#S(is scanned and then unparseable:spell1declares the terminal but has no production using it, and LFE answers{illegal,'#S('}. Lexed anyway, because orphaning#Sin an already-broken file is worse.|opens a quoted symbol only at a token's first byte —start_symbol_char($|)is false butsymbol_charfalls through to the printable range — soa|b c|dis two symbols. Reusing the CL "anywhere" rule would have fused them; henceBarQuoting::{None, Anywhere, TokenStart}.$cis not LFE ($is an ordinary symbol char), so the$(concern in my brief doesn't apply. And#\cis exactly one character with no named forms:lfe_scan:string("#\\newline")→[{number,1,110},{symbol,1,ewline}].#B(deliberately does not reuseHashLiteral, whoseas_source()is"#"— that would makeinline-function's renderer turn a binary into a tuple. New silent corruption in place of old.Differential
Aligned by byte span rather than position, over the 2639 files both builds parse:
#"…"that had been closing itsdefunearly (9→15 children)Against LFE's scanner directly: 2113/2549 → 2540/2549. Differential fuzz 2500 cases: 670 → 2500 agreeing, with zero "both reject" cases, so no agreement is coincidental.
Other dialects: 76,164 parses, zero differences
CL 13,090 · Elisp 10,101 · Scheme 12,002 · Racket 208 · Clojure 1,709 · Fennel 236 · Janet 279 · Hy 119 · Carp 248 · legacy 38,172. A test pins
bar_quotingandcharacter_literal_is_exactly_one_charper dialect againstDialect::ALL.len(), so adding a dialect forces a decision.Lint findings
lfe-catch-swallows-exitis unmoved — 1542 = 1542, zero pre-only, zero post-only, at the individual-finding level. The +41 in the whole-corpus run comes entirely from the 74 files that previously could not be linted at all;partial_failures96 → 22.No golden or pinned count moved, because
tests/fixtures/corpus/lfe.lfecontains none of#B(,#M(,#",#\or|…|. That fixture is thin for LFE; left alone rather than moving several pinned counts for coverage the unit tests give precisely.Deferred, reported not hidden
Triple-quoted strings (
scan_tq_string, LFE 2.2) — 1 file still fails."not being an atom boundary, so%"%"is one atom where LFE sees symbol%+ string — cross-dialect, byte-identical pre and post, 2 files. Improper-list.read as an ordinary atom child — 140 files.One normalisation documented on the enum rather than left to be discovered:
as_source()spells these upper case (#B), matching LFE's own printer inlfe_io_write.erl, so inlining a macro body containing#b(…)returns#B(…). Same literal to LFE's reader.Verification
cargo build --workspace,cargo test --workspace,cargo test --test cli(3083 passed),cargo fmt --all --check,cargo clippy --all-targets --all-features -- -D warnings— all exit 0, re-run after rebasing onto #96.The repo's own 38 Lisp files format byte-identically pre and post, so no treefmt regeneration is needed.
Two intermediate runs failed on wall-clock assertions at load average 87 and were settled rather than assumed:
lint-introspection's ratio test was run interleaved under matched load and pristineorigin/mainfailed 6/14 against this tree's 6/14, with the failing rule differing each run — which a real regression would not do. (That test is now split by #94.)