fix(syntax): implement Carp's reader macros and literals - #104
Merged
Conversation
`docs/LanguageGuide.md:170-173` defines `&` and `@` under a literal
"Reader Macros" heading -- `&x ;; same as (ref x)`, `@x ;; same as
(copy x)` -- but Carp routed through `classify_legacy`, which implements
neither. So `@(f x)` split into a bare `@` atom **plus a sibling**,
inflating the enclosing call's arity in **116 of 248 files (47%)**, 1493
atoms in all. Byte spans survived, so round-trips were lossless; what
broke was structure, which makes any argument-counting analysis unsound
for Carp.
Also fixed: `@"..."` silently split string literals, because `@` glued
to the next token and swallowed the opening quote; character literals
were unrecognized, so `\{ \} \[ \] \( \) \"` were read as real
delimiters; and `#"..."` Pattern literals were unrecognized.
Three more the original brief missed, all real: **`~` (deref) is also a
reader macro** (76 glued atoms in 15 files); **`$[...]` static arrays**
(55 bare `$` atoms, the same arity-inflating defect); and **`,` is
whitespace in Carp, not unquote** (39 sites, where the legacy reader
gave `max` and `val` phantom `Unquote` prefixes in `[min, max, val]`).
`src/Parsing.hs` settles the shape: `readerMacro` is `string macroStr`
then `expr <- sexpr`, recursing into `sexpr` rather than `atom`, so
prefixes bind any form, `@@x` stacks, and `@"..."` is a copy of a
genuine string literal. `validCharacters` excludes `& ~ @ % $ #`, so a
sigil can never be inside a symbol. `aChar` is `\` then one character
with no terminator, so `\{` and `\ ` are valid.
Corpus differential over all 248 files: parse failures **6 -> 0**, 171
trees changed, **1623 child slots removed** and reconciled
token-by-token, **0 files gained atoms**.
`%`/`%@` (Carp's real unquote) is deliberately deferred: it needs two
new `ReaderPrefix` variants threaded through 40+ sites and *adds*
findings inside macro templates, so it wants its own FP audit. Leaving
it keeps templates inert, which is the suppressing direction. `r"..."`
has zero corpus occurrences.
`Ref`/`Copy`/`Deref`/`StaticArray` are new `ReaderPrefix` variants
rather than reuses of `Function`, whose `as_source()` is `#'` and which
feeds three real re-emission paths -- reusing it would have been latent
corruption in place of the old kind.
No `carp` binary was obtainable (removed from nixpkgs 2026-02-05 as
broken; no ghc/stack/cabal), so the semantics are **grammar-derived from
`src/Parsing.hs`, not oracle-checked** -- 20 hand-verified cases. Said
plainly rather than implied.
Rebased across #96, #98, #100 and #102. The `classify_reader_macro`
dispatch table was resolved arm by arm: ours had `Unknown | Hy | Carp`
with `Lfe` pulled out by #98, theirs had `Unknown | Lfe | Hy` with
`Carp` pulled out -- each side's group line still named the other's
dialect, so taking either would have silently deleted a dialect's reader
support. A pristine `origin/main` control binary was built and diffed
over 36 repo files x 10 dialects plus a 28-case battery: **444
comparisons, 0 differences**. Nothing outside Carp changed.
Three things in `lint-carp-idiom` (#95) needed updating, because the
reader change was authored before that package existed and it documents
these defects as *current behaviour*. One was a real test failure:
`the_correct_corpus_exercises_the_readers_arity_inflation` pinned the
bug with `assert!(bare > 0)`. Inverted rather than deleted -- it now
asserts `bare == 0` *and* `prefixed_lists == 8`, keeping the
corpus-coverage pin it existed for, with the 8 derived from the corpus
text rather than read off the parser.
`edit format` on `@(unsafe-nth world i)` is confirmed fixed by #100 --
that was the original agent's top-ranked follow-up, and #100's
`carries_reader_prefix` fix is dialect-agnostic, so Carp's new variants
were covered automatically.
takeokunn
force-pushed
the
fix/carp-reader-macros
branch
from
August 3, 2026 15:13
df74847 to
55655fd
Compare
takeokunn
added a commit
that referenced
this pull request
Aug 3, 2026
…alect pin (#108) `main` was broken by a semantic conflict between two PRs that were each green alone and merged cleanly. #106 added `hy_comma_arm_does_not_change_other_dialects`, asserting `,` still yields `ReaderPrefix::Unquote` in every dialect but Clojure -- including Carp, which at the time inherited the legacy reader where that was true. #104 then split Carp out and made `,` **whitespace**, which is what Carp actually does: `[min, max, val]` and `[x Int, y Int]` are separator syntax, and reading the commas as unquote gave `max` and `val` phantom prefixes at 39 corpus sites. Neither PR could have caught this. git saw no textual conflict, both CI runs were green, and the collision only exists once both are on the same tree. Carp moves from the unquote loop into the whitespace loop beside Clojure, and the test's doc comment now records why -- it previously said the split-out arm "still serves `Unknown` and `Carp`", which #104 made false.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs/LanguageGuide.md:170-173defines&and@under a literal "Reader Macros" heading —&x ;; same as (ref x),@x ;; same as (copy x)— but Carp routed throughclassify_legacy, which implements neither.So
@(f x)split into a bare@atom plus a sibling, inflating the enclosing call's arity in 116 of 248 files (47%), 1493 atoms in all. Byte spans survived, so round-trips were lossless; what broke was structure, which makes any argument-counting analysis unsound for Carp.Also fixed:
@"…"silently split string literals (@glued to the next token and swallowed the opening quote); character literals unrecognized, so\{ \} \[ \] \( \) \"were read as real delimiters;#"…"Pattern literals unrecognized.Three more the brief missed, all real:
~(deref) is also a reader macro (76 glued atoms, 15 files);$[…]static arrays (55 bare$atoms, same arity-inflating defect); and,is whitespace in Carp, not unquote — 39 sites where the legacy reader gavemaxandvalphantomUnquoteprefixes in[min, max, val].What
src/Parsing.hssettlesreaderMacroisstring macroStrthenexpr <- sexpr— recursing intosexpr, notatom— so prefixes bind any form,@@xstacks, and@"…"is a copy of a genuine string literal.validCharactersexcludes& ~ @ % $ #, so a sigil can never be inside a symbol.aCharis\then exactly one character with no terminator, so\{and\(space) are valid and\abis charathen symbolb.Differential over all 248 files
Deliberately deferred
%/%@— Carp's real unquote — needs two newReaderPrefixvariants threaded through 40+ sites and adds findings inside macro templates, so it wants its own FP audit. Leaving it keeps templates inert, which is the suppressing direction.r"…"has zero corpus occurrences.Ref/Copy/Deref/StaticArrayare new variants rather than reuses ofFunction, whoseas_source()is#'and which feeds three real re-emission paths — reusing it would have been latent corruption in place of the old kind.Honest limitation
No
carpbinary was obtainable — it was removed from nixpkgs on 2026-02-05 as broken for over a year, and there's no ghc/stack/cabal. So the semantics are grammar-derived fromsrc/Parsing.hs, not oracle-checked: 20 hand-verified cases covering prefix-on-list, stacking,@"a b", all seven delimiter char literals,\space,#"a\"b",$[…], comma, and refusals. Every other dialect fix in this series had a real oracle; this one does not, and that's stated rather than implied.Rebase: resolved arm by arm
Crossed #96, #98, #100 and #102. The
classify_reader_macrotable is the trap — ours hadUnknown | Hy | CarpwithLfepulled out by #98; theirs hadUnknown | Lfe | HywithCarppulled out. Each side's group line still named the other's dialect, so taking either would have silently deleted a dialect's reader support.Verified by building a pristine
origin/maincontrol binary and diffing it against the rebased one over 36 repo Lisp files × 10 dialects plus a 28-case snippet battery: 444 comparisons, 0 differences. Nothing outside Carp changed at all.The agent also caught that the source worktree's HEAD was
b61fe5d, not the commit I named — so a naivegit diff HEADwould have swept in all of #95 as spurious content.Three updates to
lint-carp-idiom(#95), one a real test failureThat package was authored before this fix and documents these defects as current behaviour.
the_correct_corpus_exercises_the_readers_arity_inflationpinned the bug outright withassert!(bare > 0, "…which the reader splits into a bare sigil atom").Inverted rather than deleted: it now asserts
bare == 0andprefixed_lists == 8, keeping the corpus-coverage pin it existed for — with the 8 derived from the corpus text (7&(/@(paren forms plus one&[…]) rather than read off the parser, which then independently agreed.edit format— confirmed fixed by #100@(unsafe-nth world i)survives inexamples/game_of_life.carp. That was the original agent's top-ranked follow-up; #100'scarries_reader_prefixfix is dialect-agnostic, so Carp's new variants were covered automatically. Corpus-wide: format exits 0, output reparses,@ & ~ $counts unchanged, idempotent.Verification
cargo build --workspace,cargo test --workspace(13915 passed),cargo test --test cli(3085 passed),cargo fmt --all --check,cargo clippy --all-targets --all-features -- -D warnings— all exit 0. No golden or pinned count moved.Note for sequencing: #103 (Hy) is open and will pull
Hyout ofUnknown | Hyin that same dispatch table. Whichever merges second needs the same arm-by-arm treatment.