Skip to content

fix(syntax): implement Carp's reader macros and literals - #104

Merged
takeokunn merged 1 commit into
mainfrom
fix/carp-reader-macros
Aug 3, 2026
Merged

fix(syntax): implement Carp's reader macros and literals#104
takeokunn merged 1 commit into
mainfrom
fix/carp-reader-macros

Conversation

@takeokunn

Copy link
Copy Markdown
Collaborator

docs/LanguageGuide.md:170-173 defines & and @ under a literal "Reader Macros" heading — &x ;; same as (ref x), @x ;; same as (copy x) — but Carp routed through classify_legacy, which implements neither.

So @(f x) split into a bare @ atom plus a sibling, inflating the enclosing call's arity in 116 of 248 files (47%), 1493 atoms in all. Byte spans survived, so round-trips were lossless; what broke was structure, which makes any argument-counting analysis unsound for Carp.

Also fixed: @"…" silently split string literals (@ glued to the next token and swallowed the opening quote); character literals unrecognized, so \{ \} \[ \] \( \) \" were read as real delimiters; #"…" Pattern literals unrecognized.

Three more the brief missed, all real: ~ (deref) is also a reader macro (76 glued atoms, 15 files); $[…] static arrays (55 bare $ atoms, same arity-inflating defect); and , is whitespace in Carp, not unquote — 39 sites where the legacy reader gave max and val phantom Unquote prefixes in [min, max, val].

What src/Parsing.hs settles

readerMacro is string macroStr then expr <- sexpr — recursing into sexpr, not atom — so prefixes bind any form, @@x stacks, and @"…" is a copy of a genuine string literal. validCharacters excludes & ~ @ % $ #, so a sigil can never be inside a symbol. aChar is \ then exactly one character with no terminator, so \{ and \ (space) are valid and \ab is char a then symbol b.

Differential over all 248 files

parse failures        6 → 0     (exactly the six the README names)
trees changed              171
child slots removed       1623   (reconciled token-by-token)
files that GAINED atoms      0

Deliberately deferred

%/%@ — Carp's real unquote — needs two new ReaderPrefix variants threaded through 40+ sites and adds findings inside macro templates, so it wants its own FP audit. Leaving it keeps templates inert, which is the suppressing direction. r"…" has zero corpus occurrences.

Ref/Copy/Deref/StaticArray are new variants rather than reuses of Function, whose as_source() is #' and which feeds three real re-emission paths — reusing it would have been latent corruption in place of the old kind.

Honest limitation

No carp binary was obtainable — it was removed from nixpkgs on 2026-02-05 as broken for over a year, and there's no ghc/stack/cabal. So the semantics are grammar-derived from src/Parsing.hs, not oracle-checked: 20 hand-verified cases covering prefix-on-list, stacking, @"a b", all seven delimiter char literals, \space, #"a\"b", $[…], comma, and refusals. Every other dialect fix in this series had a real oracle; this one does not, and that's stated rather than implied.

Rebase: resolved arm by arm

Crossed #96, #98, #100 and #102. The classify_reader_macro table is the trap — ours had Unknown | Hy | Carp with Lfe pulled out by #98; theirs had Unknown | Lfe | Hy with Carp pulled out. Each side's group line still named the other's dialect, so taking either would have silently deleted a dialect's reader support.

Verified by building a pristine origin/main control binary and diffing it against the rebased one over 36 repo Lisp files × 10 dialects plus a 28-case snippet battery: 444 comparisons, 0 differences. Nothing outside Carp changed at all.

The agent also caught that the source worktree's HEAD was b61fe5d, not the commit I named — so a naive git diff HEAD would have swept in all of #95 as spurious content.

Three updates to lint-carp-idiom (#95), one a real test failure

That package was authored before this fix and documents these defects as current behaviour. the_correct_corpus_exercises_the_readers_arity_inflation pinned the bug outright with assert!(bare > 0, "…which the reader splits into a bare sigil atom").

Inverted rather than deleted: it now asserts bare == 0 and prefixed_lists == 8, keeping the corpus-coverage pin it existed for — with the 8 derived from the corpus text (7 &(/@( paren forms plus one &[…]) rather than read off the parser, which then independently agreed.

edit format — confirmed fixed by #100

@(unsafe-nth world i) survives in examples/game_of_life.carp. That was the original agent's top-ranked follow-up; #100's carries_reader_prefix fix is dialect-agnostic, so Carp's new variants were covered automatically. Corpus-wide: format exits 0, output reparses, @ & ~ $ counts unchanged, idempotent.

Verification

cargo build --workspace, cargo test --workspace (13915 passed), cargo test --test cli (3085 passed), cargo fmt --all --check, cargo clippy --all-targets --all-features -- -D warnings — all exit 0. No golden or pinned count moved.

Note for sequencing: #103 (Hy) is open and will pull Hy out of Unknown | Hy in that same dispatch table. Whichever merges second needs the same arm-by-arm treatment.

`docs/LanguageGuide.md:170-173` defines `&` and `@` under a literal
"Reader Macros" heading -- `&x ;; same as (ref x)`, `@x ;; same as
(copy x)` -- but Carp routed through `classify_legacy`, which implements
neither. So `@(f x)` split into a bare `@` atom **plus a sibling**,
inflating the enclosing call's arity in **116 of 248 files (47%)**, 1493
atoms in all. Byte spans survived, so round-trips were lossless; what
broke was structure, which makes any argument-counting analysis unsound
for Carp.

Also fixed: `@"..."` silently split string literals, because `@` glued
to the next token and swallowed the opening quote; character literals
were unrecognized, so `\{ \} \[ \] \( \) \"` were read as real
delimiters; and `#"..."` Pattern literals were unrecognized.

Three more the original brief missed, all real: **`~` (deref) is also a
reader macro** (76 glued atoms in 15 files); **`$[...]` static arrays**
(55 bare `$` atoms, the same arity-inflating defect); and **`,` is
whitespace in Carp, not unquote** (39 sites, where the legacy reader
gave `max` and `val` phantom `Unquote` prefixes in `[min, max, val]`).

`src/Parsing.hs` settles the shape: `readerMacro` is `string macroStr`
then `expr <- sexpr`, recursing into `sexpr` rather than `atom`, so
prefixes bind any form, `@@x` stacks, and `@"..."` is a copy of a
genuine string literal. `validCharacters` excludes `& ~ @ % $ #`, so a
sigil can never be inside a symbol. `aChar` is `\` then one character
with no terminator, so `\{` and `\ ` are valid.

Corpus differential over all 248 files: parse failures **6 -> 0**, 171
trees changed, **1623 child slots removed** and reconciled
token-by-token, **0 files gained atoms**.

`%`/`%@` (Carp's real unquote) is deliberately deferred: it needs two
new `ReaderPrefix` variants threaded through 40+ sites and *adds*
findings inside macro templates, so it wants its own FP audit. Leaving
it keeps templates inert, which is the suppressing direction. `r"..."`
has zero corpus occurrences.

`Ref`/`Copy`/`Deref`/`StaticArray` are new `ReaderPrefix` variants
rather than reuses of `Function`, whose `as_source()` is `#'` and which
feeds three real re-emission paths -- reusing it would have been latent
corruption in place of the old kind.

No `carp` binary was obtainable (removed from nixpkgs 2026-02-05 as
broken; no ghc/stack/cabal), so the semantics are **grammar-derived from
`src/Parsing.hs`, not oracle-checked** -- 20 hand-verified cases. Said
plainly rather than implied.

Rebased across #96, #98, #100 and #102. The `classify_reader_macro`
dispatch table was resolved arm by arm: ours had `Unknown | Hy | Carp`
with `Lfe` pulled out by #98, theirs had `Unknown | Lfe | Hy` with
`Carp` pulled out -- each side's group line still named the other's
dialect, so taking either would have silently deleted a dialect's reader
support. A pristine `origin/main` control binary was built and diffed
over 36 repo files x 10 dialects plus a 28-case battery: **444
comparisons, 0 differences**. Nothing outside Carp changed.

Three things in `lint-carp-idiom` (#95) needed updating, because the
reader change was authored before that package existed and it documents
these defects as *current behaviour*. One was a real test failure:
`the_correct_corpus_exercises_the_readers_arity_inflation` pinned the
bug with `assert!(bare > 0)`. Inverted rather than deleted -- it now
asserts `bare == 0` *and* `prefixed_lists == 8`, keeping the
corpus-coverage pin it existed for, with the 8 derived from the corpus
text rather than read off the parser.

`edit format` on `@(unsafe-nth world i)` is confirmed fixed by #100 --
that was the original agent's top-ranked follow-up, and #100's
`carries_reader_prefix` fix is dialect-agnostic, so Carp's new variants
were covered automatically.
@takeokunn
takeokunn force-pushed the fix/carp-reader-macros branch from df74847 to 55655fd Compare August 3, 2026 15:13
@takeokunn
takeokunn merged commit 623e95b into main Aug 3, 2026
9 of 10 checks passed
@takeokunn
takeokunn deleted the fix/carp-reader-macros branch August 3, 2026 15:27
takeokunn added a commit that referenced this pull request Aug 3, 2026
…alect pin (#108)

`main` was broken by a semantic conflict between two PRs that were each
green alone and merged cleanly.

#106 added `hy_comma_arm_does_not_change_other_dialects`, asserting `,`
still yields `ReaderPrefix::Unquote` in every dialect but Clojure --
including Carp, which at the time inherited the legacy reader where that
was true. #104 then split Carp out and made `,` **whitespace**, which is
what Carp actually does: `[min, max, val]` and `[x Int, y Int]` are
separator syntax, and reading the commas as unquote gave `max` and `val`
phantom prefixes at 39 corpus sites.

Neither PR could have caught this. git saw no textual conflict, both
CI runs were green, and the collision only exists once both are on the
same tree.

Carp moves from the unquote loop into the whitespace loop beside
Clojure, and the test's doc comment now records why -- it previously
said the split-out arm "still serves `Unknown` and `Carp`", which #104
made false.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant