fix(syntax): Janet quote prefix, and Clojure namespaced maps with space by takeokunn · Pull Request #96 · nerima-lisp/paredit-cli

takeokunn · 2026-08-03T10:33:43Z

Two reader gaps in two dialects, each deferred by earlier work with a reproduction already recorded. Independent of the lint-rule PRs.

Janet `'` (quote)

parse.c's root puts ' in the same PFLAG_READERMAC group as , ; ~ |, and popstate expands it to (quote x) — the CL shape, taking exactly one following form. paredit had no arm, so ' glued onto whatever followed: (a '" " b) failed with an unterminated string.

Oracle: a Janet 1.41.3-dev build, compared via parse-all and a node count blind to atom values and struct key order.

	pre	post
files parsing	208/210	210/210
agreeing with Janet	146/210	202/210
trees changed	—	61
regressed (agree → disagree)	—	0

The 8 residuals are all pre-existing: 6 are the bare-@ divergence (Janet reads (@ define X) as the symbol @, and the deficits match the bare-@ counts exactly — 8/8, 3/3, 4/4, 1/1), 2 are the metric's blind spot since Janet dedups struct keys at parse time.

Differential fuzz: 2116/2500 → 2500/2500, with 0 cases rejected by Janet, so no agreement is coincidental. The generator deliberately excludes @, {}, long strings, and any reader-macro byte adjacent to a preceding token — the constructs whose divergence predates this change.

Clojure `#:ns {…}` — the premise was slightly wrong

#:foo{…} already parsed on main. The actual gap is whitespace between the namespace and the brace, which LispReader.java's NamespaceMapReader explicitly allows:

sym = read(r, true, null, false, opts, pendingForms);
nextChar = read1(r);
while(isWhitespace(nextChar))
    nextChar = read1(r);
if(nextChar != '{')
    throw Util.runtimeException("Namespaced map must specify a map");

The failing line in the reported repro file is (println #::it {:a #::it {}}) — spaced. Its own tight #::it{:a 1} lines were fine.

Denominators, and the split differs from the brief: 577 Clojure files, 18 parse failures → 16. Causes: 9 #! shebangs (babashka), 6 #^ old-style metadata (dispatchMacros['^'] = new MetaReader()) — a genuine third gap that wasn't in the brief and is left as a follow-up — 2 namespaced maps, 1 deliberately-unreadable fixture.

No JDK is available here (/usr/bin/java is the macOS stub), so the oracle is Clojure's own reader test suite, test_clojure/reader.cljc:745-771, which asserts #:a{…} ≡ #:a {…}, #::{…} ≡ #:: {…} (two spaces), and that #::: and #: s{:a 1} throw. Shape-level failures 6 → 0.

Design decision: kept the existing MultiDatum model rather than making #:ns a ReaderPrefix — ReaderPrefix is payload-free with a &'static str spelling, and #:foo has no fixed one. Documented, after measuring, that this model makes namespaced maps opaque: #:foo{:a 1} reports 0 atom occurrences, so map-literal rules see nothing inside. That's a pre-existing limitation; this change only widens which maps parse and leaves already-parsing trees byte-identical. (The first draft of that doc comment claimed the opposite and was corrected against the measurement.)

Other dialects unaffected

Pre/post canonical tree dumps, byte-compared: 7879 file-runs, 2 intended differences.

dialect	files	differ
emacs-lisp / scheme / racket / common-lisp / fennel	2502	0
hy / carp / lfe / legacy classifier	4800 runs	0
clojure	577	2 (the intended files)

An initial run showed 6 CL "differences" — that was the dumper panicking on a CL prefix, with only the thread id differing. Made total and re-run.

Lint findings that moved

Janet 304 → 341. Two files lost a leftover-print-debug finding, and both are false positives removed: '(do … (print "there") …) and '(while … (print :hi)) are quoted data, previously read as live calls.
Clojure 350 → 364, purely additive from the two newly-parsing files.
Both repo fixtures lint byte-identically — neither uses ' or #:. No golden or pinned count moved.

Two pre-existing formatter bugs found, reported not hidden

Neither is caused by this change; both reproduce on the pre-change binary.

edit format silently drops reader prefixes on binding/clause arguments. Cleanest repro is Common Lisp: (let (a b) x)→(let (a b) x). BODY_FORMSinreindent.rsapplies a CL-shaped("do", 2)` entry to Janet and Fennel too. Exit 0, output reparses — silent corruption.
A reader prefix immediately before a closing delimiter is dropped, in every dialect. (a ') → (a). Janet refuses the same input. form() accumulates prefixes, hits a close delimiter, and calls close_list() without consuming them. Pinned as a stated expectation so a future fix must update it deliberately.

The two extra Janet divergences from PR #91 — decision

Both are the same single issue, and #91's framing of the first was imprecise. A genuinely dangling macro (;, ,, ~, | alone) was already refused. What #91 saw in y; is not a dangling prefix at all — ; is not an atom boundary, so y; is the single atom y;. Same root cause as a"str": Janet's symchars excludes ' " ; , ~ |, but is_atom_boundary doesn't know it.

Left out deliberately: it's a tokenization change, not a reader-macro-table one, and it's 62 of 210 files, 1064 occurrences (916 of them ") — larger than either bug here, needing its own corpus differential and fuzz.

Verification

cargo build --workspace, cargo test --workspace (13,495 passed), cargo test --test cli (3083 passed), cargo fmt --all --check, cargo clippy --all-targets --all-features -- -D warnings — all exit 0.

Dangling/unterminated behaviour is documented: bare ' at EOF now gives MissingReaderForm, newly agreeing with Janet; #:foo{ unterminated is unchanged; #:foo at EOF, #: {…}, #:foo bar and #:foo ;;c\n{…} are all refused, matching Clojure.

Two reader gaps in two dialects, each deferred by earlier work with a reproduction already recorded. **Janet `'`.** `parse.c`'s `root` puts `'` in the same PFLAG_READERMAC group as `,` `;` `~` `|`, and `popstate` expands it to `(quote x)` -- the CL shape, taking exactly one following form. paredit had no arm, so `'` glued onto whatever followed: `(a '" " b)` failed with an unterminated string. One arm in `classify_janet` fixes it. Oracle, against a Janet 1.41.3-dev build using `parse-all` and a node count blind to atom values and struct key order: files parsing 208/210 -> 210/210 agreeing with Janet 146/210 -> 202/210 trees changed 61 regressed 0 The 8 residuals are all pre-existing: 6 are the bare-`@` divergence (Janet reads `(@ define X)` as the *symbol* `@`, and the deficits match the bare-`@` counts exactly), 2 are the metric's blind spot, since Janet dedups struct keys at parse time. Differential fuzz 2116/2500 -> 2500/2500, with 0 cases rejected by Janet, so no agreement is coincidental. The generator excludes `@`, `{}`, long strings, and any reader-macro byte adjacent to a preceding token -- the constructs whose divergence predates this change. **Clojure `#:ns {...}`.** The premise this started from was slightly wrong: `#:foo{...}` already parsed. The gap is *whitespace* between the namespace and the brace, which `LispReader.java`'s `NamespaceMapReader` explicitly allows -- it reads the symbol, then `while(isWhitespace(...))` before requiring `{`. The failing line in the reported repro file is `(println #::it {:a #::it {}})`; its own tight `#::it{:a 1}` lines were fine. Clojure files: 577 scanned, 18 parse failures -> 16. The causes split 9 `#!` shebangs (babashka), **6 `#^` old-style metadata** -- a real third gap that was not in the brief and is left for a follow-up -- 2 namespaced maps, 1 deliberately-unreadable fixture. No JDK is available here, so the oracle is Clojure's own reader test suite (`test_clojure/reader.cljc:745-771`), which asserts `#:a{...}` is `#:a {...}`, `#::{...}` is `#:: {...}`, and that `#:::` and `#: s{:a 1}` throw. Shape-level failures 6 -> 0. Kept the existing `MultiDatum` model rather than making `#:ns` a `ReaderPrefix`: `ReaderPrefix` is payload-free with a `&'static str` spelling, and `#:foo` has no fixed one. Documented, after measuring, that this model makes namespaced maps **opaque** -- `#:foo{:a 1}` reports 0 atom occurrences, so map-literal rules see nothing inside. That is pre-existing; this change only widens which maps parse. **Other dialects: 7879 file-runs, 2 intended differences.** Pre/post canonical tree dumps byte-compared across emacs-lisp, scheme, racket, common-lisp, fennel, hy, carp, lfe and the legacy classifier. Lint findings that moved: Janet 304 -> 341, and two files *lost* a `leftover-print-debug` finding -- both false positives removed, since `'(do ... (print "there"))` is quoted data that was previously read as a live call. Clojure 350 -> 364, purely additive from the two newly-parsing files. Both repo fixtures lint byte-identically; no golden or pinned count moved. Two pre-existing formatter bugs were found and are reported rather than hidden, neither caused by this change: - `edit format` silently drops reader prefixes on binding/clause arguments. Cleanest repro is Common Lisp: `(let `(a b) x)` becomes `(let (a b) x)`. `BODY_FORMS` in `reindent.rs` applies a CL-shaped `("do", 2)` entry to Janet and Fennel too. - A reader prefix immediately before a closing delimiter is dropped in every dialect: `(a ')` becomes `(a)`. Janet refuses the same input. Pinned as a *stated* expectation so a future fix must update it deliberately.

@Body

`edit format` silently dropped reader prefixes at exit 0, with output that reparses -- the corrupted-Lisp-still-reparses failure mode. On the Clojure standard library it turned (let ~(subvec bindings 0 2) ...) -> (let (subvec bindings 0 2) ...) and in system-fare-utils `(symbol-macrolet ,(loop ...) ,@Body)` lost its unquote, after which the `loop` was re-laid-out as a binding list. **The stated mechanism was wrong in every particular.** `BODY_FORMS` lives in `reindent.rs`, whose only consumers are `SyntaxTree::reindent` and the `inspect indentation` report -- `edit format` never calls it, so the `("do", 2)` entry cannot affect this, and the theory did not explain the Common Lisp `let` repro at all. What actually happens: `Formatter::format_node` writes a node's reader prefixes before dispatching on its head, so a shape renderer may open its **own subject's** delimiter freely. Six renderers also open a *child's* delimiter -- the binding list of `let`/`do`, one binding entry, `flet`/`labels` bindings and each binding, and `cond`/`case` clauses -- and nothing wrote that child's prefix. The corruption is dialect-independent; fixing the dialect table would have left the CL `let` case broken. Fixed with a shared `carries_reader_prefix` predicate at those six sites, handing a prefixed child back to `format_node`. This is also the right *layout* answer: `` `(a b) `` in a `let`'s binding slot is a quasiquoted datum, not a binding list. Opaque reader forms needed no entry -- the parser gives them `NodeKind::Atom`, so the existing guard already covers them, verified before assuming it. Second bug, in the parser: a reader prefix immediately before a closing delimiter was dropped in every dialect -- `(a ')` parsed as `(a)`. `form()` accumulated prefixes, found a close delimiter, and called `close_list()` without consuming them. It now returns `MissingReaderForm`, exactly as a prefix at EOF already did, so `form` and its scanning twin `skip_form` finally agree. PR #96 pinned this as a *stated* expectation precisely so a fix would have to update it deliberately; that test now asserts the refusal. Differential over 5353 deduplicated files from nine dialects: formatted output differs between binaries : 27 tree-inequality (BEFORE binary) : 27 tree-inequality (AFTER binary) : 0 newly refused by the AFTER binary : 0 The 27 changed files are **exactly** the 27 the old binary corrupted; the other 5326 are byte-identical. The oracle is tree equality, added as invariant 6 in `tests/corpus.rs` with spans excluded. Its negative control is the important part: reverting one guard makes it fail with "reader_prefixes: [Unquote] became reader_prefixes: []" while **idempotence and the reparse check stay silent** -- which is why this went unnoticed. That oracle also cannot see the second bug, since input and output parse identically wrong; only the refusal catches it. Janet's column-sensitive long strings are deferred, and the reason is not effort: `stringend` strips `indent_col` bytes after each newline only if all of them are spaces, so moving the literal flips a mode rather than shifting a value. Both candidate fixes are real design decisions -- freezing every form containing one (29 of 210 corpus files, and in docstring-heavy Janet that is most `defn`s), or teaching the strictly top-down column model that an atom can pin its column. Scope measured with Janet itself as oracle: 24 of 210 files changed meaning before, 23 after, of which 22 are pure long-string cases and the 23rd is the separate bare-`@` reader divergence. treefmt formats 6 tracked files; all 6 are byte-identical under both binaries, so no recipe regeneration is needed.

…ngs (#103) * fix(syntax): implement Hy's string prefixes, shebang and bracket strings Hy string prefixes were unimplemented across the board -- `f"..."`, `r"..."`, `b"..."`, `rb"..."`, `t"..."`. The atom scanner ran past the opening quote and stopped at a delimiter *inside* the literal, so `r"a)b"` closed its enclosing list early. f-strings are one instance of this, not the whole bug: it accounts for **417 of 469** baseline parse failures. Two more, both silent: `#!` shebangs were not stripped (393 files parsed at exit 0 with the shebang as two junk atoms), and bracket strings `#[[...]]` were parsed as *code* -- `#[[(defn evil [] 1)]]` yielded a real `defn` node inside a raw string, which is the shape where a lint rule fires on text that is not code. `#[delim[...]delim]` was silent too, not a loud failure as first reported. Measured against HEAD with the same code, over 2824 readable files: **493 failures -> 116**, with 378 newly parsing and 1 newly failing. Every failure class strictly decreased. Against **Hy 1.3.0's own reader** as external oracle, files we refuse that Hy accepts fell from **308 to 15**, with zero new over-refusals; all 15 already failed at HEAD with byte-identical errors. The one newly-failing file is a true positive -- real Hy refuses it too (`LexException: invalid string prefix`). It previously "passed" only because the f-string was never scanned. f-strings are modelled as **one opaque atom**, deliberately. `read_fcomponent` calls `parse_one_form`, so the interpolations are arbitrary code and finding the closing quote needs a sub-reader -- `f"{(str "}")}"` is legal. But `ExpressionKind` is only Root/List/Atom, so interleaved text-and-forms needs a new node kind rippling through the formatter, edit engine and every rule; and children are *editable*, so the formatter would reindent inside the literal segments between interpolations. Exposing them trades the `#[[...]]` silent-corruption class for a new one. Blind beats wrong, and today the whole file is invisible. A surprise worth recording: **format specs are nearly unreachable.** `:` is not in `NON_IDENT`, so `f"{x:>10}"` interpolates the *symbol* `x:>10`, and `f"{x:>{width}}"` is a LexException. You need `f"{x :>10}"`. `~` as a reader prefix is deliberately **not** included, though it was implemented and measured. Several formatter paths open a child list without writing that child's prefixes, so a prefixed list is emitted with its prefix deleted -- making `~` a prefix aimed that at the most common construct in Hy macro code, and `edit format` newly changed the meaning of 14 files. PR #100 has since fixed that family; `~` is the obvious next step, and `hy_unquote_is_not_yet_a_reader_prefix` pins the known-wrong reading so whoever does it finds the reasoning. Other dialects: **42142 file-parses, zero differences**, full-tree hash pre- vs post-change across all nine plus the permissive `Unknown` reader. A unit test pins `has_prefixed_strings`/`has_bracket_strings` false and `#[[a]]` as a *list* for all ten. Rebased across #96, #98 and #100, all of which edit the same files. The `classify_reader_macro` dispatch table was resolved arm by arm rather than by side: base had the group `Unknown | Lfe | Hy | Carp`, #98 pulled `Lfe` out and this change pulls `Hy` out, so each side's group line still named the other's dialect. Taking either side would have silently deleted a dialect's reader support. A 29-case probe confirms every recent PR's behaviour survives -- Janet's `'`, Clojure's spaced `#:ns {...}`, LFE's `#B(`/`#M(`/`#S(`/`#"`/`#\`/`|...|`, and #100's `MissingReaderForm` on `(a ')`. One pre-existing issue found and not touched: 13 of the 15 remaining over-refusals are the `,`-as-`Unquote` divergence colliding with #100. Hy's empty tuple is `(,)`, and a trailing `,` before `}` is common, so those are now hard refusals. `classify_hy`'s comment claims `,` "occurs at a token start essentially never" -- the corpus says 13 files, including `hylang/simalq`, `kanaka/mal` and Hy's own `contrib/walk.hy`. Live on main today, and the obvious next Hy item. * perf(syntax): keep the Hy string check off the other dialects' parse path CI's bench gate measured `parse-scaling/reader-conditional` +11.7% at 1MiB and +10.6% at 8MiB. Both sizes moving together by the same amount is the real-regression signature, not the `edit-all-loop` measurement artifact -- and unlike that one, this branch genuinely does touch the parser's inner loop. `at_hy_string` is called once per *form* from `parse_form`'s match and again from the discarded-form scanner. Its dialect test sat inside `hy_string_prefix_width`, one call deep, with a second `has_bracket_strings()` test after it -- so for the other nine dialects it was two guarded calls and two byte reads per form rather than one comparison against a loop-invariant field. Hoisted the dialect gate to the top and marked the function `#[inline]`. On an 8MiB Common Lisp document that is millions of calls that can only ever answer `false`. This is the ordering the Carp reader change adopted deliberately for the same reason: `self.dialect` is loop-invariant across the per-byte and per-form calls, so testing it first lets the rest fold away.

`docs/LanguageGuide.md:170-173` defines `&` and `@` under a literal "Reader Macros" heading -- `&x ;; same as (ref x)`, `@x ;; same as (copy x)` -- but Carp routed through `classify_legacy`, which implements neither. So `@(f x)` split into a bare `@` atom **plus a sibling**, inflating the enclosing call's arity in **116 of 248 files (47%)**, 1493 atoms in all. Byte spans survived, so round-trips were lossless; what broke was structure, which makes any argument-counting analysis unsound for Carp. Also fixed: `@"..."` silently split string literals, because `@` glued to the next token and swallowed the opening quote; character literals were unrecognized, so `\{ \} \[ \]  \"` were read as real delimiters; and `#"..."` Pattern literals were unrecognized. Three more the original brief missed, all real: **`~` (deref) is also a reader macro** (76 glued atoms in 15 files); **`$[...]` static arrays** (55 bare `$` atoms, the same arity-inflating defect); and **`,` is whitespace in Carp, not unquote** (39 sites, where the legacy reader gave `max` and `val` phantom `Unquote` prefixes in `[min, max, val]`). `src/Parsing.hs` settles the shape: `readerMacro` is `string macroStr` then `expr <- sexpr`, recursing into `sexpr` rather than `atom`, so prefixes bind any form, `@@x` stacks, and `@"..."` is a copy of a genuine string literal. `validCharacters` excludes `& ~ @ % $ #`, so a sigil can never be inside a symbol. `aChar` is `\` then one character with no terminator, so `\{` and `\ ` are valid. Corpus differential over all 248 files: parse failures **6 -> 0**, 171 trees changed, **1623 child slots removed** and reconciled token-by-token, **0 files gained atoms**. `%`/`%@` (Carp's real unquote) is deliberately deferred: it needs two new `ReaderPrefix` variants threaded through 40+ sites and *adds* findings inside macro templates, so it wants its own FP audit. Leaving it keeps templates inert, which is the suppressing direction. `r"..."` has zero corpus occurrences. `Ref`/`Copy`/`Deref`/`StaticArray` are new `ReaderPrefix` variants rather than reuses of `Function`, whose `as_source()` is `#'` and which feeds three real re-emission paths -- reusing it would have been latent corruption in place of the old kind. No `carp` binary was obtainable (removed from nixpkgs 2026-02-05 as broken; no ghc/stack/cabal), so the semantics are **grammar-derived from `src/Parsing.hs`, not oracle-checked** -- 20 hand-verified cases. Said plainly rather than implied. Rebased across #96, #98, #100 and #102. The `classify_reader_macro` dispatch table was resolved arm by arm: ours had `Unknown | Hy | Carp` with `Lfe` pulled out by #98, theirs had `Unknown | Lfe | Hy` with `Carp` pulled out -- each side's group line still named the other's dialect, so taking either would have silently deleted a dialect's reader support. A pristine `origin/main` control binary was built and diffed over 36 repo files x 10 dialects plus a 28-case battery: **444 comparisons, 0 differences**. Nothing outside Carp changed. Three things in `lint-carp-idiom` (#95) needed updating, because the reader change was authored before that package existed and it documents these defects as *current behaviour*. One was a real test failure: `the_correct_corpus_exercises_the_readers_arity_inflation` pinned the bug with `assert!(bare > 0)`. Inverted rather than deleted -- it now asserts `bare == 0` *and* `prefixed_lists == 8`, keeping the corpus-coverage pin it existed for, with the 8 derived from the corpus text rather than read off the parser. `edit format` on `@(unsafe-nth world i)` is confirmed fixed by #100 -- that was the original agent's top-ranked follow-up, and #100's `carries_reader_prefix` fix is dialect-agnostic, so Carp's new variants were covered automatically.

takeokunn merged commit d3e8266 into main Aug 3, 2026
10 checks passed

takeokunn deleted the fix/janet-quote-clojure-nsmap branch August 3, 2026 11:01

This was referenced Aug 3, 2026

fix(syntax): implement LFE's hash-dispatch forms and quoted symbols #98

Merged

fix(syntax): stop the formatter dropping reader prefixes #100

Merged

This was referenced Aug 3, 2026

fix(syntax): implement Hy's string prefixes, shebang and bracket strings #103

Merged

fix(syntax): implement Carp's reader macros and literals #104

Merged

Navigation Menu

Uh oh!

fix(syntax): Janet quote prefix, and Clojure namespaced maps with space - #96

fix(syntax): Janet quote prefix, and Clojure namespaced maps with space#96
takeokunn merged 1 commit into
mainfrom
fix/janet-quote-clojure-nsmap

takeokunn commented Aug 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

takeokunn commented Aug 3, 2026

Janet ' (quote)

Clojure #:ns {…} — the premise was slightly wrong

Other dialects unaffected

Lint findings that moved

Two pre-existing formatter bugs found, reported not hidden

The two extra Janet divergences from PR #91 — decision

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Janet `'` (quote)

Clojure `#:ns {…}` — the premise was slightly wrong