Give a rule a terminator it can only know at match time

.github/workflows/ci.yml

@@ -118,6 +118,9 @@ jobs:
       - name: Validate version metadata
         run: npm run check:versions
 
+      - name: Check translated docs agree
+        run: npm run check:docs-parity
+
       - name: Run tests
         run: npm test
 

CHANGELOG.md

@@ -7,6 +7,65 @@ versioning follows [SemVer](https://semver.org/).
 
 ## [Unreleased]
 
+### Added
+
+- **Literals whose end is decided at runtime.** A grammar rule may now carry
+  `close`, a function that receives the opening match and reports where the
+  form ends. Two builders cover everything the grammars needed: a heredoc ends
+  at the word its own opening line named — `<<<SQL` at SQL — and a
+  delimiter-chosen literal ends at whatever closes the character it opened
+  with, counting depth where brackets nest. Neither is expressible as a
+  RegExp, which is why these rendered as ordinary code until now:
+  - heredoc and nowdoc in **PHP**, including an indented closing word and a
+    `;` after it;
+  - heredoc in **shell**, with `<<-` permitting an indented terminator, and
+    the redirect on an opening line (`cat <<EOF > out.txt`) left as shell
+    rather than pulled into the literal;
+  - heredoc in **Ruby** (`<<~`, `<<-`, plain `<<`), on uppercase words only,
+    so `items << thing` stays the append operator;
+  - `%w[]`, `%i()`, `%q{}`, `%Q<>` and `%r{}` in **Ruby**, `q{}`, `qq{}`,
+    `qw()` and `qr{}` in **Perl**, and `~s{}`, `~w[]`, `~r//` sigils in
+    **Elixir** — each nesting correctly through inner brackets, and each
+    holding a `#` without it becoming a comment.
+
+  An opening whose terminator never arrives reports no match, leaving the text
+  to the rules behind it. A false opening is likelier than a genuinely
+  unterminated literal, and the alternative is a stray `<<` swallowing the
+  rest of the file.
+
+- **`check:docs-parity`.** Every document here is written twice, and nothing
+  checked that the two agreed. The check compares what cannot legitimately
+  differ between translations — versions, paths, package specifiers, links,
+  section structure, table row counts — and leaves prose, sentence counts and
+  translated placeholders alone. It is wired into CI beside `check:versions`.
+
+### Fixed
+
+- **The Chinese docs said less than the English ones.** The Core sync rule
+  merged in #22 existed only in English; the Chinese repository table was
+  headed 状态 and answered a different question than the English column, while
+  the sentence under it explained the English one; and the versioning doc
+  dropped the npm account the package publishes from.
+
+- **The roadmap described jsray-terminal as having a release.** It has none —
+  the only way in is the GitHub install line. It also carried "syncing them to
+  0.0.2-beta.2 is the next step", written when Core was on beta.2 and wrong in
+  two ways since: it named a superseded release, and an integration syncs Core
+  as part of its own release, so being behind between releases is the expected
+  state rather than a queued chore.
+
+### Changed
+
+- **Two deadlines became conditions.** "Revisit at public beta" named
+  2026-07-17; that date passed and the entry became something every planning
+  round rediscovered and re-argued. Minification now carries its answer — a
+  readable, auditable `dist/` is worth more than the 9 KB brotli saves, and
+  the comments are 31% of the file — together with a trigger that cannot
+  expire: a real size complaint, or Core growing substantially. Detection
+  tuning no longer "belongs with the 0.0.2 engine work" while we are inside
+  0.0.2; it needs a beta round of its own, because retuning the scores
+  reorders all 83 grammars at once.
+
 ## [0.0.2-beta.3] — 2026-09-06
 
 Documentation and the guards around it. No runtime change: `dist/` differs from

CONTRIBUTING.md

@@ -155,6 +155,7 @@ Every change lands the same way:
 git checkout -b my-change
 # ... work, then:
 npm test && node tools/check-versions.mjs && node tools/integrity.mjs --check
+npm run check:docs-parity   # if the change touched a document
 git push origin my-change
 gh pr create --fill
 ```
@@ -164,6 +165,11 @@ merge; the branch also has to be up to date with `main`.
 
 - One PR per concern, to keep reviews easy.
 - Engine or grammar changes must come with added / updated tests.
+- Every document here is written twice, as `X.md` and `X.zh-CN.md`. Both
+  translations belong in the same pull request: a rule that exists in one
+  language is a rule half the readers never see, and it reads perfectly well
+  in the language you happen to be checking. `check:docs-parity` fails when a
+  version, path, package specifier, or link appears in only one of a pair.
 - Palette changes should include demo screenshots (both dark and light).
 
 ## Releasing

dist/jsray.js

@@ -23,7 +23,17 @@
    *   { cls: 'tk-xxx',
    *     pattern: /re/,         // must be globalizable; the 'g' flag is forced internally
    *     inside?: rules,        // nested grammar (recursively tokenize captured text)
-   *     lookbehind?: true }    // capture group 1 is consumed as prefix but not colored
+   *     lookbehind?: true,     // capture group 1 is consumed as prefix but not colored
+   *     close?: fn }           // pattern matches the opening only; fn finds the end
+   *
+   * `close(match, text, from) -> index | -1` exists for the forms whose end is
+   * not knowable when the rule is written. A heredoc ends at the word its own
+   * opening line named — `<<<EOT` at EOT, `<<<SQL` at SQL — and `%w[…]` ends
+   * at the bracket matching the one that opened it. No single RegExp can say
+   * that, which is why these forms rendered as ordinary code until now.
+   * Returning -1 means "no terminator here": the opening is left to the rules
+   * behind this one rather than swallowing the rest of the file, because a
+   * false opening is likelier than a genuinely unterminated literal.
    */
   function tokenize(code, rules) {
     let stream = [code];
@@ -43,14 +53,22 @@
         while ((m = re.exec(piece)) !== null) {
           const lbLen = rule.lookbehind && m[1] ? m[1].length : 0;
           const start = m.index + lbLen;
-          const text = m[0].slice(lbLen);
+          let text = m[0].slice(lbLen);
+          if (rule.close) {
+            const end = rule.close(m, piece, m.index + m[0].length);
+            if (end < 0) { re.lastIndex = m.index + 1; continue; }
+            text = piece.slice(start, end);
+          }
           if (!text) { re.lastIndex++; continue; }
           if (start > last) next.push(piece.slice(last, start));
           next.push({
             type: rule.cls,
             content: rule.inside ? tokenize(text, rule.inside) : text,
           });
           last = start + text.length;
+          // The body of a close-delimited form has already been consumed;
+          // resuming inside it would re-match its own contents.
+          if (rule.close) re.lastIndex = last;
         }
         if (last < piece.length) next.push(piece.slice(last));
       }
@@ -74,6 +92,60 @@
 
   const G = {}; // grammars
 
+  // ---------- runtime terminators ----------
+  // Two `close` builders cover every delimited form the grammars below need.
+  // Both take the opening match and report where the form ends.
+
+  /**
+   * A heredoc ends at the word its opening line named. `nameGroup` is the
+   * capture holding that word; `indentGroup` is the capture holding the `-` or
+   * `~` that permits an indented terminator (pass `true` where the language
+   * always permits one, as PHP does since 7.3). `trailing` overrides what may
+   * follow the word on its closing line.
+   *
+   * The name is interpolated into a RegExp, which is only safe because every
+   * opening pattern here restricts it to `[A-Za-z_]\w*` — no metacharacters
+   * can reach this.
+   */
+  function heredocEnd(nameGroup, indentGroup, trailing) {
+    return (m, text, from) => {
+      const name = m[nameGroup];
+      if (!name) return -1;
+      const indented = indentGroup === true ? true : !!m[indentGroup];
+      const re = new RegExp(
+        '^' + (indented ? '[ \\t]*' : '') + name + (trailing || '[ \\t]*$'),
+        'm'
+      );
+      const hit = re.exec(text.slice(from));
+      return hit ? from + hit.index + hit[0].length : -1;
+    };
+  }
+
+  const CLOSERS = { '(': ')', '[': ']', '{': '}', '<': '>' };
+
+  /**
+   * A delimiter-chosen literal — Ruby's `%w[…]`, Perl's `q{…}`, an Elixir
+   * sigil — ends at whatever closes the character it opened with. Bracket
+   * pairs nest; a symmetric delimiter such as `%w!…!` cannot, and counting
+   * depth on one would end the literal at its own opening character.
+   */
+  function pairedEnd(openGroup) {
+    return (m, text, from) => {
+      const open = m[openGroup];
+      if (!open) return -1;
+      const close = CLOSERS[open] || open;
+      const nests = close !== open;
+      let depth = 1;
+      for (let i = from; i < text.length; i++) {
+        const c = text[i];
+        if (c === '\\') { i++; continue; }
+        if (nests && c === open) depth++;
+        else if (c === close && --depth === 0) return i + 1;
+      }
+      return -1;
+    };
+  }
+
   // ---------- shared fragments ----------
   const RX = {
     string1: /"(?:\\.|[^"\\\n])*"/,
@@ -386,6 +458,15 @@
     'docker kubectl python python3 pip pip3 ruby go cargo make brew apt yum';
 
   G.shell = [
+    // Heredocs first: their body may hold quotes and `#`, and every rule
+    // after this one would claim those. The opening line is group 1 and is
+    // consumed as an uncolored prefix, so `cat <<EOF > out.txt` keeps its
+    // redirect as shell rather than dragging it into the literal.
+    { cls: 'tk-string',
+      pattern: /(<<(-?)[ \t]*(['"]?)([A-Za-z_]\w*)\3[^\n]*\n)/,
+      lookbehind: true,
+      close: heredocEnd(4, 2) },
+
     // Strings before comments, else # inside "..." is eaten as a comment.
     // Each `$` form is matched exactly, never by a greedy run that could also
     // swallow the ones after it — the old `\$[\w{][^"\n]*` overlapped itself
@@ -431,6 +512,15 @@
   ).split(' ');
 
   G.php = [
+    // Heredoc and nowdoc ahead of the comment rules: `#` and `//` are
+    // ordinary text inside one. The closing word may be indented and may be
+    // followed by `;` or `,`, which is why the terminator ends at a word
+    // boundary rather than at end of line.
+    { cls: 'tk-string',
+      pattern: /(<<<[ \t]*(['"]?)([A-Za-z_]\w*)\2\r?\n)/,
+      lookbehind: true,
+      close: heredocEnd(3, true, '\\b') },
+
     // Block comments before strings; line comments (// and #) after strings
     // so "https://..." and "#anchor" inside strings never become comments.
     { cls: 'tk-comment', pattern: /\/\*[\s\S]*?\*\// },
@@ -732,12 +822,32 @@
   const RB_BUILTINS = 'puts print p gets raise lambda proc loop each map select reject reduce new'.split(' ');
 
   G.ruby = [
+    // Heredocs, and only with an uppercase word: `<<` is also the append
+    // operator, and `items << thing` must not open one. An uppercase name is
+    // the convention, and where a constant does follow `<<` the terminator
+    // line will not exist, so the form declines itself rather than eating the
+    // file. `<<~` and `<<-` permit an indented terminator; plain `<<` does not.
+    { cls: 'tk-string',
+      pattern: /(<<([-~]?)(['"]?)([A-Z_]\w*)\3[^\n]*\n)/,
+      lookbehind: true,
+      close: heredocEnd(4, 2) },
+
     // `=begin` / `=end` blocks come before the strings that come before
     // everything else. The markers are only special at column zero, so the
     // anchors here are load-bearing rather than decorative. Without this rule
     // a documentation block was read as ordinary code — the body's words came
     // out coloured as function calls and keywords.
     { cls: 'tk-comment', pattern: /^=begin\b[\s\S]*?^=end.*$/m },
+
+    // %w[…] %i(…) %q{…} %Q<…>: the delimiter is picked at the call site, so
+    // the closer is only knowable once the opener has been read, and bracket
+    // pairs nest. %r is a regex, not a string. The bare `%(…)` form is left
+    // out on purpose — it cannot be told from the modulo operator without
+    // parsing, and it is rare enough not to be worth mistaking `a %(b)` for a
+    // literal.
+    { cls: 'tk-regex',  pattern: /%r([([{<|!\/])/, close: pairedEnd(1) },
+    { cls: 'tk-string', pattern: /%[wWiIqQsx]([([{<|!\/])/, close: pairedEnd(1) },
+
     // Strings must come before comments, else # inside "..." (incl. #{} interpolation)
     // is eaten as a comment. String bodies stay single-line so an unpaired quote
     // in a comment can't swallow following lines.
@@ -921,6 +1031,13 @@
         { cls: 'tk-var', pattern: /[$@][A-Za-z_]\w*/ },
     ]},
     { cls: 'tk-string', pattern: /'(?:\\.|[^'\\\n])*'/ },
+
+    // q{…} qq{…} qw{…} qr{…} — ahead of the comment rule, because `#` is
+    // ordinary text inside one. `/` is excluded as a delimiter for the
+    // quoting forms: after a bare word it is far more often division.
+    { cls: 'tk-regex',  pattern: /\bqr[ \t]*([([{<|!\/])/, close: pairedEnd(1) },
+    { cls: 'tk-string', pattern: /\b(?:qq|qw|q)[ \t]*([([{<|!])/, close: pairedEnd(1) },
+
     { cls: 'tk-comment', pattern: /#.*/ },
     { cls: 'tk-regex', pattern: /((?:=~|!~)\s*)(?:m|s|tr|y)?\/(?:\\.|[^/\n])*\/[a-z]*/, lookbehind: true },
     { cls: 'tk-var-builtin', pattern: /\$[_0-9&`'+^!]|\$\^\w|\@ARGV\b|\%ENV\b|\$0\b/ },
@@ -983,6 +1100,14 @@
         { cls: 'tk-operator', pattern: /#\{[^}\n]*\}/ },
     ]},
     { cls: 'tk-string', pattern: /'(?:\\.|[^'\\\n])*'/ },
+
+    // Sigils ~s{…} ~w[…] ~r/…/, ahead of the comment rule for the same reason
+    // the strings are. A `"` delimiter is not accepted here: it would end
+    // `~s"""…"""` at the second quote, and the triple-quote rule above
+    // already renders that form correctly.
+    { cls: 'tk-regex',  pattern: /~[rR]([([{<|\/'])/, close: pairedEnd(1) },
+    { cls: 'tk-string', pattern: /~[a-zA-Z]([([{<|\/'])/, close: pairedEnd(1) },
+
     { cls: 'tk-comment', pattern: /#.*/ },
     { cls: 'tk-decorator', pattern: /@[a-z_]\w*/ },
     { cls: 'tk-var-const', pattern: /:[a-z_]\w*[?!]?/ },

docs/development.md

@@ -403,7 +403,10 @@ deliberately deferred).
   warning when a security-grade Core update ships — security fixes are
   never silently pinnable.
 - **jsray-vscode / jsray-terminal**: both public since 2026-09-03 and
-  2026-09-05, each with a release carrying an installable build. They now
+  2026-09-05. jsray-vscode has a release carrying an installable `.vsix`;
+  jsray-terminal has no release yet, so the only way in is
+  `npm i -g github:jsrayorg/jsray-terminal` — a tag exists to cut one from.
+  They now
   carry what `jsray-wp` gained on 2026-08-26/27, because all three drift the
   same way:
   `tools/sync-core-version.mjs` deriving the README Core badge instead of
@@ -412,32 +415,48 @@ deliberately deferred).
   the right one is what let "Internal test build · no public beta yet" survive
   the whole public beta; and the plugin version ladder (`0.0.1-beta →
   0.0.2-beta`, no counter) rather than Core's. Both bundle Core
-  `0.0.2-beta.1`; syncing them to `0.0.2-beta.2` is the next step.
-
-- **Core**: minification is deliberately absent (zero-build); revisit at
-  public beta.
-- **Literal forms with no rule** (found while auditing for beta.5, deferred
-  because they are absent features rather than wrong spans): heredocs —
-  `<<<EOT` in PHP, `<<~EOT` in Ruby, `<<EOF` in shell — plus Ruby `%w[]` and
-  `%q()`, Perl `q{}` and `qq{}`, and Elixir sigils. Each renders its body as
-  ordinary code today. Haskell's nested `{- {- -} -}` comments close at the
-  first inner terminator and cannot be fixed with a flat pattern at all; they
-  need the same nesting support embedded languages will need.
+  `0.0.2-beta.1`, and stay there until each cuts its own release: an
+  integration syncs Core as part of releasing, not when Core ships. The rule
+  and its one exception are in `docs/projects.md`.
+
+- **Core**: minification is deliberately absent, and that is a decision now
+  rather than a pending one. Stripping comments and indentation takes the
+  brotli transfer from 23.9 KB to 14.8 KB — 9 KB is a real saving, and the
+  comments alone are 31% of the file, written for maintainers and downloaded
+  by every visitor. What it buys against that is a second artifact travelling
+  the whole chain — `integrity.json`, the three bundled snapshots, the
+  `/v/<version>/` paths, the SRI examples in both READMEs — and a new way for
+  a release to ship corrupted. A `dist/` a user can read and audit against its
+  digest is worth more than 9 KB today. Reopen this on a real size complaint
+  or a substantial growth in Core, not at a version number: "revisit at public
+  beta" named 2026-07-17, a date that passed, after which the entry sat here
+  being rediscovered as an open question every planning round.
+- **Nested block comments** are what remains of the literal forms audited for
+  beta.5. Haskell's `{- {- -} -}` closes at the first inner terminator: the
+  outer comment ends early and the rest of the line is read as code. A flat
+  pattern cannot count depth, and `close` does not help here either — the
+  opener carries no information about its own end, which is exactly what a
+  heredoc's does. It needs the nesting support embedded languages will need
+  anyway, so the two belong in one round.
 - **Language detection tuning** (audited for beta.5, deferred): detection is
   correct on all 22 realistic multi-line samples, and correctly returns empty
   for prose, digits and single words rather than guessing. On one-line
   snippets it misreads Go's `func f(x int) int` as Swift and a shell
   `x=1; echo "$x"` as PHP, and returns empty for short C#, Kotlin and TOML.
   Retuning the scores changes the relative ranking of all 83 grammars at
-  once, so it needs its own corpus and belongs with the 0.0.2 engine work —
+  once, so it needs its own corpus and a beta round of its own —
   the failure is mild (usually plain text) and only reachable when the caller
   supplies no language, which the integrations normally do.
 - **Cosmetic, deliberately left**: a literal prefix that sits outside its
   string — `@"…"` and `$"…"` in C#, `r"…"` in Rust, `#"…"#` in Swift, `s"…"`
   in Scala — and the sign in a CSS `-1.5em` or the leading dot in JavaScript's
   `.5`. The literal is coloured; one character in front of it is not.
-- **The string rules themselves** are hand-written once per grammar family
-  without encoding a terminator model, which is what produced every fix in
-  beta.5. Rewriting them onto one builder changes the shape of the objects in
-  `languages`, which is a declared public type, so it waits for the API pass in
-  0.0.2. `tests/constructs.test.mjs` is what guards the behaviour until then.
+- **The string rules themselves** are still hand-written once per grammar
+  family without encoding a terminator model, which is what produced every fix
+  in beta.5. What changed in beta.4 is that a terminator can now be computed
+  at match time (`close`), which is what the delimited forms needed; the rules
+  that were already working were left where they are. Migrating them onto one
+  builder is the remaining half, and it does change the shape of the objects
+  in `languages` — a declared public type, so it belongs in a beta round of
+  its own inside the 0.0.x window rather than riding along with unrelated
+  work. `tests/constructs.test.mjs` is what guards the behaviour until then.