jq: a raw control character in a string is accepted on --slurpfile and primary input; jq rejects it

[newhoggy]

Severity: Low (accept-where-jq-refuses, on malformed JSON only — no wrong value for well-formed input)

Summary

A raw, unescaped control character inside a JSON string is rejected by jq on every path and accepted by succinctly on --slurpfile and on the primary document input. RFC 8259 requires U+0000–U+001F to be escaped; json::validate::Validator enforces that (ValidationErrorKind::ControlCharacter), but neither of these paths runs it — they delimit values with find_json_values/serde_json::Deserializer::byte_offset() and then materialize each span with this crate's own decoder, which tolerates the raw byte.

Found while resolving #2052 (which moved --argjson/--jsonargs/--seq onto the crate's own validator; those three reject it correctly).

Repro

printf '["tab\tin"]' — a literal TAB inside the string — captured live against /usr/bin/jq 1.7.1:

$ jq -c . f.json
jq: parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 1, column 9
$ succinctly jq -c . f.json
["tab	in"]                          # accepted, raw TAB round-tripped

$ jq -nc --slurpfile x f.json '$x'
jq: Bad JSON in --slurpfile x f.json: Invalid string: control characters ...
$ succinctly jq -nc --slurpfile x f.json '$x'
[["tab\tin"]]                        # accepted, and re-escaped on output

$ succinctly jq -nc --argjson x '["tab<TAB>in"]' '$x'   # correctly rejected (#2052)

The --slurpfile output is the more surprising half: the raw byte is silently normalized to \t on the way out, so a malformed document becomes a well-formed one with no diagnostic.

Root cause

src/bin/succinctly/jq_runner.rs — parse_json_stream (backing --slurpfile and, per its own doc comment, the primary input path) validates the stream shape with serde_json::Deserializer only to find each value's byte span, then hands the span to json_bytes_to_owned_value_checked. serde_json's span walk is not consulted for string content at that granularity, and the decoder has no control-character rule. find_json_values, the non-serde fallback, is explicitly "boundary-only" and has none either.

Why this is separate from #2052

#2052 replaced the --argjson family's validation gate; parse_json_stream's gate is a different mechanism (stream splitting, not value validation) and its fix has a much wider blast radius — the primary document-input path is the crate's hottest route, so adding a full-content validation pass there is a performance question as well as a correctness one, and the semi-index already walks those bytes once.

Suggested fix direction

Prefer making the decoder (or the semi-index scan already walking the bytes) reject an unescaped U+0000–U+001F inside a string span, rather than adding a second full pass — json::validate's own rule is the reference for what to reject. Measure the primary-input path before and after (succinctly bench run jq_bench), since that is where the cost would land. Check whether succinctly json validate (strict mode) already covers the same input, and add a docs/compliance/jq/limitations.md row if any acceptance is deliberately kept.

[newhoggy]

Triage plan (2026-09-14)

Planned against db974b52e. Oracle: /usr/bin/jq 1.7.1, Homebrew yq v4.53.3.

1. Premise check — confirmed, and wider than filed

$ printf '["tab\tin"]' > ctl.json          # literal TAB inside the string
$ jq -c . ctl.json;                    echo $?   # parse error: Invalid string: control characters ... ; 5
$ sjq -c . ctl.json;                   echo $?   # ["tab<TAB>in"] ; 0
$ sjq -c -s . ctl.json;                echo $?   # [["tab\tin"]]  ; 0   <- silently re-escaped
$ sjq -nc --slurpfile x ctl.json '$x'; echo $?   # [["tab\tin"]]  ; 0

A full sweep of 0x00–0x1F plus 0x7F (33 documents, each ["a<byte>b"], driven through
jq, sjq, sjq -s) gives a clean split: jq rejects exactly 0x00–0x1F and accepts
0x7F; succinctly accepts all 33 on both paths. That is byte-for-byte the rule
json::validate's validate_string already implements (Some(b) if b < 0x20), and the same
rule jq_seq_reader::found_string already ports from jq's own lexer — including its comment
that jq compares a signed char, which is why 0x7F passes. No leniency to model; no
per-byte exceptions.

Two sites the issue does not name:

• fromjson accepts it too, on both --rawfile-sourced and literal text:

  $ jq   -nc '"[\"a\tb\"]" | fromjson'   # error: Invalid string: control characters ...
  $ sjq  -nc '"[\"a\tb\"]" | fromjson'   # ["a\tb"]

• Real yq accepts control characters — yq -n '"[\"a\tb\"]" | fromjson' answers
  ["a\tb"], and yq -p json reads ctl.json fine. So the fromjson fix must be
  jq-mode-gated, per ADR-0018 (the mode decides, not the format).

Paths that are already correct, and stay untouched: --argjson/--jsonargs (#2052),
--validate, succinctly json validate, and --seq, which already matches jq's stderr
byte-for-byte, control-char message and (need RS to resync) column included.

2. Root cause — the issue's stated mechanism is not the one

The issue says serde_json's span walk "is not consulted for string content at that
granularity". It is: serde_json does reject a raw control character. The real mechanism
is parse_json_stream's #1243 leading-zero fallback:

match parse_json_stream_strict(s) {          // serde_json -- rejects the control char
    Ok(values) => Ok(values),
    Err(e) => match find_json_values(bytes) { // lenient rescan -- accepts it
        Ok(spans) => ...,                     // <- the accept happens here
        Err(_) => Err(e),
    },
}

And the primary document path never calls parse_json_stream at all. run_jq's per-file loop
calls find_json_values(raw) directly, then either the identity fast path (raw bytes copied
straight to out, which is why the TAB round-trips unescaped) or JsonIndex::build + M2
streaming.

So there is one accepting site behind all three CLI symptoms:
jq_runner.rs::find_string_end, which scans " and \ and treats every other byte as
ordinary content. It is reached from scan_one_json_token (top-level strings) and from
find_matching_close (strings nested in containers), and therefore backs the primary input
path, --slurp, and --slurpfile alike. fromjson is the second, independent site
(eval.rs::parse_json_string_value).

3. This does not contradict "a filter validates only what it materializes"

limitations.md §"Every M2 filter streams" records, as a deliberate ADR-0018-sanctioned
divergence, that succinctly will not validate a document up front — because it costs "roughly
a second index-building pass over the input". That cost argument does not reach this rule:
find_json_values already runs on every input and already inspects every byte of every
string, so the control-character rule is available for zero extra passes.

The scanner is also already a document-level gate, not a novel one — it rejects truncated
strings and containers today, ahead of any filter:

$ printf '[1,2,'         > t.json; sjq -c '1+1' t.json   # exit 5, "Invalid JSON text"  (jq: 5)
$ printf '"unterminated' > t.json; sjq -c '1+1' t.json   # exit 5, "Invalid JSON text"  (jq: 5)
$ printf '{invalid}'     > t.json; sjq -c '1+1' t.json   # exit 0, `2`                  (jq: 5)  <- stays

The fix extends the first class, and leaves the documented {invalid} divergence exactly
where it is. The resulting asymmetry (a raw control character is caught document-wide; a
malformed member still is not) is worth one sentence in the docs, and it is the same
cost-driven line that section already draws.

4. Plan

1. Add pub fn string_literal_end(bytes: &[u8], pos: usize) -> Option<usize> to
   src/json/light.rs, beside number_literal_end. Returns the exclusive end of the string
   starting at pos, or None for an unterminated string or a raw byte < 0x20. This
   follows the precedent scan_one_json_token's own comment already names for
   number_literal_end: "one validated implementation instead of independently-maintained
   copies". It has to live in the library rather than the bin because util is
   pub(crate) — see step 2.

2. Implement it with the existing SIMD scanner, not a fourth arm on the byte loop.
   util::simd::escape::find_json_escape searches for exactly ", \ and < 0x20 — the
   precise three-way predicate this needs — at 16–32 bytes per iteration. Today's
   find_string_end is a scalar byte-at-a-time loop, and it is hot: sjq '1+1' on an 8.4 MB
   document is ~10 ms wall, i.e. ~840 MB/s, essentially all of it this scan. Routed through
   find_json_escape, the change plausibly lands as a net speed-up rather than a cost.
   Keep the scalar three-way match as the reference/fallback.

3. Delegate jq_runner.rs::find_string_end to it, so find_json_values,
   find_matching_close and scan_one_json_token all inherit the rule from one definition.
   That closes the primary input path (identity fast path, M2 streaming and eager alike),
   --slurp, and --slurpfile — the last because parse_json_stream_strict already rejects
   and the fallback now does too, so parse_json_stream returns serde_json's error.

4. Gate fromjson in eval.rs::parse_json_string_value on the yq_mode: bool that
   function already carries: reject b < 0x20 when !yq_mode. There is existing precedent
   for a mode split in that exact function (jq: an unpaired LOW surrogate escape is echoed verbatim where jq 1.7.1 substitutes U+FFFD #2008/jq: fromjson accepts a lone high surrogate that real jq rejects (pre-existing, found during #2008 review) #2013's surrogate arms). Real yq accepts, so
   yq mode must not change. eval_generic.rs delegates Builtin::FromJson to this site, so
   there is no second evaluator arm to fix.

5. Confirm the --seq path is unmoved. seq_record_ends_unresolved calls
   scan_one_json_token, but only for the EOF/unresolved boolean, and seq_value_is_valid
   (validate_jq_lenient) already rejects control characters — so both routes were already
   answering true. Pin it with the byte-identical stderr comparison captured above rather
   than reasoning about it.

Out of scope, deliberately: yq -p json (goes straight to JsonIndex::build, never through
find_json_values, and real yq accepts — so it is already correct), and the semi-index
decoder itself (the issue's suggestion; the scanner is strictly cheaper, since the decoder is
per-materialized-value while the scanner already walks the bytes once unconditionally).

5. Tests

• tests/jq_cli_tests.rs: an accept/reject matrix over all of 0x00–0x1F plus 0x7F as
  the negative control (the hand-picked-matrix trap: 0x7F is the byte that proves we
  implemented jq's signed-char rule and not "any non-printable"), driven through the primary
  path, -s, --slurpfile, --seq and fromjson, each asserted against the recorded jq
  verdict. Cover the byte in an object key as well as a value — both are reached via
  find_matching_close, but through different call sites.
• Nested and multi-value documents, so find_matching_close's recursive string skip and the
  per-value loop are both exercised, not just a top-level string.
• A yq-mode counter-test: syq -n '"[\"a\tb\"]" | fromjson' must keep answering, pinning the
  mode gate against a future "consistency" sweep.
• src/json/light.rs unit tests for string_literal_end, including a string that is longer
  than one SIMD chunk with the control character in the remainder tail — the scalar/SIMD
  boundary is exactly where a three-way predicate silently loses an arm.
• Consider a RawControlChar variant in tests/jq_cli_tests.rs's FuzzMalformation alphabet
  (test(jq): cross-site consistency test for #1803's 5 duplicated traversal functions #1962/test(jq): the cross-site consistency net's fuzz alphabet cannot express delimiter/gap corruption, and never drives sort_by or path() #2350). It fits the net's "every driven path must raise" contract more strictly
  than the existing variants, since the rejection is document-level; check the expected
  substring wiring before adding it.
• tests/json_test_suite.rs drives json::validate, not the CLI, so it is unaffected — worth
  a sentence in the PR rather than a change.

6. Docs

• docs/compliance/jq/limitations.md: a short entry under the malformed-document family
  recording that a raw U+0000–U+001F inside a string is now rejected at scan time on every
  jq-mode path, why that is free where whole-document validation is not (§3 above), and the
  residual {invalid} asymmetry.
• docs/compliance/yq/limitations.md: a row for fromjson accepting control characters in yq
  mode, matching real yq — so the divergence between the two modes is recorded as intentional.

7. Perf protocol (the real risk in this issue)

The touched function is the hottest scan in the crate, and perf-guard fails on drift in
either direction.

• Interleave the two binaries within each repetition; never all-A-then-all-B.
• Build a never-triggering holdout (the new code compiled in, the rule unreachable) as well
  as pre/post. x86_64 identity queries move ±5–6% on codegen partitioning alone, and the
  release profile is pinned to codegen-units=1 + fat LTO, so a bare pre/post pair cannot
  separate the rule's cost from code-layout drift.
• Measure on both an M4 Pro and the 7950X; this is a memory-bandwidth-adjacent scan and the
  result will not port.
• Workloads: succinctly bench run jq_bench, plus explicitly the string-heavy and
  identity/1+1 shapes, where find_string_end is the largest share of runtime and any
  cost shows first. Gate on output identity before believing any timing.

8. Follow-ups to file (found while planning, out of scope here)

• --slurp/--slurpfile exit code is 1 where jq exits 5/2 on malformed input
  (printf '[1,2,' > t.json; sjq -s . t.json → Error: Invalid JSON in stream, exit 1; jq:
  exit 5). parse_json_stream's fallback returns serde_json's bare anyhow error rather
  than the MalformedJsonError wrapper jq: a malformed object field (e.g. {invalid}) silently disappears instead of erroring #1194 added precisely so the caller could report it in
  jq's channel at exit 5. Undocumented and untracked. The fix here inherits this exit code, so
  it should be fixed alongside but not inside this issue.
• Error-position fidelity for Invalid JSON text. The scanner reports Err(start_of_value),
  so the diagnostic names the value's start line, not the offending byte's line/column. jq
  names the byte. EvalError::malformed_json_text already has the pattern for recovering the
  real reason on the error path only (re-run the validator, pay nothing on valid input) — a
  candidate, but it changes the message for every scanner rejection, so it needs its own
  golden-test impact check.

9. Class — Opus, confirmed (no relabel)

The edit itself is small, but the judgment around it is not:

• a 2-architecture interleaved A/B with a holdout build on the crate's hottest scan, under
  a perf guard that fails on improvements too;
• an ADR-0018 argument that a document-level gate is legitimate here when the recorded
  policy is "validate only what you materialize" — the reasoning is a cost distinction, not a
  rule change, and has to be written down correctly or it reads as a contradiction;
• a per-mode semantic split (fromjson), because real yq accepts what real jq rejects;
• an accept-set re-derived from a live oracle sweep, which also corrected the issue's own
  stated root cause.

That is squarely the Opus band. It is not Fable: no ADR is created or amended, no new
representation, no cross-cutting invariant. It is above Sonnet: not mechanical, not localised,
and decidedly not low-regression-risk.

[newhoggy]

🔧 Claimed by agent Neville — branch issue-2878-control-char-reject

[newhoggy]

Two follow-ups filed while working this, both deliberately out of scope for the accept-vs-reject fix in PR #2959:

• jq/yq: yq's tonumber accepts hex/octal/underscored numeric spellings verbatim; succinctly rejects them #2960 — real yq's tonumber accepts hex/octal/underscored numeric spellings (0x10, 0o17, 1_000) and echoes them verbatim; succinctly yq rejects them. Surfaced only because this issue's review made me sweep tonumber's whole yq-mode error surface to establish which error family real yq uses.
• jq: a rejected multi-value document prints no partial output; jq streams the clean prefix first #2961 — a rejected multi-value document prints no partial output, where jq streams the clean prefix before erroring. Pre-existing (a truncated later value behaves identically on main); this issue routes one more input class into it, and it is now recorded in limitations.md.

[newhoggy]

Third follow-up: #2963 — the perf-guard measured this change's scan as −8.2% instructions on ARM64 and +2.2% on x86_64 (users_keys_unsorted; deterministic cachegrind counts against this PR's own merge-base, consistent in sign across every row on each arch). NEON wins the short-string scan, x86_64's movemask path lands slightly above the scalar loop it replaced — the same short-input effect O3/#87 already documented for this scanner family.

PR #2959 adds a documented QUERY_THRESHOLDS override for that row per 9d555f3's precedent (the guard is abs(drift), so a faster row needs one exactly as a slower one would). #2963 tracks both the x86_64 short-input tuning and — importantly — removing that override once main's merge-base includes this change, so the row isn't blinded permanently.