Skip to content

fix(jq): reject a raw control character in a JSON string, matching jq - #2959

Merged
newhoggy merged 7 commits into
mainfrom
issue-2878-control-char-reject
Sep 14, 2026
Merged

newhoggy merged 7 commits into
mainfrom
issue-2878-control-char-reject

Conversation

@newhoggy

@newhoggy newhoggy commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Real jq rejects a raw, unescaped U+0000U+001F inside a JSON string on every input path. succinctly accepted one on the primary document path, on -s, on --slurpfile, and in fromjson — and on --slurpfile it silently re-escaped the raw byte to \t on the way out, turning a malformed document into a well-formed one with no diagnostic.

The issue's stated root cause is not the one (its triage comment had already corrected this, and I re-confirmed it live). serde_json does reject the control character; the accept came from parse_json_stream's #1243 leading-zero fallback rescan, and the primary path never calls parse_json_stream at all. All three CLI symptoms trace to a single site: jq_runner.rs::find_string_end, which scanned for " and \ and treated every other byte as content.

Three changes:

  1. json::light::string_literal_end — a shared scanner beside number_literal_end, for the reason that function's own comment already gives (one validated implementation instead of independently-maintained copies). jq_runner.rs now delegates from both scan_one_json_token and find_matching_close, so top-level strings, nested values and object keys inherit the rule from one definition. It lives in the library because crate::util is pub(crate).
  2. It scans with the existing SIMD escape scanner, not a fourth arm on a byte loop: util::simd::escape::find_json_escape searches for exactly ", \ and < 0x20 — precisely this three-way question — at 16–32 bytes per iteration, replacing a byte-at-a-time scalar loop on the hottest input route.
  3. fromjson is gated on jq mode (eval.rs::parse_json_string_value). Real yq accepts the same input and answers ["a\tb"] (confirmed live against yq v4.53.3), so the mode decides, not the format (ADR-0018) — the same per-mode precedent as the surrogate arms directly above it (jq: an unpaired LOW surrogate escape is echoed verbatim where jq 1.7.1 substitutes U+FFFD #2008/jq: fromjson accepts a lone high surrogate that real jq rejects (pre-existing, found during #2008 review) #2013).

The part that isn't in the plan: tonumber had to become mode-sensitive

tonumber shares that parser, and its "valid JSON but not a number" vs "not valid JSON at all" probe picks between two error messages. It was mode-blind on purpose — the only split it could previously see (#2008's surrogates) errors in both modes, so it could never change the verdict. This rule can, and each oracle classifies the same string its own way:

control-char string ["a<TAB>b"] | tonumber
jq 1.7.1 Invalid string: control characters … must be escaped → the parse-error family
yq 4.53.3 cannot convert node value […] of tag !!str to number → the tag-conversion family

Leaving that probe hard-coded to jq's mode was measured to flip yq's message to the wrong family, so the real mode is threaded through it — in both evaluators (eval.rs and eval_generic.rs). The yq counter-test pins it; I confirmed it fails against the hard-coded-false version.

0x7F is deliberately still accepted

jq's own check compares a signed char, so DEL passes it. A full 0x000x7F sweep against jq 1.7.1 confirms exactly that split, with no per-byte exceptions — which is why the tests sweep the whole range instead of spot-checking TAB: 0x7F is the byte that separates "jq's actual rule" from the wrong generalisation "reject anything non-printable".

Scope held

--seq is unmoved (verified: its stderr is still byte-identical to jq's), --argjson/--jsonargs (#2052) and --validate were already correct, and yq -p json is untouched. {invalid} stays accepted — the documented, cost-driven divergence — and docs/compliance/jq/limitations.md now records why the new rule is not an exception to "a filter validates only what it materializes": that rule governs the filter, never the document splitter, which already rejected [1,2, and "unterminated ahead of any filter. This change is the stricter, jq-matching direction, so it recovers fidelity in that table.

Performance — measured, and the two architectures disagree in sign

I deliberately posted no local wall-clock number (this box is a laptop running many
concurrent sessions, which docs/guides/benchmarking.md rules out). CI's Perf Regression
Guard is the authoritative measurement here: valgrind --tool=cachegrind instruction
counts against this PR's own merge-base (#1582), deterministic and runner-noise-immune.

users_keys_unsorted wide_keys_unsorted users_identity
ARM64-Linux −8.2% −1.9% −1.3%
x86_64 +2.2% +0.7% +0.4%

Every ARM64 row moved negative; every x86_64 row moved positive — consistent in sign
within each arch, so this is a real property of the scanner on short-string input, not
one noisy row. NEON wins the scan; x86_64's movemask path lands slightly above the scalar
loop it replaced. users_keys_unsorted is the smallest query in the matrix, so it swings
furthest both ways. This is the same short-input effect O3 (#87) documented for this
scanner family.

Only ARM64's −8.2% clears the 5% default, so this PR adds a documented
QUERY_THRESHOLDS override for that row, following 9d555f3's precedent — the guard is
abs(drift), so a faster row needs an override exactly as a slower one would, and that
commit explicitly corrected the "wrong direction" misreading. The override carries its
own removal note, and #2963 tracks both removing it once main's merge-base includes
this change and the x86_64 short-input tuning.

Test plan

  • cargo build --features cli
  • cargo test --features cli,simd,regex,serde8,606 tests / 63 binaries, 0 failures
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo clippy --all-targets --features std,simd,serde,cli,regex,bench-runner,large-tests,mmap-tests -- -D warnings
  • cargo fmt --check
  • Oracle diff, not just goldens: full 0x000x1F + 0x7F sweep against /usr/bin/jq 1.7.1 across the primary path, -s and --slurpfile0 mismatches in 33×3 comparisons; fromjson/tonumber checked against both jq 1.7.1 and yq v4.53.3 in their own modes
  • Negative-tested the tests: each fix reverted in turn, confirming the matching tests fail and only those (splitter → CLI matrix + unit tests; fromjson gate → jq-mode test; mode plumbing → yq counter-test)
  • --seq stderr confirmed byte-identical to jq's

Fixes #2878

Real jq refuses an unescaped U+0000-U+001F inside a string on every
input path; succinctly accepted one on the primary document path, on
`-s`, and on `--slurpfile`, where it was additionally re-escaped to
`\\t` on the way out, turning a malformed document into a well-formed
one with no diagnostic.

All three symptoms came from one site: `jq_runner.rs`'s `find_string_end`
scanned for `"` and `\\` and treated every other byte as content. Replace
it with `json::light::string_literal_end`, a shared scanner beside
`number_literal_end` and for the same stated reason (one validated
implementation instead of independently-maintained copies), reached from
`scan_one_json_token` and `find_matching_close` alike so top-level
strings, nested values and object keys all inherit the rule.

It lives in the library because `crate::util` is `pub(crate)`: the scan
delegates to `util::simd::escape::find_json_escape`, whose predicate
(`"`, `\\`, or `< 0x20`) is already exactly this three-way question, at
16-32 bytes per iteration rather than the byte-at-a-time loop it
replaces.

0x7F is deliberately still accepted: jq's own check compares a signed
char, so DEL passes it. A full 0x00-0x7F sweep against jq 1.7.1 confirms
that split with no per-byte exceptions.

This does not make the splitter a validator -- `{invalid}` stays
accepted, the documented cost-driven divergence. The control-character
rule is affordable precisely because it rides a scan that already
inspects every byte of every string.

Refs #2878
`fromjson` decodes through `eval.rs`'s own parser, never the document
splitter, so it needed its own arm of the same rule. Real jq rejects a
raw U+0000-U+001F there too.

Gated on yq mode, because real yq *accepts* it and answers ["a\\tb"]
(confirmed live against yq v4.53.3) -- the mode decides, not the format
(ADR-0018). Same per-mode split precedent as the surrogate arms directly
above it (#2008/#2013).

`tonumber` shares that parser, and its "valid JSON but not a number" vs
"not valid JSON at all" probe picks between two error messages. That
probe was mode-blind on purpose, because the only split it could see
(#2008's surrogates) errors in both modes and so could never change its
verdict. This rule can: each oracle classifies a control-character
string its own way --

  jq 1.7.1:  Invalid string: control characters ... must be escaped
  yq 4.53.3: cannot convert node value [...] of tag !!str to number

-- which are this crate's `invalid_numeric_literal` and
`cannot_parse_as_number` families respectively. Leaving the probe
hard-coded to jq's mode was measured to flip yq's message to the wrong
family, so thread the real mode through it, in both evaluators.

Refs #2878
Sweeps the whole 0x00-0x1F range plus 0x7F across the primary document
path, `-s` and `--slurpfile`, rather than spot-checking TAB: 0x7F is the
byte that proves which rule is implemented, since jq's signed-char check
accepts DEL and "reject anything non-printable" would be the wrong
generalisation. A hand-picked matrix cannot separate those two.

Also covers the sites the top-level arm does not reach -- an object key
and a value nested in a container, both of which arrive through
`find_matching_close` -- and a two-value document, so the per-value loop
has to keep checking after a clean first value.

The escaped spellings are the second negative control: without them a
fix that rejected every tab, raw or escaped, would pass every other
assertion here.

The yq-mode counter-tests keep a future consistency sweep from importing
jq's stricter rule into yq, and pin `tonumber`'s message family on both
sides of the mode split.

Refs #2878
…validation section

The section states 'a filter validates only what it materializes'. #2878
adds a document-wide rejection, so say explicitly why that is not an
exception to it: the rule governs the filter, never the document
splitter, which already rejected an unterminated string or container
ahead of any filter.

Also records the deliberate asymmetry it leaves -- a raw control
character is caught document-wide, a malformed member still is not --
and that 0x7F sits on the accepting side of the line in both tools.

Refs #2878
Four findings, all confirmed live against the pinned oracles first.

1. Threading yq mode into tonumber's message-family probe flipped the
   *lone low surrogate* case: only yq's reader rejects a lone surrogate
   (#2008) and only jq's rejects a raw control character (#2878), so the
   two sit at opposite ends of the mode split and no single flag gets
   both right. The premise of the comment I wrote was simply wrong.

   The probe itself is the bug in yq mode. Real yq draws no
   valid-JSON-vs-not distinction at all: [1,2], abc, 1 2, "", a lone
   \udc00 and a raw control character all answer "cannot convert node
   value [...] of tag !!str to number" (7/7, captured live from v4.53.3).
   So yq mode no longer runs the probe -- it answers with its single
   family unconditionally, which matches the oracle in every case above
   and is less code than the plumbing it replaces. Incidentally fixes the
   bare-word, trailing-content and empty-string cases, which took jq's
   family on main.

2. A document whose later value holds the control character is rejected
   whole, where jq prints the clean prefix first. The exit code agrees,
   the streamed prefix does not. That is the splitter's pre-existing
   all-or-nothing shape -- a truncated later value behaves identically on
   main -- so it is recorded rather than changed, and the test now
   asserts stdout instead of only the exit code, since an exit-code-only
   assertion cannot tell the two apart.

3. The new fromjson message is an internal signal, never output: both
   parse_complete_json callers discard it. Say so, so a future reader
   does not believe this fix widened jq's per-reason wording.

4. scan_one_json_token's "'Ends' is structural, not a validity verdict"
   contract is what --seq is documented against, and the new string arm
   made it false. Restate it with both deliberate exceptions named.

Refs #2878
`core::str::from_utf8(doc).unwrap()` sat in the message argument of a
passing assertion, so it was an executable line that only ever runs when
the assertion fails -- the one uncovered added line in this branch, and
uncoverable by construction. Inline `{doc:?}` says the same thing with
no call, and removes a panic path from a test besides.

Refs #2878
@github-actions

github-actions Bot commented Sep 14, 2026

Copy link
Copy Markdown

Coverage

Total: 93.52% ⚪ 0 pp vs main

Comparing c76a4f3..95ddd4b (merge-base → PR head)

File Before After Δ
src/json/light.rs 92.44% 92.55% 🟢 0.12 pp
🔇 0 ignored region(s), 90 tolerated region(s)

ignore removes the lines from both reports; tolerate keeps them in the reported percentage but scores them against the baseline, so a cross-run flip cannot move a delta. Regions are read from each revision's own source.

File Kind Lines Rev Reason
src/bin/succinctly/jq_runner.rs tolerate 572-584 both unreachable: try_parse_meta_op only fires under ParserMode::Yq (src/jq/parser.rs), and rewrite_namespaced_calls is only reached via ModuleProcessor::process_program, which jq_runner's own jq-mode run is the sole caller of -- so a MetaAssign node can never reach this function (#798)
src/bin/succinctly/jq_runner.rs tolerate 1539 both unreachable: widening the shadow-candidate set never rejects a program the first parse accepted -- a newly covered name only wraps an already-successful dedicated parse, and a failing one would have propagated its error in the first parse too, so the retry budget is charged at the identical sites in both (#2395)
src/bin/succinctly/jq_runner.rs tolerate 7414 both unreachable in a passing suite by design -- the fixed b\
src/bin/succinctly/yq_runner.rs tolerate 1638 both unreachable: bytes already parsed successfully by every caller (#1350)
src/bin/succinctly/yq_runner.rs tolerate 3299 both unreachable: path is always the raw output of the path(TARGET) builtin evaluated a few lines up in resolve_one_meta_assign -- path/1 is a jq/yq language invariant that always answers an array of path components (see Expr::Builtin(Builtin::PathNoArg) => Ok(Some(OwnedValue::Array(..))) in eval_generic.rs), never any other shape (#798)
src/bin/succinctly/yq_runner.rs tolerate 3399 both unreachable: resolve_meta_assign_writes runs expr through this before any evaluation begins (see its own doc comment), and Expr::Shared is never constructed by the parser -- only at eval time, by function-call argument substitution (substitute_func_param in eval.rs) -- so a pre-evaluation AST can never contain one here (#798)
src/bin/succinctly/yq_runner.rs tolerate 4028 both unreachable: every arm of the match result { .. } above that assigns docs (L3492-3622) constructs Ok(..) -- none ever produces Err, so this if let's implicit else can't be taken; symmetric to L1625's ? (#798)
src/jq/document.rs tolerate 1069-1076 both unreachable: both implementors (StandardJson, YamlValue) override this to decode once; the default exists as the contract a future implementor inherits, and is deliberately the two-call sequence it replaces (#965)
src/jq/eval.rs tolerate 1140 both unreachable: def is always a collect_alias_groups anchor path, which step_to_expr never fails on (#1351)
src/jq/eval.rs tolerate 1161 both unreachable: redirect_paths with Redirect::SINGLE always contributes exactly one output per input, so a 1-element paths always pops Some (#1351)
src/jq/eval.rs tolerate 1170 both unreachable: a concrete setpath/delpaths path's components are always Field/Index -- step_to_expr never produces another shape (#1351)
src/jq/eval.rs tolerate 1174 both unreachable: the map above never yields None, since it only ever matches Field/Index (#1351)
src/jq/eval.rs tolerate 4247 both unreachable: is_escape() is exactly `Error
src/jq/eval.rs tolerate 4248 both unreachable: see the if let above -- push_owned_values never answers None for an is_escape() result (#2180)
src/jq/eval.rs tolerate 5396 both unreachable today: to_owned's only failures are is_decode_failure()-tagged, and suppresses() answers false for those whatever optional is -- the same defensive-but-dead arm eval_generic's own Builtin::Path materialization documents under #2280 (#2908)
src/jq/eval.rs tolerate 7277 both unreachable: optional is never true here. eval_each is entered with a forced true at exactly one site (Expr::Optional over an IndexExpr/SliceExpr), and both of those evaluate their target (eval_index_expr) and their key (eval_each(key, .., false)) with a hardcoded false, so only the final index/slice step ever sees it -- nothing carries it down to an Expr::Object (#2180)
src/jq/eval.rs tolerate 10587 both unreachable: only ever constructed by builtin_sort_keys's own eval_update_no_vivify call, whose enclosing eval_update_impl already runs to_owned on the whole document up front (#2855) -- a decode failure anywhere raises there, before this filter ever sees a value to re-decode; confirmed live, sort_keys(.a)/sort_keys(..) on a document with a decode-failure subtree both raise from the outer to_owned
src/jq/eval.rs tolerate 21159 both unreachable: escape_with_prefix! sets terminal before Demand::Stop; already returned above (#2138)
src/jq/eval.rs tolerate 21528 both unreachable: escape! sets terminal before Demand::Stop; already returned above (#2546)
src/jq/eval.rs tolerate 30354 both unreachable: is_primitive admits only Identity/Field/Index/Slice, and of those only a Slice's computed bounds can halt -- all four have their own arm in resolve_node_sink/resolve_node_eager, so none reaches this function. Pre-existing; #2694 only wrapped the return in Some (#2694)
src/jq/eval.rs tolerate 30388 both unreachable, as this arm's own comment above says: indexing or slicing a value yields zero or one result, so is_primitive never produces more than one -- kept as a named error rather than a panic. Pre-existing; #2694 only wrapped the enclosing return in Some (#2694)
src/jq/eval.rs tolerate 32033 both unreachable: PatternStep::component only ever builds Expr::Field/Expr::Index, and navigation_element answers Some for both (#2649)
src/jq/eval.rs tolerate 35614 both unreachable in a passing suite by design -- a panic-message format argument for the #682 single-valued-tail pin, evaluated only if that assert's own condition is false (#2190)
src/jq/eval.rs tolerate 35665 both unreachable: classify_static_component answers Field only for OwnedValue::Object, and it was handed this very value (#2190)
src/jq/eval.rs tolerate 35670 both unreachable: classify_static_component answers Index only for OwnedValue::Array, and it was handed this very value (#2190)
src/jq/eval.rs tolerate 35747 both unreachable: the only caller reaches this after classify_static_component answered Field for this same value, which it does only for an object (#2190)
src/jq/eval.rs tolerate 35791 both unreachable: both callers establish the container first -- navigate_static_component_ref via classify_static_component's Index arm, and walk_path's Expr::Iterate arm by matching on the container itself (#2190)
src/jq/eval.rs tolerate 39306 both unreachable: stop_with_escape's only write is slot.set(Some(control)) with the control it was handed, which is always the Control::Error built one line above (#2180)
src/jq/eval.rs tolerate 40153 both unreachable by construction: the per-fork match only ever hands stop_with_downstream a non-Exhausted flow, so terminal can never hold Exhausted. foreach_forks' identical arm is 0-hit for the same reason and is only unflagged because it predates this diff (#2899)
src/jq/eval.rs tolerate 52475 both unreachable in practice: a def with more than 64 parameters; the fallback exists so ScopeMask's one-bit-per-parameter u64 is a performance ceiling rather than a correctness limit (#2633)
src/jq/eval.rs tolerate 52487 both unreachable: bind_def_call only calls this for a non-empty params, and install_def_calls only builds a DefCall whose args.len() equals params.len(), so the zip is never empty here (#2560)
src/jq/eval.rs tolerate 52507 both unreachable in practice: needs more than 32 duplicated-name parameters; see the non-duplicate path's own note (#2633)
src/jq/eval.rs tolerate 52528 both unreachable: name was just read from params, so the zip over (params, args) has a matching pair unless args is shorter than params, which install_def_calls' own arity guard rules out (#2560)
src/jq/eval.rs tolerate 52545 both unreachable: params is non-empty here (bind_def_call's own guard) and its first entry is never skipped, so at least one substitution always ran (#2560)
src/jq/eval.rs tolerate 52606 both unreachable: same arity invariant as bind_def_call_params' own copy of this loop -- name came from params, so the zip has a matching pair unless args is shorter, which install_def_calls rules out (#2560)
src/jq/eval.rs tolerate 53924 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53946 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53964 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 54034 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 54048 both substitute_func_param_impl's FuncDef arm always returns FuncDef (#2555)
src/jq/eval.rs tolerate 61048 both unreachable in a passing suite by design -- this is the failure message for the assertion the test exists to make (#2190)
src/jq/eval.rs tolerate 89409 both unreachable in a passing suite by design -- every filter this helper is called with parses to an AsPattern (#2649)
src/jq/eval.rs tolerate 89441 both unreachable in a passing suite by design -- every call site passes the origin of a binding this same test already proved carries a marker (#2649)
src/jq/eval.rs tolerate 90321 both unreachable in a passing suite by design -- every row here is a shape jq accepts, confirmed live (#2649)
src/jq/eval.rs tolerate 90389 both unreachable in a passing suite by design -- every row here is a shape jq refuses, confirmed live (#2649)
src/jq/eval.rs tolerate 90753 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the let-else pattern fails to match (#2072)
src/jq/eval.rs tolerate 90764 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the let-else pattern fails to match (#2072)
src/jq/eval.rs tolerate 90785 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the match doesn't hit the expected arm above (#2072)
src/jq/eval.rs tolerate 90801 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the match doesn't hit the expected arm above (#2072)
src/jq/eval_generic.rs tolerate 3615 both unreachable: the sole remaining caller (retain_truthy_generic's Many arm) runs to_owned on an item before keeping it, so re-converting a kept item here cannot fail; the ManyCursor caller that made this reachable went with the truthiness walk (#2692, re-establishing #2661's premise)
src/jq/eval_generic.rs tolerate 6969 both unreachable in a passing suite by design -- this is the panic message for the #2368 pin itself, only formatted if the assert's own condition is false (#2368)
src/jq/eval_generic.rs tolerate 10710 both unreachable: every producer that reaches this empty-exprs tail (each_lazy_keys_iterate_sink's sorted/!sorted arms, each_lazy_index_range_iterate_sink, each_lazy_seq_iterate_sink) yields OneCursorValue/OneCursor/Owned, never a cursorless GenericItem::One -- so cursor is always Some here; kept for exhaustiveness/symmetry with the Some arm (#2103)
src/jq/eval_generic.rs tolerate 13817 both unreachable: escape_generic!/ensure_owned! set terminal before Demand::Stop; already returned above (#2138)
src/jq/eval_generic.rs tolerate 14171 both unreachable: escape! sets terminal before Demand::Stop; already returned above (#2546)
src/jq/eval_generic.rs tolerate 16624 both unreachable: len_checked and SliceBounds::resolve already bound every index in range to [0, len), so get_cursor cannot miss (#2168)
src/jq/eval_generic.rs tolerate 20282 both unreachable: malformed_object_member above already proved every key stringifies (the None half), and to_owned_cursor on an untagged key decoded_key_str decoded cannot fail (the Err half) (#2785)
src/jq/eval_generic.rs tolerate 20363 both unreachable: map(f) over an array emits exactly one array; kept so a future map shape produces no output rather than a panic (#2785)
src/jq/eval_generic.rs tolerate 20631 both unreachable by design -- eval_single's #2368 debug_assert forbids optional=true on Builtin::Reverse, so length never answers None here (#2730)
src/jq/eval_generic.rs tolerate 22490 both unreachable by construction: every shape either guard admits now has an arm above (#2771), and expr_dispatch_catchall_guards_default_conservatively_2549 pins both guards' _ => false defaults directly
src/jq/eval_generic.rs tolerate 22990 both unreachable in a passing suite by design -- owned_identity_rule maps a bare Expr::Var to Bound too (for the static gate, which sees a body before its as substitution runs), but every runtime dispatch that reaches this rule (owned_identity_after_stage/owned_identity_placed_by, from owned_identity_leaving_cursor's Bound arm) only ever sees a stage after eval_owned_identity_as's unconditional substitute_bound_var_from call, which always turns $x into Expr::TrackedVar before recursing -- confirmed by running the full suite with this arm replaced by a hard panic!(), which never fired (#2072)
src/jq/eval_generic.rs tolerate 24827 both unreachable in a passing suite by design -- the fixture's map(.+1) is always a LazySeq; this arm is the test's own diagnostic (#2666)
src/jq/eval_generic.rs tolerate 24855 both unreachable in a passing suite by design -- the fixture's only escape is Control::Error; this arm is the test's own diagnostic (#2666)
src/json/light.rs tolerate 6447 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/json/light.rs tolerate 6458 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/json/light.rs tolerate 6539 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/index.rs tolerate 1268 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1272 both unreachable: every fixture field_key_head_foot is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1283 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1285 both unreachable: every call to field_key_head_foot in this test module passes a key that the fixture's mapping actually has (#798)
src/yaml/index.rs tolerate 1295 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1299 both unreachable: every fixture seq_item_head_foot is called with in this test module is a top-level sequence (#798)
src/yaml/index.rs tolerate 1322 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1341 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1364 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1373 both unreachable: every fixture field_key_head_foot_in_doc is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1384 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1386 both unreachable: every call to field_key_head_foot_in_doc in this test module passes a key that the fixture's document actually has (#798)
src/yaml/index.rs tolerate 1397 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1401 both unreachable: every fixture nested_key_head_foot is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1405 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this let-else's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1411 both unreachable: every fixture nested_key_head_foot is called with has a nested mapping under outer (#798)
src/yaml/index.rs tolerate 1422 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1425 both unreachable: every call to nested_key_head_foot in this test module passes an outer.inner pair that the fixture actually has (#798)
src/yaml/light.rs tolerate 3364 both unreachable: an alias target is never None for a built index (#1374)
src/yaml/light.rs tolerate 15398 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/light.rs tolerate 15409 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/light.rs tolerate 15566 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/parser.rs tolerate 1583 both unreachable: every block-sequence open registers a frame at its own depth before any item of it can be parsed (#1079)
src/yaml/parser.rs tolerate 1613 both unreachable: this function's sole caller (record_standalone_comment) only invokes it from inside a match on pending_head_lines.last(), so pending_head_lines is already known non-empty here (#798)
src/yaml/parser.rs tolerate 7999 both unreachable: every byte here already passed the [0-9.eE+-] charset check above, a strict subset of ASCII, so str::from_utf8 can never fail (#2778)

Patch coverage

Patch: 100% (64/64 new lines covered)

File Patch Uncovered new lines
src/bin/succinctly/jq_runner.rs 100% (2/2)
src/jq/eval.rs 100% (13/13)
src/jq/eval_generic.rs 100% (1/1)
src/json/light.rs 100% (48/48)

Indirect coverage changes

🔴 0 lines lost coverage, 🟢 1 lines gained coverage on unchanged code.

Indirect changes
  • src/jq/eval.rs:16066 🟢 uncovered → covered

📦 Full per-file coverage summary · run summary

@github-actions

github-actions Bot commented Sep 14, 2026

Copy link
Copy Markdown

Coverage

Total: 93.43% ⚪ 0 pp vs main

Comparing c76a4f3..95ddd4b (merge-base → PR head)

File Before After Δ
src/json/light.rs 92.41% 92.52% 🟢 0.12 pp
🔇 0 ignored region(s), 91 tolerated region(s)

ignore removes the lines from both reports; tolerate keeps them in the reported percentage but scores them against the baseline, so a cross-run flip cannot move a delta. Regions are read from each revision's own source.

File Kind Lines Rev Reason
src/bin/succinctly/jq_runner.rs tolerate 572-584 both unreachable: try_parse_meta_op only fires under ParserMode::Yq (src/jq/parser.rs), and rewrite_namespaced_calls is only reached via ModuleProcessor::process_program, which jq_runner's own jq-mode run is the sole caller of -- so a MetaAssign node can never reach this function (#798)
src/bin/succinctly/jq_runner.rs tolerate 1539 both unreachable: widening the shadow-candidate set never rejects a program the first parse accepted -- a newly covered name only wraps an already-successful dedicated parse, and a failing one would have propagated its error in the first parse too, so the retry budget is charged at the identical sites in both (#2395)
src/bin/succinctly/jq_runner.rs tolerate 7414 both unreachable in a passing suite by design -- the fixed b\
src/bin/succinctly/yq_runner.rs tolerate 1638 both unreachable: bytes already parsed successfully by every caller (#1350)
src/bin/succinctly/yq_runner.rs tolerate 3299 both unreachable: path is always the raw output of the path(TARGET) builtin evaluated a few lines up in resolve_one_meta_assign -- path/1 is a jq/yq language invariant that always answers an array of path components (see Expr::Builtin(Builtin::PathNoArg) => Ok(Some(OwnedValue::Array(..))) in eval_generic.rs), never any other shape (#798)
src/bin/succinctly/yq_runner.rs tolerate 3399 both unreachable: resolve_meta_assign_writes runs expr through this before any evaluation begins (see its own doc comment), and Expr::Shared is never constructed by the parser -- only at eval time, by function-call argument substitution (substitute_func_param in eval.rs) -- so a pre-evaluation AST can never contain one here (#798)
src/bin/succinctly/yq_runner.rs tolerate 4028 both unreachable: every arm of the match result { .. } above that assigns docs (L3492-3622) constructs Ok(..) -- none ever produces Err, so this if let's implicit else can't be taken; symmetric to L1625's ? (#798)
src/jq/document.rs tolerate 1069-1076 both unreachable: both implementors (StandardJson, YamlValue) override this to decode once; the default exists as the contract a future implementor inherits, and is deliberately the two-call sequence it replaces (#965)
src/jq/eval.rs tolerate 1140 both unreachable: def is always a collect_alias_groups anchor path, which step_to_expr never fails on (#1351)
src/jq/eval.rs tolerate 1161 both unreachable: redirect_paths with Redirect::SINGLE always contributes exactly one output per input, so a 1-element paths always pops Some (#1351)
src/jq/eval.rs tolerate 1170 both unreachable: a concrete setpath/delpaths path's components are always Field/Index -- step_to_expr never produces another shape (#1351)
src/jq/eval.rs tolerate 1174 both unreachable: the map above never yields None, since it only ever matches Field/Index (#1351)
src/jq/eval.rs tolerate 4247 both unreachable: is_escape() is exactly `Error
src/jq/eval.rs tolerate 4248 both unreachable: see the if let above -- push_owned_values never answers None for an is_escape() result (#2180)
src/jq/eval.rs tolerate 5396 both unreachable today: to_owned's only failures are is_decode_failure()-tagged, and suppresses() answers false for those whatever optional is -- the same defensive-but-dead arm eval_generic's own Builtin::Path materialization documents under #2280 (#2908)
src/jq/eval.rs tolerate 7277 both unreachable: optional is never true here. eval_each is entered with a forced true at exactly one site (Expr::Optional over an IndexExpr/SliceExpr), and both of those evaluate their target (eval_index_expr) and their key (eval_each(key, .., false)) with a hardcoded false, so only the final index/slice step ever sees it -- nothing carries it down to an Expr::Object (#2180)
src/jq/eval.rs tolerate 10587 both unreachable: only ever constructed by builtin_sort_keys's own eval_update_no_vivify call, whose enclosing eval_update_impl already runs to_owned on the whole document up front (#2855) -- a decode failure anywhere raises there, before this filter ever sees a value to re-decode; confirmed live, sort_keys(.a)/sort_keys(..) on a document with a decode-failure subtree both raise from the outer to_owned
src/jq/eval.rs tolerate 21159 both unreachable: escape_with_prefix! sets terminal before Demand::Stop; already returned above (#2138)
src/jq/eval.rs tolerate 21528 both unreachable: escape! sets terminal before Demand::Stop; already returned above (#2546)
src/jq/eval.rs tolerate 30354 both unreachable: is_primitive admits only Identity/Field/Index/Slice, and of those only a Slice's computed bounds can halt -- all four have their own arm in resolve_node_sink/resolve_node_eager, so none reaches this function. Pre-existing; #2694 only wrapped the return in Some (#2694)
src/jq/eval.rs tolerate 30388 both unreachable, as this arm's own comment above says: indexing or slicing a value yields zero or one result, so is_primitive never produces more than one -- kept as a named error rather than a panic. Pre-existing; #2694 only wrapped the enclosing return in Some (#2694)
src/jq/eval.rs tolerate 32033 both unreachable: PatternStep::component only ever builds Expr::Field/Expr::Index, and navigation_element answers Some for both (#2649)
src/jq/eval.rs tolerate 35614 both unreachable in a passing suite by design -- a panic-message format argument for the #682 single-valued-tail pin, evaluated only if that assert's own condition is false (#2190)
src/jq/eval.rs tolerate 35665 both unreachable: classify_static_component answers Field only for OwnedValue::Object, and it was handed this very value (#2190)
src/jq/eval.rs tolerate 35670 both unreachable: classify_static_component answers Index only for OwnedValue::Array, and it was handed this very value (#2190)
src/jq/eval.rs tolerate 35747 both unreachable: the only caller reaches this after classify_static_component answered Field for this same value, which it does only for an object (#2190)
src/jq/eval.rs tolerate 35791 both unreachable: both callers establish the container first -- navigate_static_component_ref via classify_static_component's Index arm, and walk_path's Expr::Iterate arm by matching on the container itself (#2190)
src/jq/eval.rs tolerate 39306 both unreachable: stop_with_escape's only write is slot.set(Some(control)) with the control it was handed, which is always the Control::Error built one line above (#2180)
src/jq/eval.rs tolerate 40153 both unreachable by construction: the per-fork match only ever hands stop_with_downstream a non-Exhausted flow, so terminal can never hold Exhausted. foreach_forks' identical arm is 0-hit for the same reason and is only unflagged because it predates this diff (#2899)
src/jq/eval.rs tolerate 52475 both unreachable in practice: a def with more than 64 parameters; the fallback exists so ScopeMask's one-bit-per-parameter u64 is a performance ceiling rather than a correctness limit (#2633)
src/jq/eval.rs tolerate 52487 both unreachable: bind_def_call only calls this for a non-empty params, and install_def_calls only builds a DefCall whose args.len() equals params.len(), so the zip is never empty here (#2560)
src/jq/eval.rs tolerate 52507 both unreachable in practice: needs more than 32 duplicated-name parameters; see the non-duplicate path's own note (#2633)
src/jq/eval.rs tolerate 52528 both unreachable: name was just read from params, so the zip over (params, args) has a matching pair unless args is shorter than params, which install_def_calls' own arity guard rules out (#2560)
src/jq/eval.rs tolerate 52545 both unreachable: params is non-empty here (bind_def_call's own guard) and its first entry is never skipped, so at least one substitution always ran (#2560)
src/jq/eval.rs tolerate 52606 both unreachable: same arity invariant as bind_def_call_params' own copy of this loop -- name came from params, so the zip has a matching pair unless args is shorter, which install_def_calls rules out (#2560)
src/jq/eval.rs tolerate 53924 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53946 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53964 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 54034 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 54048 both substitute_func_param_impl's FuncDef arm always returns FuncDef (#2555)
src/jq/eval.rs tolerate 61048 both unreachable in a passing suite by design -- this is the failure message for the assertion the test exists to make (#2190)
src/jq/eval.rs tolerate 89409 both unreachable in a passing suite by design -- every filter this helper is called with parses to an AsPattern (#2649)
src/jq/eval.rs tolerate 89441 both unreachable in a passing suite by design -- every call site passes the origin of a binding this same test already proved carries a marker (#2649)
src/jq/eval.rs tolerate 90321 both unreachable in a passing suite by design -- every row here is a shape jq accepts, confirmed live (#2649)
src/jq/eval.rs tolerate 90389 both unreachable in a passing suite by design -- every row here is a shape jq refuses, confirmed live (#2649)
src/jq/eval.rs tolerate 90753 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the let-else pattern fails to match (#2072)
src/jq/eval.rs tolerate 90764 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the let-else pattern fails to match (#2072)
src/jq/eval.rs tolerate 90785 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the match doesn't hit the expected arm above (#2072)
src/jq/eval.rs tolerate 90801 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the match doesn't hit the expected arm above (#2072)
src/jq/eval_generic.rs tolerate 3615 both unreachable: the sole remaining caller (retain_truthy_generic's Many arm) runs to_owned on an item before keeping it, so re-converting a kept item here cannot fail; the ManyCursor caller that made this reachable went with the truthiness walk (#2692, re-establishing #2661's premise)
src/jq/eval_generic.rs tolerate 6969 both unreachable in a passing suite by design -- this is the panic message for the #2368 pin itself, only formatted if the assert's own condition is false (#2368)
src/jq/eval_generic.rs tolerate 10710 both unreachable: every producer that reaches this empty-exprs tail (each_lazy_keys_iterate_sink's sorted/!sorted arms, each_lazy_index_range_iterate_sink, each_lazy_seq_iterate_sink) yields OneCursorValue/OneCursor/Owned, never a cursorless GenericItem::One -- so cursor is always Some here; kept for exhaustiveness/symmetry with the Some arm (#2103)
src/jq/eval_generic.rs tolerate 13817 both unreachable: escape_generic!/ensure_owned! set terminal before Demand::Stop; already returned above (#2138)
src/jq/eval_generic.rs tolerate 14171 both unreachable: escape! sets terminal before Demand::Stop; already returned above (#2546)
src/jq/eval_generic.rs tolerate 16624 both unreachable: len_checked and SliceBounds::resolve already bound every index in range to [0, len), so get_cursor cannot miss (#2168)
src/jq/eval_generic.rs tolerate 20282 both unreachable: malformed_object_member above already proved every key stringifies (the None half), and to_owned_cursor on an untagged key decoded_key_str decoded cannot fail (the Err half) (#2785)
src/jq/eval_generic.rs tolerate 20363 both unreachable: map(f) over an array emits exactly one array; kept so a future map shape produces no output rather than a panic (#2785)
src/jq/eval_generic.rs tolerate 20631 both unreachable by design -- eval_single's #2368 debug_assert forbids optional=true on Builtin::Reverse, so length never answers None here (#2730)
src/jq/eval_generic.rs tolerate 22490 both unreachable by construction: every shape either guard admits now has an arm above (#2771), and expr_dispatch_catchall_guards_default_conservatively_2549 pins both guards' _ => false defaults directly
src/jq/eval_generic.rs tolerate 22990 both unreachable in a passing suite by design -- owned_identity_rule maps a bare Expr::Var to Bound too (for the static gate, which sees a body before its as substitution runs), but every runtime dispatch that reaches this rule (owned_identity_after_stage/owned_identity_placed_by, from owned_identity_leaving_cursor's Bound arm) only ever sees a stage after eval_owned_identity_as's unconditional substitute_bound_var_from call, which always turns $x into Expr::TrackedVar before recursing -- confirmed by running the full suite with this arm replaced by a hard panic!(), which never fired (#2072)
src/jq/eval_generic.rs tolerate 24827 both unreachable in a passing suite by design -- the fixture's map(.+1) is always a LazySeq; this arm is the test's own diagnostic (#2666)
src/jq/eval_generic.rs tolerate 24855 both unreachable in a passing suite by design -- the fixture's only escape is Control::Error; this arm is the test's own diagnostic (#2666)
src/json/light.rs tolerate 6447 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/json/light.rs tolerate 6458 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/json/light.rs tolerate 6539 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/util/simd/x86.rs tolerate 208-258 both CPU-gated: the avx512f early-return only executes on Zen 4+ / Skylake-X runners, and its absence changes which AMD/Intel branch below executes too (#2449)
src/yaml/index.rs tolerate 1268 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1272 both unreachable: every fixture field_key_head_foot is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1283 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1285 both unreachable: every call to field_key_head_foot in this test module passes a key that the fixture's mapping actually has (#798)
src/yaml/index.rs tolerate 1295 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1299 both unreachable: every fixture seq_item_head_foot is called with in this test module is a top-level sequence (#798)
src/yaml/index.rs tolerate 1322 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1341 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1364 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1373 both unreachable: every fixture field_key_head_foot_in_doc is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1384 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1386 both unreachable: every call to field_key_head_foot_in_doc in this test module passes a key that the fixture's document actually has (#798)
src/yaml/index.rs tolerate 1397 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1401 both unreachable: every fixture nested_key_head_foot is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1405 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this let-else's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1411 both unreachable: every fixture nested_key_head_foot is called with has a nested mapping under outer (#798)
src/yaml/index.rs tolerate 1422 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1425 both unreachable: every call to nested_key_head_foot in this test module passes an outer.inner pair that the fixture actually has (#798)
src/yaml/light.rs tolerate 3364 both unreachable: an alias target is never None for a built index (#1374)
src/yaml/light.rs tolerate 15398 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/light.rs tolerate 15409 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/light.rs tolerate 15566 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/parser.rs tolerate 1583 both unreachable: every block-sequence open registers a frame at its own depth before any item of it can be parsed (#1079)
src/yaml/parser.rs tolerate 1613 both unreachable: this function's sole caller (record_standalone_comment) only invokes it from inside a match on pending_head_lines.last(), so pending_head_lines is already known non-empty here (#798)
src/yaml/parser.rs tolerate 7999 both unreachable: every byte here already passed the [0-9.eE+-] charset check above, a strict subset of ASCII, so str::from_utf8 can never fail (#2778)

Patch coverage

Patch: 100% (64/64 new lines covered)

File Patch Uncovered new lines
src/bin/succinctly/jq_runner.rs 100% (2/2)
src/jq/eval.rs 100% (13/13)
src/jq/eval_generic.rs 100% (1/1)
src/json/light.rs 100% (48/48)

Indirect coverage changes

🔴 0 lines lost coverage, 🟢 1 lines gained coverage on unchanged code.

Indirect changes
  • src/jq/eval.rs:16066 🟢 uncovered → covered

📦 Full per-file coverage summary · run summary

… change

Routing the document splitter through find_json_escape replaced a
byte-at-a-time scalar loop, and the two architectures disagree in sign.
Measured by this guard on its own runners against the PR's merge-base:

         users_keys_unsorted  wide_keys_unsorted  users_identity
  ARM64        -8.2%               -1.9%             -1.3%
  x86_64       +2.2%               +0.7%             +0.4%

NEON wins the scan; x86_64's movemask path costs slightly more than the
scalar loop it replaced on short strings, which the users fixture is made
of. users_keys_unsorted is the smallest query in the matrix, so it swings
furthest in both directions. Only the ARM64 number clears the 5% default.

The guard is abs(drift), so a faster row needs an override exactly as a
slower one would -- 9d555f3 corrected that same misreading.

To be removed once main has moved past #2878: with --baseline-binary on
every run the checked-in file is never consulted, so once the merge-base
includes this change the row reads ~0% again and the override only blinds
it. Follow-up filed on the issue.

Refs #2878
@newhoggy
newhoggy added this pull request to the merge queue Sep 14, 2026
Merged via the queue into main with commit 79d86d0 Sep 14, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

jq: a raw control character in a string is accepted on --slurpfile and primary input; jq rejects it

1 participant