Skip to content

fix(jq): give a ? over a fan-out group jq's single abort scope - #2930

Merged
newhoggy merged 4 commits into
mainfrom
issue-2909-optional-group-scope
Sep 14, 2026
Merged

fix(jq): give a ? over a fan-out group jq's single abort scope#2930
newhoggy merged 4 commits into
mainfrom
issue-2909-optional-group-scope

Conversation

@newhoggy

Copy link
Copy Markdown
Contributor

Summary

(A | B)? is try (A | B)one abort scope over the whole group: a
failure anywhere inside prunes the group's branch and stops the group's own
generators. Two sites rewrote it into A? | B? instead —
push_path_components (#1311) for the resolver, splice_optional_group
(#1294) for the write walkers — turning one abort scope into N independent
per-step ones.

Both rewrites exist to stop the group's ? leaking onto what follows the
group, and both are correct for the pure-navigation groups they were written
for: a single-valued chain has one branch, so pruning at step k and aborting
the group are indistinguishable. Put a generator inside the group and they
part company — it resumes past an error a later component raised.

The issue understated this

#2909 filed it as a side-effect count with output matching. The output
diverges, and silently. All captured live from /usr/bin/jq 1.7.1:

$ echo '{"a":{"b":1},"c":[1,2]}' | jq -c 'del((.[] | .[0])?)'
{"a":{"b":1},"c":[1,2]}          # was: {"a":{"b":1},"c":[2]}   <- deletes an element jq keeps

$ echo '[{"a":1},5,{"a":3}]' | jq -c '((.[] | .a)?) |= 99'
[{"a":99},5,{"a":3}]             # was: [{"a":99},5,{"a":99}]   <- writes a slot jq never reaches

$ echo '{"a":{"b":1},"c":[1,2]}' | jq -c '[path(.. | ((.[], .a) | .b)?)]'
[["a","b"]]                      # was: [["a","b"],["a","b"],["a","a","b"]]

Two further corrections to the issue's own diagnosis: .. is not required
([path(. | (.[]|stderr|.+0)?)] diverges identically), and it is not
resolve_optional_sink vs another ?-handling arm — in the diverging shapes
the ? never reaches a ?-handling arm at all, because it is dissolved before
resolution.

One row of the issue's own list does not diverge here: ((.[] | .a)?) = 99
already matched, so it is in the must-not-change set rather than the fixed set.

How

optional_group_is_scope_safe — one predicate, both rewrites gated on it:
distribute only when the group has fewer than two components, or every one of
them is single-valued (Identity/Field/Index/Slice, through
Paren/Optional). Conservative on purpose: a group like (.[] | .a?) that
happens not to diverge today is also called unsafe, which only changes which
correct arm handles it.

An unsafe group then has to reach that arm, so needs_path_prepass and
needs_fanout_pass both answer true for it. Both are needed, for different
reasons: the former is what routes del/=/|=/path through the resolver
at all; the latter peels Optional before testing, so
Optional(Pipe([Iterate, Field])) would otherwise be mistaken for a
single-valued static tail. With both set, the group reaches
resolve_optional_sink — the arm that was already right, and the reason the
issue's three "neighbouring spellings agree" rows agree.

wrap_optional_branch's depth() != 1 arm was annotated "unreached today" and
coverage-tolerated, on reasoning that held only while a group was always
distributed before reaching a resolver. It is live now — verified by
instrumenting it — and needed no new logic: the old comment had already stated
jq's answer for exactly this shape.

Verification beyond the suite

Test plan

Fixes #2909

@github-actions

github-actions Bot commented Sep 13, 2026

Copy link
Copy Markdown

Coverage

Total: 93.5% ⚪ 0 pp vs main

Comparing 950b8a5..46bf851 (merge-base → PR head)

No per-file coverage changes vs main.

🔇 0 ignored region(s), 91 tolerated region(s)

ignore removes the lines from both reports; tolerate keeps them in the reported percentage but scores them against the baseline, so a cross-run flip cannot move a delta. Regions are read from each revision's own source.

File Kind Lines Rev Reason
src/bin/succinctly/jq_runner.rs tolerate 572-584 both unreachable: try_parse_meta_op only fires under ParserMode::Yq (src/jq/parser.rs), and rewrite_namespaced_calls is only reached via ModuleProcessor::process_program, which jq_runner's own jq-mode run is the sole caller of -- so a MetaAssign node can never reach this function (#798)
src/bin/succinctly/jq_runner.rs tolerate 1539 both unreachable: widening the shadow-candidate set never rejects a program the first parse accepted -- a newly covered name only wraps an already-successful dedicated parse, and a failing one would have propagated its error in the first parse too, so the retry budget is charged at the identical sites in both (#2395)
src/bin/succinctly/jq_runner.rs tolerate 7404 both unreachable in a passing suite by design -- the fixed b\
src/bin/succinctly/yq_runner.rs tolerate 1638 both unreachable: bytes already parsed successfully by every caller (#1350)
src/bin/succinctly/yq_runner.rs tolerate 3299 both unreachable: path is always the raw output of the path(TARGET) builtin evaluated a few lines up in resolve_one_meta_assign -- path/1 is a jq/yq language invariant that always answers an array of path components (see Expr::Builtin(Builtin::PathNoArg) => Ok(Some(OwnedValue::Array(..))) in eval_generic.rs), never any other shape (#798)
src/bin/succinctly/yq_runner.rs tolerate 3399 both unreachable: resolve_meta_assign_writes runs expr through this before any evaluation begins (see its own doc comment), and Expr::Shared is never constructed by the parser -- only at eval time, by function-call argument substitution (substitute_func_param in eval.rs) -- so a pre-evaluation AST can never contain one here (#798)
src/bin/succinctly/yq_runner.rs tolerate 4028 both unreachable: every arm of the match result { .. } above that assigns docs (L3492-3622) constructs Ok(..) -- none ever produces Err, so this if let's implicit else can't be taken; symmetric to L1625's ? (#798)
src/jq/document.rs tolerate 1069-1076 both unreachable: both implementors (StandardJson, YamlValue) override this to decode once; the default exists as the contract a future implementor inherits, and is deliberately the two-call sequence it replaces (#965)
src/jq/eval.rs tolerate 1128 both unreachable: def is always a collect_alias_groups anchor path, which step_to_expr never fails on (#1351)
src/jq/eval.rs tolerate 1149 both unreachable: redirect_paths with Redirect::SINGLE always contributes exactly one output per input, so a 1-element paths always pops Some (#1351)
src/jq/eval.rs tolerate 1158 both unreachable: a concrete setpath/delpaths path's components are always Field/Index -- step_to_expr never produces another shape (#1351)
src/jq/eval.rs tolerate 1162 both unreachable: the map above never yields None, since it only ever matches Field/Index (#1351)
src/jq/eval.rs tolerate 4235 both unreachable: is_escape() is exactly `Error
src/jq/eval.rs tolerate 4236 both unreachable: see the if let above -- push_owned_values never answers None for an is_escape() result (#2180)
src/jq/eval.rs tolerate 5384 both unreachable today: to_owned's only failures are is_decode_failure()-tagged, and suppresses() answers false for those whatever optional is -- the same defensive-but-dead arm eval_generic's own Builtin::Path materialization documents under #2280 (#2908)
src/jq/eval.rs tolerate 7265 both unreachable: optional is never true here. eval_each is entered with a forced true at exactly one site (Expr::Optional over an IndexExpr/SliceExpr), and both of those evaluate their target (eval_index_expr) and their key (eval_each(key, .., false)) with a hardcoded false, so only the final index/slice step ever sees it -- nothing carries it down to an Expr::Object (#2180)
src/jq/eval.rs tolerate 10524 both unreachable: only ever constructed by builtin_sort_keys's own eval_update_no_vivify call, whose enclosing eval_update_impl already runs to_owned on the whole document up front (#2855) -- a decode failure anywhere raises there, before this filter ever sees a value to re-decode; confirmed live, sort_keys(.a)/sort_keys(..) on a document with a decode-failure subtree both raise from the outer to_owned
src/jq/eval.rs tolerate 21052 both unreachable: escape_with_prefix! sets terminal before Demand::Stop; already returned above (#2138)
src/jq/eval.rs tolerate 21421 both unreachable: escape! sets terminal before Demand::Stop; already returned above (#2546)
src/jq/eval.rs tolerate 29723 base unreachable today: postfix ? attaches to a single path element until #367 reopens it, so no path this resolver builds has depth() != 1 here (#2649)
src/jq/eval.rs tolerate 30241 both unreachable: is_primitive admits only Identity/Field/Index/Slice, and of those only a Slice's computed bounds can halt -- all four have their own arm in resolve_node_sink/resolve_node_eager, so none reaches this function. Pre-existing; #2694 only wrapped the return in Some (#2694)
src/jq/eval.rs tolerate 30275 both unreachable, as this arm's own comment above says: indexing or slicing a value yields zero or one result, so is_primitive never produces more than one -- kept as a named error rather than a panic. Pre-existing; #2694 only wrapped the enclosing return in Some (#2694)
src/jq/eval.rs tolerate 31920 both unreachable: PatternStep::component only ever builds Expr::Field/Expr::Index, and navigation_element answers Some for both (#2649)
src/jq/eval.rs tolerate 35501 both unreachable in a passing suite by design -- a panic-message format argument for the #682 single-valued-tail pin, evaluated only if that assert's own condition is false (#2190)
src/jq/eval.rs tolerate 35552 both unreachable: classify_static_component answers Field only for OwnedValue::Object, and it was handed this very value (#2190)
src/jq/eval.rs tolerate 35557 both unreachable: classify_static_component answers Index only for OwnedValue::Array, and it was handed this very value (#2190)
src/jq/eval.rs tolerate 35634 both unreachable: the only caller reaches this after classify_static_component answered Field for this same value, which it does only for an object (#2190)
src/jq/eval.rs tolerate 35678 both unreachable: both callers establish the container first -- navigate_static_component_ref via classify_static_component's Index arm, and walk_path's Expr::Iterate arm by matching on the container itself (#2190)
src/jq/eval.rs tolerate 39193 both unreachable: stop_with_escape's only write is slot.set(Some(control)) with the control it was handed, which is always the Control::Error built one line above (#2180)
src/jq/eval.rs tolerate 40040 both unreachable by construction: the per-fork match only ever hands stop_with_downstream a non-Exhausted flow, so terminal can never hold Exhausted. foreach_forks' identical arm is 0-hit for the same reason and is only unflagged because it predates this diff (#2899)
src/jq/eval.rs tolerate 52362 both unreachable in practice: a def with more than 64 parameters; the fallback exists so ScopeMask's one-bit-per-parameter u64 is a performance ceiling rather than a correctness limit (#2633)
src/jq/eval.rs tolerate 52374 both unreachable: bind_def_call only calls this for a non-empty params, and install_def_calls only builds a DefCall whose args.len() equals params.len(), so the zip is never empty here (#2560)
src/jq/eval.rs tolerate 52394 both unreachable in practice: needs more than 32 duplicated-name parameters; see the non-duplicate path's own note (#2633)
src/jq/eval.rs tolerate 52415 both unreachable: name was just read from params, so the zip over (params, args) has a matching pair unless args is shorter than params, which install_def_calls' own arity guard rules out (#2560)
src/jq/eval.rs tolerate 52432 both unreachable: params is non-empty here (bind_def_call's own guard) and its first entry is never skipped, so at least one substitution always ran (#2560)
src/jq/eval.rs tolerate 52493 both unreachable: same arity invariant as bind_def_call_params' own copy of this loop -- name came from params, so the zip has a matching pair unless args is shorter, which install_def_calls rules out (#2560)
src/jq/eval.rs tolerate 53811 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53833 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53851 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53921 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53935 both substitute_func_param_impl's FuncDef arm always returns FuncDef (#2555)
src/jq/eval.rs tolerate 60935 both unreachable in a passing suite by design -- this is the failure message for the assertion the test exists to make (#2190)
src/jq/eval.rs tolerate 89296 both unreachable in a passing suite by design -- every filter this helper is called with parses to an AsPattern (#2649)
src/jq/eval.rs tolerate 89328 both unreachable in a passing suite by design -- every call site passes the origin of a binding this same test already proved carries a marker (#2649)
src/jq/eval.rs tolerate 90208 both unreachable in a passing suite by design -- every row here is a shape jq accepts, confirmed live (#2649)
src/jq/eval.rs tolerate 90276 both unreachable in a passing suite by design -- every row here is a shape jq refuses, confirmed live (#2649)
src/jq/eval.rs tolerate 90640 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the let-else pattern fails to match (#2072)
src/jq/eval.rs tolerate 90651 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the let-else pattern fails to match (#2072)
src/jq/eval.rs tolerate 90672 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the match doesn't hit the expected arm above (#2072)
src/jq/eval.rs tolerate 90688 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the match doesn't hit the expected arm above (#2072)
src/jq/eval_generic.rs tolerate 3554 both unreachable: the sole remaining caller (retain_truthy_generic's Many arm) runs to_owned on an item before keeping it, so re-converting a kept item here cannot fail; the ManyCursor caller that made this reachable went with the truthiness walk (#2692, re-establishing #2661's premise)
src/jq/eval_generic.rs tolerate 6908 both unreachable in a passing suite by design -- this is the panic message for the #2368 pin itself, only formatted if the assert's own condition is false (#2368)
src/jq/eval_generic.rs tolerate 10649 both unreachable: every producer that reaches this empty-exprs tail (each_lazy_keys_iterate_sink's sorted/!sorted arms, each_lazy_index_range_iterate_sink, each_lazy_seq_iterate_sink) yields OneCursorValue/OneCursor/Owned, never a cursorless GenericItem::One -- so cursor is always Some here; kept for exhaustiveness/symmetry with the Some arm (#2103)
src/jq/eval_generic.rs tolerate 13756 both unreachable: escape_generic!/ensure_owned! set terminal before Demand::Stop; already returned above (#2138)
src/jq/eval_generic.rs tolerate 14110 both unreachable: escape! sets terminal before Demand::Stop; already returned above (#2546)
src/jq/eval_generic.rs tolerate 16563 both unreachable: len_checked and SliceBounds::resolve already bound every index in range to [0, len), so get_cursor cannot miss (#2168)
src/jq/eval_generic.rs tolerate 20221 both unreachable: malformed_object_member above already proved every key stringifies (the None half), and to_owned_cursor on an untagged key decoded_key_str decoded cannot fail (the Err half) (#2785)
src/jq/eval_generic.rs tolerate 20302 both unreachable: map(f) over an array emits exactly one array; kept so a future map shape produces no output rather than a panic (#2785)
src/jq/eval_generic.rs tolerate 20570 both unreachable by design -- eval_single's #2368 debug_assert forbids optional=true on Builtin::Reverse, so length never answers None here (#2730)
src/jq/eval_generic.rs tolerate 22429 both unreachable by construction: every shape either guard admits now has an arm above (#2771), and expr_dispatch_catchall_guards_default_conservatively_2549 pins both guards' _ => false defaults directly
src/jq/eval_generic.rs tolerate 22929 both unreachable in a passing suite by design -- owned_identity_rule maps a bare Expr::Var to Bound too (for the static gate, which sees a body before its as substitution runs), but every runtime dispatch that reaches this rule (owned_identity_after_stage/owned_identity_placed_by, from owned_identity_leaving_cursor's Bound arm) only ever sees a stage after eval_owned_identity_as's unconditional substitute_bound_var_from call, which always turns $x into Expr::TrackedVar before recursing -- confirmed by running the full suite with this arm replaced by a hard panic!(), which never fired (#2072)
src/jq/eval_generic.rs tolerate 24764 both unreachable in a passing suite by design -- the fixture's map(.+1) is always a LazySeq; this arm is the test's own diagnostic (#2666)
src/jq/eval_generic.rs tolerate 24792 both unreachable in a passing suite by design -- the fixture's only escape is Control::Error; this arm is the test's own diagnostic (#2666)
src/json/light.rs tolerate 6311 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/json/light.rs tolerate 6322 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/json/light.rs tolerate 6403 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/index.rs tolerate 1268 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1272 both unreachable: every fixture field_key_head_foot is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1283 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1285 both unreachable: every call to field_key_head_foot in this test module passes a key that the fixture's mapping actually has (#798)
src/yaml/index.rs tolerate 1295 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1299 both unreachable: every fixture seq_item_head_foot is called with in this test module is a top-level sequence (#798)
src/yaml/index.rs tolerate 1322 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1341 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1364 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1373 both unreachable: every fixture field_key_head_foot_in_doc is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1384 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1386 both unreachable: every call to field_key_head_foot_in_doc in this test module passes a key that the fixture's document actually has (#798)
src/yaml/index.rs tolerate 1397 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1401 both unreachable: every fixture nested_key_head_foot is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1405 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this let-else's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1411 both unreachable: every fixture nested_key_head_foot is called with has a nested mapping under outer (#798)
src/yaml/index.rs tolerate 1422 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1425 both unreachable: every call to nested_key_head_foot in this test module passes an outer.inner pair that the fixture actually has (#798)
src/yaml/light.rs tolerate 3364 both unreachable: an alias target is never None for a built index (#1374)
src/yaml/light.rs tolerate 15398 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/light.rs tolerate 15409 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/light.rs tolerate 15566 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/parser.rs tolerate 1583 both unreachable: every block-sequence open registers a frame at its own depth before any item of it can be parsed (#1079)
src/yaml/parser.rs tolerate 1613 both unreachable: this function's sole caller (record_standalone_comment) only invokes it from inside a match on pending_head_lines.last(), so pending_head_lines is already known non-empty here (#798)
src/yaml/parser.rs tolerate 7999 both unreachable: every byte here already passed the [0-9.eE+-] charset check above, a strict subset of ASCII, so str::from_utf8 can never fail (#2778)

Patch coverage

Patch: 100% (27/27 new lines covered)

File Patch Uncovered new lines
src/jq/eval.rs 100% (27/27)

Indirect coverage changes

🔴 0 lines lost coverage, 🟢 1 lines gained coverage on unchanged code.

Indirect changes
  • src/jq/eval.rs:37019 🟢 uncovered → covered

📦 Full per-file coverage summary · run summary

@github-actions

github-actions Bot commented Sep 13, 2026

Copy link
Copy Markdown

Coverage

Total: 93.4% ⚪ 0 pp vs main

Comparing 950b8a5..46bf851 (merge-base → PR head)

No per-file coverage changes vs main.

🔇 0 ignored region(s), 92 tolerated region(s)

ignore removes the lines from both reports; tolerate keeps them in the reported percentage but scores them against the baseline, so a cross-run flip cannot move a delta. Regions are read from each revision's own source.

File Kind Lines Rev Reason
src/bin/succinctly/jq_runner.rs tolerate 572-584 both unreachable: try_parse_meta_op only fires under ParserMode::Yq (src/jq/parser.rs), and rewrite_namespaced_calls is only reached via ModuleProcessor::process_program, which jq_runner's own jq-mode run is the sole caller of -- so a MetaAssign node can never reach this function (#798)
src/bin/succinctly/jq_runner.rs tolerate 1539 both unreachable: widening the shadow-candidate set never rejects a program the first parse accepted -- a newly covered name only wraps an already-successful dedicated parse, and a failing one would have propagated its error in the first parse too, so the retry budget is charged at the identical sites in both (#2395)
src/bin/succinctly/jq_runner.rs tolerate 7404 both unreachable in a passing suite by design -- the fixed b\
src/bin/succinctly/yq_runner.rs tolerate 1638 both unreachable: bytes already parsed successfully by every caller (#1350)
src/bin/succinctly/yq_runner.rs tolerate 3299 both unreachable: path is always the raw output of the path(TARGET) builtin evaluated a few lines up in resolve_one_meta_assign -- path/1 is a jq/yq language invariant that always answers an array of path components (see Expr::Builtin(Builtin::PathNoArg) => Ok(Some(OwnedValue::Array(..))) in eval_generic.rs), never any other shape (#798)
src/bin/succinctly/yq_runner.rs tolerate 3399 both unreachable: resolve_meta_assign_writes runs expr through this before any evaluation begins (see its own doc comment), and Expr::Shared is never constructed by the parser -- only at eval time, by function-call argument substitution (substitute_func_param in eval.rs) -- so a pre-evaluation AST can never contain one here (#798)
src/bin/succinctly/yq_runner.rs tolerate 4028 both unreachable: every arm of the match result { .. } above that assigns docs (L3492-3622) constructs Ok(..) -- none ever produces Err, so this if let's implicit else can't be taken; symmetric to L1625's ? (#798)
src/jq/document.rs tolerate 1069-1076 both unreachable: both implementors (StandardJson, YamlValue) override this to decode once; the default exists as the contract a future implementor inherits, and is deliberately the two-call sequence it replaces (#965)
src/jq/eval.rs tolerate 1128 both unreachable: def is always a collect_alias_groups anchor path, which step_to_expr never fails on (#1351)
src/jq/eval.rs tolerate 1149 both unreachable: redirect_paths with Redirect::SINGLE always contributes exactly one output per input, so a 1-element paths always pops Some (#1351)
src/jq/eval.rs tolerate 1158 both unreachable: a concrete setpath/delpaths path's components are always Field/Index -- step_to_expr never produces another shape (#1351)
src/jq/eval.rs tolerate 1162 both unreachable: the map above never yields None, since it only ever matches Field/Index (#1351)
src/jq/eval.rs tolerate 4235 both unreachable: is_escape() is exactly `Error
src/jq/eval.rs tolerate 4236 both unreachable: see the if let above -- push_owned_values never answers None for an is_escape() result (#2180)
src/jq/eval.rs tolerate 5384 both unreachable today: to_owned's only failures are is_decode_failure()-tagged, and suppresses() answers false for those whatever optional is -- the same defensive-but-dead arm eval_generic's own Builtin::Path materialization documents under #2280 (#2908)
src/jq/eval.rs tolerate 7265 both unreachable: optional is never true here. eval_each is entered with a forced true at exactly one site (Expr::Optional over an IndexExpr/SliceExpr), and both of those evaluate their target (eval_index_expr) and their key (eval_each(key, .., false)) with a hardcoded false, so only the final index/slice step ever sees it -- nothing carries it down to an Expr::Object (#2180)
src/jq/eval.rs tolerate 10524 both unreachable: only ever constructed by builtin_sort_keys's own eval_update_no_vivify call, whose enclosing eval_update_impl already runs to_owned on the whole document up front (#2855) -- a decode failure anywhere raises there, before this filter ever sees a value to re-decode; confirmed live, sort_keys(.a)/sort_keys(..) on a document with a decode-failure subtree both raise from the outer to_owned
src/jq/eval.rs tolerate 21052 both unreachable: escape_with_prefix! sets terminal before Demand::Stop; already returned above (#2138)
src/jq/eval.rs tolerate 21421 both unreachable: escape! sets terminal before Demand::Stop; already returned above (#2546)
src/jq/eval.rs tolerate 29723 base unreachable today: postfix ? attaches to a single path element until #367 reopens it, so no path this resolver builds has depth() != 1 here (#2649)
src/jq/eval.rs tolerate 30241 both unreachable: is_primitive admits only Identity/Field/Index/Slice, and of those only a Slice's computed bounds can halt -- all four have their own arm in resolve_node_sink/resolve_node_eager, so none reaches this function. Pre-existing; #2694 only wrapped the return in Some (#2694)
src/jq/eval.rs tolerate 30275 both unreachable, as this arm's own comment above says: indexing or slicing a value yields zero or one result, so is_primitive never produces more than one -- kept as a named error rather than a panic. Pre-existing; #2694 only wrapped the enclosing return in Some (#2694)
src/jq/eval.rs tolerate 31920 both unreachable: PatternStep::component only ever builds Expr::Field/Expr::Index, and navigation_element answers Some for both (#2649)
src/jq/eval.rs tolerate 35501 both unreachable in a passing suite by design -- a panic-message format argument for the #682 single-valued-tail pin, evaluated only if that assert's own condition is false (#2190)
src/jq/eval.rs tolerate 35552 both unreachable: classify_static_component answers Field only for OwnedValue::Object, and it was handed this very value (#2190)
src/jq/eval.rs tolerate 35557 both unreachable: classify_static_component answers Index only for OwnedValue::Array, and it was handed this very value (#2190)
src/jq/eval.rs tolerate 35634 both unreachable: the only caller reaches this after classify_static_component answered Field for this same value, which it does only for an object (#2190)
src/jq/eval.rs tolerate 35678 both unreachable: both callers establish the container first -- navigate_static_component_ref via classify_static_component's Index arm, and walk_path's Expr::Iterate arm by matching on the container itself (#2190)
src/jq/eval.rs tolerate 39193 both unreachable: stop_with_escape's only write is slot.set(Some(control)) with the control it was handed, which is always the Control::Error built one line above (#2180)
src/jq/eval.rs tolerate 40040 both unreachable by construction: the per-fork match only ever hands stop_with_downstream a non-Exhausted flow, so terminal can never hold Exhausted. foreach_forks' identical arm is 0-hit for the same reason and is only unflagged because it predates this diff (#2899)
src/jq/eval.rs tolerate 52362 both unreachable in practice: a def with more than 64 parameters; the fallback exists so ScopeMask's one-bit-per-parameter u64 is a performance ceiling rather than a correctness limit (#2633)
src/jq/eval.rs tolerate 52374 both unreachable: bind_def_call only calls this for a non-empty params, and install_def_calls only builds a DefCall whose args.len() equals params.len(), so the zip is never empty here (#2560)
src/jq/eval.rs tolerate 52394 both unreachable in practice: needs more than 32 duplicated-name parameters; see the non-duplicate path's own note (#2633)
src/jq/eval.rs tolerate 52415 both unreachable: name was just read from params, so the zip over (params, args) has a matching pair unless args is shorter than params, which install_def_calls' own arity guard rules out (#2560)
src/jq/eval.rs tolerate 52432 both unreachable: params is non-empty here (bind_def_call's own guard) and its first entry is never skipped, so at least one substitution always ran (#2560)
src/jq/eval.rs tolerate 52493 both unreachable: same arity invariant as bind_def_call_params' own copy of this loop -- name came from params, so the zip has a matching pair unless args is shorter, which install_def_calls rules out (#2560)
src/jq/eval.rs tolerate 53811 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53833 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53851 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53921 both substitute_var_impl's FuncDef arm always returns FuncDef (#2283)
src/jq/eval.rs tolerate 53935 both substitute_func_param_impl's FuncDef arm always returns FuncDef (#2555)
src/jq/eval.rs tolerate 60935 both unreachable in a passing suite by design -- this is the failure message for the assertion the test exists to make (#2190)
src/jq/eval.rs tolerate 89296 both unreachable in a passing suite by design -- every filter this helper is called with parses to an AsPattern (#2649)
src/jq/eval.rs tolerate 89328 both unreachable in a passing suite by design -- every call site passes the origin of a binding this same test already proved carries a marker (#2649)
src/jq/eval.rs tolerate 90208 both unreachable in a passing suite by design -- every row here is a shape jq accepts, confirmed live (#2649)
src/jq/eval.rs tolerate 90276 both unreachable in a passing suite by design -- every row here is a shape jq refuses, confirmed live (#2649)
src/jq/eval.rs tolerate 90640 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the let-else pattern fails to match (#2072)
src/jq/eval.rs tolerate 90651 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the let-else pattern fails to match (#2072)
src/jq/eval.rs tolerate 90672 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the match doesn't hit the expected arm above (#2072)
src/jq/eval.rs tolerate 90688 both unreachable in a passing suite by design -- this is the panic message for the #2072 pin itself, only formatted if the match doesn't hit the expected arm above (#2072)
src/jq/eval_generic.rs tolerate 3554 both unreachable: the sole remaining caller (retain_truthy_generic's Many arm) runs to_owned on an item before keeping it, so re-converting a kept item here cannot fail; the ManyCursor caller that made this reachable went with the truthiness walk (#2692, re-establishing #2661's premise)
src/jq/eval_generic.rs tolerate 6908 both unreachable in a passing suite by design -- this is the panic message for the #2368 pin itself, only formatted if the assert's own condition is false (#2368)
src/jq/eval_generic.rs tolerate 10649 both unreachable: every producer that reaches this empty-exprs tail (each_lazy_keys_iterate_sink's sorted/!sorted arms, each_lazy_index_range_iterate_sink, each_lazy_seq_iterate_sink) yields OneCursorValue/OneCursor/Owned, never a cursorless GenericItem::One -- so cursor is always Some here; kept for exhaustiveness/symmetry with the Some arm (#2103)
src/jq/eval_generic.rs tolerate 13756 both unreachable: escape_generic!/ensure_owned! set terminal before Demand::Stop; already returned above (#2138)
src/jq/eval_generic.rs tolerate 14110 both unreachable: escape! sets terminal before Demand::Stop; already returned above (#2546)
src/jq/eval_generic.rs tolerate 16563 both unreachable: len_checked and SliceBounds::resolve already bound every index in range to [0, len), so get_cursor cannot miss (#2168)
src/jq/eval_generic.rs tolerate 20221 both unreachable: malformed_object_member above already proved every key stringifies (the None half), and to_owned_cursor on an untagged key decoded_key_str decoded cannot fail (the Err half) (#2785)
src/jq/eval_generic.rs tolerate 20302 both unreachable: map(f) over an array emits exactly one array; kept so a future map shape produces no output rather than a panic (#2785)
src/jq/eval_generic.rs tolerate 20570 both unreachable by design -- eval_single's #2368 debug_assert forbids optional=true on Builtin::Reverse, so length never answers None here (#2730)
src/jq/eval_generic.rs tolerate 22429 both unreachable by construction: every shape either guard admits now has an arm above (#2771), and expr_dispatch_catchall_guards_default_conservatively_2549 pins both guards' _ => false defaults directly
src/jq/eval_generic.rs tolerate 22929 both unreachable in a passing suite by design -- owned_identity_rule maps a bare Expr::Var to Bound too (for the static gate, which sees a body before its as substitution runs), but every runtime dispatch that reaches this rule (owned_identity_after_stage/owned_identity_placed_by, from owned_identity_leaving_cursor's Bound arm) only ever sees a stage after eval_owned_identity_as's unconditional substitute_bound_var_from call, which always turns $x into Expr::TrackedVar before recursing -- confirmed by running the full suite with this arm replaced by a hard panic!(), which never fired (#2072)
src/jq/eval_generic.rs tolerate 24764 both unreachable in a passing suite by design -- the fixture's map(.+1) is always a LazySeq; this arm is the test's own diagnostic (#2666)
src/jq/eval_generic.rs tolerate 24792 both unreachable in a passing suite by design -- the fixture's only escape is Control::Error; this arm is the test's own diagnostic (#2666)
src/json/light.rs tolerate 6311 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/json/light.rs tolerate 6322 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/json/light.rs tolerate 6403 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/util/simd/x86.rs tolerate 208-258 both CPU-gated: the avx512f early-return only executes on Zen 4+ / Skylake-X runners, and its absence changes which AMD/Intel branch below executes too (#2449)
src/yaml/index.rs tolerate 1268 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1272 both unreachable: every fixture field_key_head_foot is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1283 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1285 both unreachable: every call to field_key_head_foot in this test module passes a key that the fixture's mapping actually has (#798)
src/yaml/index.rs tolerate 1295 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1299 both unreachable: every fixture seq_item_head_foot is called with in this test module is a top-level sequence (#798)
src/yaml/index.rs tolerate 1322 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1341 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1364 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1373 both unreachable: every fixture field_key_head_foot_in_doc is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1384 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1386 both unreachable: every call to field_key_head_foot_in_doc in this test module passes a key that the fixture's document actually has (#798)
src/yaml/index.rs tolerate 1397 both unreachable: YamlIndex::build always wraps the parsed document(s) in a virtual root Sequence, at TY index 0 (#798)
src/yaml/index.rs tolerate 1401 both unreachable: every fixture nested_key_head_foot is called with in this test module is a top-level mapping (#798)
src/yaml/index.rs tolerate 1405 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this let-else's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1411 both unreachable: every fixture nested_key_head_foot is called with has a nested mapping under outer (#798)
src/yaml/index.rs tolerate 1422 both unreachable: a mapping key is always emitted as YamlValue::String -- it is never type-inferred like a value (#222), so this if-let's pattern can never fail to match (#798)
src/yaml/index.rs tolerate 1425 both unreachable: every call to nested_key_head_foot in this test module passes an outer.inner pair that the fixture actually has (#798)
src/yaml/light.rs tolerate 3364 both unreachable: an alias target is never None for a built index (#1374)
src/yaml/light.rs tolerate 15398 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/light.rs tolerate 15409 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/light.rs tolerate 15566 both unreachable in a passing suite by design -- this is a panic-message format argument for the #2072 pin itself, only evaluated if the assert's own condition is false (#2072)
src/yaml/parser.rs tolerate 1583 both unreachable: every block-sequence open registers a frame at its own depth before any item of it can be parsed (#1079)
src/yaml/parser.rs tolerate 1613 both unreachable: this function's sole caller (record_standalone_comment) only invokes it from inside a match on pending_head_lines.last(), so pending_head_lines is already known non-empty here (#798)
src/yaml/parser.rs tolerate 7999 both unreachable: every byte here already passed the [0-9.eE+-] charset check above, a strict subset of ASCII, so str::from_utf8 can never fail (#2778)

Patch coverage

Patch: 100% (27/27 new lines covered)

File Patch Uncovered new lines
src/jq/eval.rs 100% (27/27)

Indirect coverage changes

🔴 0 lines lost coverage, 🟢 1 lines gained coverage on unchanged code.

Indirect changes
  • src/jq/eval.rs:37019 🟢 uncovered → covered

📦 Full per-file coverage summary · run summary

@newhoggy

Copy link
Copy Markdown
Contributor Author

Both findings addressed; the crash one was serious and the diagnosis exactly right.

The crash was self-inflicted duplication. I wrote the scope-safety rule twice — the expression form skipped Identity and recursed into Pipe/Paren, the slice form counted Identity and treated a Paren(Pipe(..)) element as multi-valued. That let a group be safe to the routing gate (kept on the native walker) and unsafe to splice_optional_group, which then returned the very list the caller had just decomposed. This repo's own "duplicated predicates diverge silently" lesson (#106), in a commit whose message claimed "one predicate". There is one walk and one entry point now.

The better half of the fix was dropping the splice gate entirely. Gating that site is what makes it a fixed point, and it turns out to be unnecessary — needs_path_prepass already routes a fan-out group through resolve_optional_sink before any walker sees it. I re-ran the sweep with that site ungated, widened to 3,840 combinations with your crash shapes and ./paren-heavy randomised bodies added: 0 output differences, 0 crashes. Six of those shapes are now CLI tests (they fail with exit 134 rather than subtly if either half regresses).

Your point about the 756-combination sweep is the one I want to keep: it varied neither a bare . nor a parenthesised sub-pipe inside the group, which is precisely why it could not see this. The alphabet a fuzz sweep draws from is part of the claim it supports.

The three documentation findings are fixed too — including is_atomic_path_component's comment, which stated the false invariant as the safety argument for the whole change, and push_path_components' doc comment, which I had accidentally absorbed into the new predicate's rustdoc (the same defect b6abf5d76 fixed for resolve_leaf).

@newhoggy
newhoggy force-pushed the issue-2909-optional-group-scope branch from 6ac0daf to 14cacb6 Compare September 13, 2026 20:14
`(A | B)?` is `try (A | B)`: a failure anywhere inside prunes the whole
group's branch and stops the group's own generators. Two sites rewrote it
into `A? | B?` instead -- `push_path_components` (#1311) for the resolver,
`splice_optional_group` (#1294) for the write walkers -- turning one abort
scope into N independent per-step ones, so a generator inside the group
resumed past an error a later component raised.

Both rewrites exist to stop the group's `?` leaking onto what *follows* the
group, and both are correct for the pure-navigation groups they were written
for. A single-valued chain has one branch, so pruning at step k and aborting
the group are indistinguishable. Put a generator inside and they part
company.

#2909 filed this as a side-effect count. It is not: the output diverges, and
silently. Captured live against jq 1.7.1:

    echo '{"a":{"b":1},"c":[1,2]}' | jq -c 'del((.[] | .[0])?)'
      jq: {"a":{"b":1},"c":[1,2]}     was: {"a":{"b":1},"c":[2]}
    echo '[{"a":1},5,{"a":3}]' | jq -c '((.[] | .a)?) |= 99'
      jq: [{"a":99},5,{"a":3}]        was: [{"a":99},5,{"a":99}]

-- an element deleted that jq keeps, and a slot written that jq never
reaches. The issue's own diagnosis needs two corrections as well: `..` is
not required (any pipe does it), and the `?` never reaches a `?`-handling
arm in the diverging shapes -- it is dissolved before resolution.

One predicate, `optional_group_is_scope_safe`, gates both rewrites:
distribute only when the group has fewer than two components or every one of
them is single-valued. Conservative on purpose -- a group like `(.[] | .a?)`
that happens not to diverge today is also called unsafe, which only changes
which correct arm handles it.

An unsafe group then has to *reach* that arm, so `needs_path_prepass` and
`needs_fanout_pass` both answer `true` for it. Both are needed and for
different reasons: the former is what routes `del`/`=`/`|=`/`path` through
the resolver at all, and the latter peels `Optional` before testing, so
`Optional(Pipe([Iterate, Field]))` would otherwise be mistaken for a
single-valued static tail. With both set the group reaches
`resolve_optional_sink` -- the arm that was already right, and the reason
the issue's three "neighbouring spellings agree" rows agree.

`wrap_optional_branch`'s `depth() != 1` arm was annotated "unreached
today", tolerated for coverage, on reasoning that held only while a group
was always distributed before reaching a resolver. It is live now --
verified by instrumenting it -- and needed no new logic: the old comment had
already stated jq's answer for exactly this shape.

Verified by a 2,400-combination differential sweep (30 group bodies, 14 of
them randomised, x 10 outer spellings x 8 inputs) against jq 1.7.1: 8
divergences, all one pre-existing shape (`del((.[0:1] | .. | ..)?)`, which
diverges identically on the merge-base and without any `?` at all -- filed
separately). Zero regressions. A further 756-combination scan for an
"invalid path component" leak from an un-distributed group reaching a
native walker: none.

Refs #2909
Three tests, every expectation captured from jq 1.7.1 rather than derived.

The fix'\''s own rows, including the two silent wrong answers (`del` keeping
an element it used to remove, `|=` skipping a slot it used to write) and
the duplicated-path row -- plus the issue'\''s `stderr` repro and the
`.`-piped form that shows `..` was never the trigger.

The upper edge: the three neighbouring spellings that already matched, one
of which (`path((...)?)` with no enclosing pipe) reaches
`resolve_optional_sink` directly and is therefore the fix'\''s reference
behaviour, not merely a control.

The lower edge, which is the one that matters for the gate: #1311'\''s and
#1294'\''s own pure-primitive shapes. Widen `optional_group_is_scope_safe`
and these stop being distributed, which is exactly the "invalid path
component" failure #1311 was filed for. The two error rows also pin that
the group'\''s `?` still does not leak onto what follows it -- the failure
#1294 fixed, and the thing an over-correction here would re-break.

Refs #2909
The review found a **crash class** my own 756-combination sweep missed:
`del(.a | (. | .[])?)` overflowed the stack (exit 134) where jq and the
merge-base both answer `{"a":{},"c":[1,2]}`. Six shapes, jq and yq mode
alike.

Cause: I wrote the rule twice. `optional_group_is_scope_safe` skipped
`Identity` and recursed into `Pipe`/`Paren`; the slice form counted
`Identity` and treated a `Paren(Pipe(..))` element as multi-valued. So a
group could be *safe* to the routing gate -- staying on the native walker --
and *unsafe* to `splice_optional_group`, which then kept it opaque by
returning `[Optional(Pipe(inner))] ++ rest`: the very list the caller had
just decomposed. The walker re-derived the same `here` and re-entered.

That is this repo's own "duplicated predicates diverge silently" lesson
(#106), in a commit whose message claims "one predicate". There is one walk
now, with one entry point.

The second half of the fix is to stop gating `splice_optional_group` at
all. Gating it is what makes it a fixed point, and it turns out to be
unnecessary: `needs_path_prepass` already answers `true` for a group that
can fan out, so such a group is resolved through `resolve_optional_sink`
before any walker sees it. Verified by re-running the sweep with that site
ungated -- widened to 3,840 combinations, with the review's crash shapes and
`.`/paren-heavy randomised bodies added: zero output differences, zero
crashes. Six of those shapes are now CLI tests, which fail loudly rather
than subtly if either half regresses.

Three documentation findings, all mine:

* `is_atomic_path_component`'s comment asserted that an opaque group "never
  reaches this walker anyway" as the safety argument for the whole change.
  The crash proved it was an assertion, not an invariant. It now names the
  routing gate as the actual mechanism and points at
  `splice_optional_group`'s own comment for why that site stays ungated.
* `push_path_components` had lost its doc comment -- absorbed into the new
  predicate's rustdoc, the same defect `b6abf5d76` fixed for
  `resolve_leaf`. Restored.
* `path_component_is_single_valued`'s lead question was inverted relative to
  its return value.

Refs #2909
…per arm

Patch coverage flagged the `Optional`/`Paren` arm as never hit, and the
call graph says why: `walk_optional_group`, its only caller, has already
looked through both wrappers by the time it asks. The arm was dead the
moment the two scope-safety predicates were merged into one walk.

Removed rather than tolerated -- the function collapses to a flat
`matches!` -- with the reason recorded so it is not re-added by symmetry
with the walk above it.

Refs #2909
@newhoggy
newhoggy force-pushed the issue-2909-optional-group-scope branch from 99d35e5 to 46bf851 Compare September 14, 2026 05:18
@newhoggy
newhoggy added this pull request to the merge queue Sep 14, 2026
Merged via the queue into main with commit c76a4f3 Sep 14, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

jq: a ?-wrapped generator piped from .. in path mode resumes after the error, so its side effects over-fire

1 participant