jq: a `?`-wrapped generator piped from `..` in path mode resumes after the error, so its side effects over-fire

[newhoggy]

Severity: Low (side-effect count; output matches)

Summary

In path mode only, and only when the ?-wrapped generator is reached through .., an error
inside the generator does not backtrack out of the enclosing .[] — evaluation resumes with
the next element, so a side effect ahead of the error fires once per element instead of once.

Repro

Confirmed live against jq 1.7.1 and succinctly at a953112f2; pre-existing (the same
divergence reproduces on 1b12aa528, before #2694's streaming work, so it is not caused by it):

$ echo '[true,false]' | jq -c '[path(.. | (.[]|stderr|.+0)?)]'
true[]

$ echo '[true,false]' | succinctly jq -c '[path(.. | (.[]|stderr|.+0)?)]'
truefalse[]

.+0 on a boolean raises; jq's try aborts the whole body, so .[] is never resumed and
stderr writes once. Here .[] continues to the second element and writes twice.

Three neighbouring spellings all agree with jq, which is what localises it to the
..-piped path-mode route rather than to ? or to .[]:

$ echo '[true,false]' | jq -c '[(.[]|stderr|.+0)?]'            # true[]   — both agree
$ echo '[true,false]' | jq -c '[.. | (.[]|stderr|.+0)?]'       # true[]   — both agree (value mode)
$ echo '[true,false]' | jq -c '[path((.[]|stderr|.+0)?)]'      # true[]   — both agree

So the difference appears only when the ? sits downstream of .. and inside path().
Likely the node .. delivers is not the document's own tracked value, routing the ? through
a different arm than the standalone path((...)?) spelling takes
(resolve_optional_sink vs. whatever the piped stage reaches).

Fix direction

Find which arm the .. | stage routes the Optional through and make its error handling stop
the enclosing generator, as resolve_optional_sink already does for the standalone spelling.
Needs a live-oracle differential sweep against jq 1.7.1 per ADR-0018 before landing; the
neighbouring rows above are the regression guards.

[newhoggy]

Triage plan (2026-09-13)

Investigated on db974b52e with cargo build --release --features cli, oracle /usr/bin/jq 1.7.1.
The premise needs correcting in three ways, all widening the issue.

Premise check — what the repro actually localises to

1. .. is not required. Any pipe does it:

$ echo '[true,false]' | jq -c        '[path(. | (.[]|stderr|.+0)?)]'
true[]
$ echo '[true,false]' | succinctly jq -c '[path(. | (.[]|stderr|.+0)?)]'
truefalse[]

2. path() is not required, and the output diverges — this is not a side-effect-count bug.
del/=/|= diverge on a bare ?-group with no enclosing pipe, no .., and no error(...):

$ echo '{"a":{"b":1},"c":[1,2]}' | jq -c         'del((.[] | .[0])?)'
{"a":{"b":1},"c":[1,2]}
$ echo '{"a":{"b":1},"c":[1,2]}' | succinctly jq -c 'del((.[] | .[0])?)'
{"a":{"b":1},"c":[2]}          <- silently deletes an element jq keeps

$ echo '[{"a":1},5,{"a":3}]' | jq -c         '((.[] | .a)?) |= 99'
[{"a":99},5,{"a":3}]
$ echo '[{"a":1},5,{"a":3}]' | succinctly jq -c '((.[] | .a)?) |= 99'
[{"a":99},5,{"a":99}]          <- writes a slot jq never reaches

So severity is Medium, not Low: a silent wrong write / wrong delete, plus duplicated output
paths ([path(.. | ((.[], .a) | .b)?)] on {"a":{"b":1},"c":[1,2]} is [["a","b"]] in jq and
[["a","b"],["a","b"],["a","a","b"]] here).

3. It is not resolve_optional_sink vs "whatever the piped stage reaches". The ? never
reaches a ?-handling arm at all in the diverging shapes — it is dissolved before resolution.

A differential sweep of 462 combinations (11 group bodies x 7 outer spellings x 6 inputs,
jq 1.7.1 vs this build) found 19 divergences, every one of them a ? over a
multi-component group containing a generator. Zero divergences for a ? over a
multi-component group of single-valued primitives ((.a|.b)?, (.a[0:1])? | .[0]) — which is
exactly the class #1294/#1311 were about.

Root cause — one rule, applied at two independent sites

? over a group is try (group): a failure anywhere inside prunes the whole group's branch
and stops the group's own generators. succinctly instead distributes the ? onto each
component of the group, turning one abort scope into N independent per-step ones — so a
generator inside the group resumes past an error a later component raised.

Both distribution sites were added to fix a real, different problem (don't let the group's ?
leak onto what follows the group), and both are correct for the pure-navigation groups they were
written for:

• Route A — the resolver. push_path_components's Expr::Optional arm
  (jq: a ? on a multi-component group ahead of more path still hits 'invalid path component' for = #1311) rewrites Optional(Pipe([Iterate, Field a])) into
  [Optional(Iterate), Optional(Field a)]. resolve_seq_stage then resolves each as its own
  stage; the failing stage returns ResolveFlow::Exhausted (pruned), which the Iterate stage
  reads as Demand::Continue and resumes. Reached whenever needs_path_prepass(expr) is true
  — which is what .. and error("x") in the repro were doing, nothing more.
• Route B — the write walkers. splice_optional_group (jq: update_path's Expr::Pipe splice over-suppresses a ? guard past its own group #1294) does the
  identical rewrite for update_path/delete_at_path, and flatten_path_components ->
  push_path_components does it for set_path_steps. Reached when needs_path_prepass(expr)
  is false — which is why del((.[]|.a)?) / |= / = diverge while path((.[]|.a)?) on the
  same expression does not: path() on that route uses walk_path, which handles the group as
  one opaque unit and is already correct.

That two-route split explains every row of the sweep, including why the issue's own three
"neighbouring spellings agree" rows agree: path((.[]|stderr|.+0)?) has no enclosing Pipe, so
resolve_node_sink's Expr::Optional arm catches it first and resolve_optional_sink gives it
jq's single abort scope. That arm is already correct and is the fix's reference behaviour —
the work is getting the two distributing sites to route into it instead of around it.

Plan

1. One shared predicate, one definition (CLAUDE.md's "duplicated predicates diverge
   silently"): optional_group_is_scope_safe(inner) -> bool — true when the ?'s inner
   flattens to fewer than two path components, or when every component is single-valued
   (Identity | Field | Index | Slice, seen through Paren/Optional). Distribution is
   observationally identical to one abort scope exactly in that class: a single-valued chain has
   one branch, so pruning at step k and aborting the group both emit nothing for it. Write it
   without allocating (it is consulted from needs_path_prepass, which is on the per-call path
   for every path/del/=/|=).

2. Gate both distribution sites on it. push_path_components's Expr::Optional arm and
   splice_optional_group distribute only when safe; otherwise the group stays one opaque
   Optional(Pipe([...])) element.

3. Keep the unsafe shape off the native walkers. Extend the Expr::Optional arm of
   needs_path_prepass and needs_fanout_pass to return true when the group is not
   scope-safe. Both are needed and for different reasons: needs_path_prepass is what routes
   del/=/|=/path through the resolver at all (gates at resolve_del_path_branches and
   resolve_dynamic_indexes), and needs_fanout_pass currently peels the Optional before
   testing, so Optional(Pipe([Iterate, Field])) answers false and would be mistaken for
   resolve_seq_sink's single-valued static tail. With both set, the opaque element reaches
   resolve_against_cow_sink -> resolve_node_sink's Expr::Optional arm ->
   resolve_optional_sink — the arm that is already right.

4. The newly-reachable arm. wrap_optional_branch's components.depth() != 1 branch is
   annotated "unreached today ... postfix ? attaches to a single path element until jq: postfix ? is only accepted after a path expression #367" and
   carries an omni-dev: coverage tolerate-line. An opaque multi-component group makes it live
   (the group resolves to two-plus components under one ?). Drop the annotation, keep
   flatten_components, and cover it directly. Its doc comment already predicts this shape and
   states jq's answer for it, so no new semantics are needed.

5. Audit the other push_path_components callers (~10, incl. set_path_steps's
   flatten_path_components, resolve_dynamic_indexes's own top-level flatten, the del trie
   builder, resolve_del_path_branches) and confirm step 3 means none of them can still be
   handed an un-distributed group — a walker that matches components element-wise would report
   "invalid path component" for one. If any can, give that site the legacy (distributing)
   behaviour explicitly via the same predicate rather than by omission.

6. Fix the prose that asserts the old invariant: the doc comments on
   push_path_components, splice_optional_group, needs_fanout_pass ("flat ... by
   construction never contains a raw Pipe"), set_path_steps and is_atomic_path_component
   all state the distribution as unconditional.

Tests

• Unit table in src/jq/eval.rs's tests module, all rows captured live from jq 1.7.1:
  • the issue's [path(.. | (.[]|stderr|.+0)?)] plus the .-piped form that also fails;
  • the three write shapes: del((.[] | .[0])?), ((.[] | .a)?) |= 99, ((.[] | .a)?) = 99;
  • the duplicate-output row [path(.. | ((.[], .a) | .b)?)];
  • the issue's three already-passing neighbours, as the "do not regress" half;
  • the scope-safe class the gate must leave alone — ((.a[0:1])? | .[0]) = 9 (jq: a ? on a multi-component group ahead of more path still hits 'invalid path component' for = #1311's own
    test, test_assign_optional_group_with_more_path_1311), (.a|.c)[0] = 9 (jq: update_path's Expr::Pipe splice over-suppresses a ? guard past its own group #1294's), and
    del((.a[0:1])?) in tests/yq_cli_tests.rs. All three use pure-primitive groups, so they
    are the direct guard on the gate's lower edge.
• The 462-combination differential sweep re-run to zero divergences, per ADR-0018 and the
  "hand-picked oracle matrix is biased" rule — extend it with randomised group bodies rather
  than relying on the hand-picked 11.
• succinctly yq needs no oracle row: real yq's lexer rejects (.[]|.a)? outright
  (1:15: lexer: invalid input text "?)"), so there is no yq behaviour to match. The evaluator
  is shared, so yq mode inherits the jq-mode fix; add a yq row only to pin that it stays
  consistent, not as a fidelity claim.
• No limitations.md entry: this is a bug against jq, not a permitted divergence.

Risks

• Perf. Step 3 moves exactly one shape (? over a multi-component group containing a
  generator) from the native walker onto the resolver. needs_path_prepass's doc comment records
  that widening it carelessly once cost 250-1500x on del(.foo[]); that shape has no ? group
  and is untouched here. Still worth an A/B on del/|= over a large container to confirm the
  common paths are flat, with a baseline that predates the branch.
• Blast radius of push_path_components. Ten callers. Step 5 is the mitigation; if the audit
  turns up a caller that genuinely needs distribution for an unsafe group, prefer giving that one
  site the explicit legacy mode over widening the gate.
• Over-conservative gate. Treating any non-primitive component as unsafe also catches groups
  that never diverge today ((.[] | .a?)). That only changes which correct arm handles them;
  the tests above pin that the answers do not move.

Class: Opus (keep)

Not Sonnet: it is not localised or mechanical — the fix spans two independent distribution
mechanisms (resolver pre-pass and write walkers), two route-selection predicates, and a
currently-dead arm it makes live, with the correct behaviour re-derived from a 462-row oracle
matrix rather than read off the issue.

Not Fable: no ADR, no new representation, no cross-cutting invariant change. The reference arm
(resolve_optional_sink) already exists and is correct; the work is routing into it and gating
an existing rewrite, not designing new semantics.

[newhoggy]

🔧 Claimed by agent Neville — branch issue-2909-optional-group-scope

Premise re-verified against /usr/bin/jq 1.7.1 before claiming: [path(.. | (.[]|stderr|.+0)?)] diverges and all three neighbouring spellings agree, exactly as filed. Working the 2026-09-13 triage plan, including its three corrections to the premise — in particular the silent wrong-write/wrong-delete rows, which make this a correctness issue rather than a side-effect count.