jq: two path() shapes still collect after #2908 — the reindex-bridge route and a generator in index position

[newhoggy]

Severity: Low (side-effect count under a bound; delivered values are correct)

Summary

#2908 made path(f) stream its
results, so a bound outside path(...) stops the generator inside it. Two shapes still
collect, both found by /code-review on PR #2923 and both unchanged from before that PR
— it did not introduce either, it just did not reach them.

1. The reindex-bridge route

eval_each_generic's new arm mirrors the eager arm's own two gates, one of which sends a
document the reindex bridge would not round-trip identically (any Float, or a number literal
too large to survive the trip) to eval_on_owned — which collects.

$ python3 -c "print('{\"a\":1,\"b\":2,\"n\":' + '9'*300 + '}')" | jq -c \
    '[limit(1; path((.a|stderr),(.b|stderr)))]'
1
[["a"]]
# succinctly: 12

Drop the 9*300 and the same filter writes 1, matching jq — the document, not the filter,
is what selects the route.

2. A generator in index position

The stop reaches the branch producer but not a generator supplying an index key:

$ echo '{"a":1,"b":2}' | jq -c '[limit(1; path(.[("a"|stderr),("b"|stderr)]))]'
a
[["a"]]
# succinctly: ab

resolve_index_expr evaluates the key generator eagerly, so this is adjacent to
#2032 (E[K] evaluates its target
once where jq re-runs it per key) — same function, different axis.

Fix direction

1. is a routing question: either give the bridge route a streaming entry (it re-enters the
   full evaluator, which now has a lazy Builtin::Path arm — the entry point
   eval_on_owned is what collects), or narrow reindex_bridge_is_identity so fewer
   documents need the bridge at all.
2. needs the key generator driven by demand inside resolve_index_expr, which is where jq: E[K] evaluates its target once where jq re-runs it per key (side-effect count) #2032
   already sits.

Both need live-oracle differential verification against jq 1.7.1 per ADR-0018. The rows above
are the regression guards, and both are pinned by
path_results_stream_to_their_consumer_2908's residual block, which asserts they still differ
so closing either trips the test.

[newhoggy]

Half of this closes in PR #2972 (#2267's remaining step).

That PR makes Expr::IndexExpr/Expr::SliceExpr native resolve_node_sink arms — the
last two resolve_node_eager ones — so a consumer's demand reaches the key and bound
generators. This issue's second recorded shape, "the stop reaches the branch producer
but not a generator in index position", now matches jq 1.7.1:

$ echo '{"a":1,"b":2}' | succinctly jq -c '[limit(1; path(.[("a"|stderr),("b"|stderr)]))]'
a                 # was `ab`; jq writes `a`
[["a"]]

and so does the slice sibling, which this issue did not record:

$ echo '[1,2,3]' | succinctly jq -c '[limit(1; path(.[(0|stderr):((1|stderr),(2|stderr))]))]'
01                # jq writes `01`
[[{"start":0,"end":1}]]

path_results_stream_to_their_consumer_2908 is written to fail when either of this
issue's two shapes closes, and it did — both rows above are now in that test's main table
with jq's own stderr.

The first shape is untouched and this issue stays open on it. A document the reindex
bridge will not round-trip identically still takes the bridge, which collects, so the
document selects the route rather than the filter:

$ echo '{"a":1,"b":2,"n":999…(300 nines)}' | succinctly jq -c '[limit(1; path((.a|stderr),(.b|stderr)))]'
12                # jq writes `1`

That case remains as the residual in the same test, now as a single case rather than a
loop, with the same "if it closed, move it into the table above" assertion guarding it.

[newhoggy]

Triage plan (2026-09-17)

Premise check on current main (b509a8019)

Shape 2 (a generator in index position) closed with #2267 / PR #2972, as the earlier comment says. Shape 1 still reproduces, but the trigger is narrower than the body says. Since #2902 a bare Float survives the bridge unchanged, so reindex_bridge_is_identity accepts every float. Every NaN spelling I tried also stays on the streaming route. The only document that still reaches the collecting route is one holding a number literal longer than REINDEX_LITERAL_LEN_CAP (256 characters):

# the literal is 256 nines, then 257 nines
256: succinctly 1[["a"]]   jq 1[["a"]]
257: succinctly 12[["a"]]  jq 1[["a"]]

$ S='[limit(1; path((.a|stderr),(.b|stderr)))]'
# {"a":1,"b":1.5}              -> 1   (float: streams)
# yq `b: .nan` / `.NaN` / `nan` -> 1   (streams)

On a document with a 300-digit literal, every bounded consumer over-runs, not only limit:

filter	jq 1.7.1	succinctly
[limit(1; path((.a|stderr),(.b|stderr)))]	1[["a"]]	12[["a"]]
first(path((.a|stderr),(.b|stderr)))	1["a"]	12["a"]
[label $o | path((.a|stderr),(.c|stderr)) | ., break $o]	{"b":1}[["a"]]	{"b":1}1[["a"]]
isempty(path((.a|stderr),(.b|stderr)))	1false	1false (agrees)
path(.a.b, .c.d) (prefix, then error)	agrees	agrees

syq --jq-extensions shows the same over-run on a 300-digit YAML scalar ([limit(1; path((.a|stderr),(.n|stderr)))] writes both).

Root cause

eval_each_generic's lazy Builtin::Path arm (src/jq/eval_generic.rs, the #2908 arm) sends a document that fails reindex_bridge_is_identity down this route:

return drain_result_generic(eval_on_owned::<S, _>(&builtin_expr, owned, optional), sink);

eval_on_owned goes through eval::eval_full → eval_single → builtin_path, which collects every path. drain_result_generic only hands the finished list to sink. The collecting happens at that entry point, not across the bridge as a whole: eval.rs's own eval_each already has a lazy Builtin::Path arm (#2908) that calls each_path.

A demand-forwarding version of this bridge already exists. eval::eval_each_owned makes the same round trip (to_json_for_reindex + JsonIndex::build), then calls eval_each instead of eval_full. bridge_to_each_owned_flow already relies on that equivalence for isempty/any/all/IN, which explains why isempty agrees in the table above. eval_owned_fast_path has no Builtin::Path arm, so path(..) always reaches eval_each.

Plan

1. Swap the entry point in the lazy arm. In eval_each_generic's Builtin::Path arm, replace the non-identity branch with:

   return eval_each_owned::<S>(&builtin_expr, &owned, optional, &mut |v| {
       sink.push(GenericItem::Owned(v))
   });

   Keep the gate itself. The same reindexed document feeds the same resolver (builtin_path and each_path both call to_owned and then resolve and walk with optional), so every delivered value, error and prefix stays as it is today. Only demand changes. Leave the eager eval_single Builtin::Path arm alone. Its consumer already collects.

2. Drop the residual guard. In path_results_stream_to_their_consumer_2908 (tests/jq_cli_tests.rs), delete the assert_ne!/our_err residual block and move the case into the main table with jq's stderr (1).

3. Add over-cap rows to that table, each captured whole from /usr/bin/jq 1.7.1 (stdout, stderr, exit code), using a 300-digit and a 257-digit literal:

   • limit(1; …), first(…), limit(2; …) over three probes
   • the label/break row
   • the jq: path()'s DOM walker drops an output prefix jq keeps when a nested pipe stage errors #2680 prefix-then-error row (path(.a.b, .c.d)) and its first(...) twin
   • [path(...)?]
   • the ?// retry row

   Also add a 256-digit row as a must-not-change guard at the cap boundary. It streams today through the identity route.

4. Add a yq-mode row. Put it in the matching yq CLI test, or add one next to it: syq --jq-extensions '[limit(1; path((.a|stderr),(.n|stderr)))]' over a 300-digit scalar. path(f) doesn't exist in real yq v4.53.3 (bad expression), so this is an extension under --jq-extensions and jq's generator order is the model.

5. Update the docs.

   • Rewrite the arm's comment ("keeps the collecting behaviour, a recorded residual (jq: two path() shapes still collect after #2908 — the reindex-bridge route and a generator in index position #2925)") to say the bridge now streams.
   • In docs/compliance/jq/limitations.md, rewrite the "Two shapes still collect" paragraph (currently next to the jq: path()'s own consumer side still collects, so a bound outside path() can't stop the fold (#2694 residual) #2908 bullet). Both shapes are closed; also drop its stale "any Float" wording.
   • Fix the stale parenthetical in eval.rs's builtin_path doc ("a Float anywhere in the document"). A float no longer reaches that route; only an over-cap or NaN literal does.

Verification

• Run the checks listed in ci.yml: cargo test --features cli (includes tests/*.rs), jq_evaluator_parity_tests, yq_golden_tests, and cargo clippy --all-targets --all-features -- -D warnings.
• Run the jq: path()'s own consumer side still collects, so a bound outside path() can't stop the fold (#2694 residual) #2908 test on the base build first to confirm the new over-cap rows fail there. A row that passes on the base proves nothing.
• Differential probe: run a small matrix of bounded consumers × {300-, 257-, 256-digit, float, plain} documents against /usr/bin/jq. Record it in the PR.

Out of scope, and a separate bug found while triaging

The bridge's value fidelity is wrong on the same documents, independently of demand. to_json_for_reindex drops an over-cap literal's source text (the #1211 per-iteration cost cap) and respells it as 1E+300. Any filter evaluated across a reindex round trip then sees a different number. This isn't specific to path():

filter (on {"a":1,"b":2,"n":<300 nines>})	jq 1.7.1	succinctly
[path(.n|select(tostring|length == 300))]	[["n"]]	[]
[path(.n|select(. == 1e300))]	[]	[["n"]]
to_entries|map(select(.value|tostring|length==300))|length	1	0
del(.[]|select(tostring|length==300))|keys	["a","b"]	["a","b","n"]
(.[]|select(tostring|length==300)) |= 0 | .n	0	1E+300
.n|tostring|length (native, no bridge)	300	300 (agrees)

This plan leaves that behaviour byte-for-byte as it is: step 1 keeps the same bridge. Removing the reindex_bridge_is_identity gate from the path arms wouldn't fix it either: each_path_on_owned evaluates select's condition through eval_each_owned (via resolve_cond_fork_stream), which makes the same round trip on the sub-value. The fix belongs in to_json_for_reindex's cap. It should be filed as its own issue. It is Opus-class: it's a fork between #1211's performance cap and fidelity under ADR-0018, it affects every bridged filter, and it needs an A/B on the #1211 loop workload. Once it lands, over-cap literals become bridge-identity and this route stops seeing them at all.

Dependencies / risks

• No dependency. jq: eager path-resolution vs jq's lazy generator can over/under-fire target side effects when a downstream write fails partway #2267 / PR fix(jq): apply an assignment's writes as its paths resolve (#2267) #2972 (shape 2) has merged.
• Risk is low. eval_each_owned and eval_full feed the same to_owned, resolve and walk. The one known difference is the entry point's demand handling, which bridge_to_each_owned_flow already relies on. The unchanged rows (errors, prefix, ?) cover the error-path equivalence.
• No performance concern. The route only runs for documents with a literal over 256 characters, and it does the same work, stopping earlier.

Class: Sonnet (was Opus)

The body described this as a routing question with two possible designs. With shape 2 closed and the trigger narrowed to over-cap literals, one design remains. It's a single-call swap in one arm, onto a documented demand-forwarding bridge that already has a precedent (bridge_to_each_owned_flow). The fix changes no semantics, all rows come from the pinned oracle, and the change is local to one evaluator's lazy arm. The eager arm and eval.rs stay untouched. The ADR-0018 judgment, the multi-site change and the performance A/B all belong to the separate fidelity bug above, not to this issue.

[newhoggy]

Follow-ups filed from this triage: #3025 (the reindex bridge respells an over-cap number literal — the value-fidelity bug in the plan's "Out of scope" section) and #3026 (pick(pathexps) is a real jq 1.7.1 builtin, not a succinctly extension).