Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions docs/compliance/jq/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -6005,10 +6005,14 @@ fix or a closed decision.

`jq::eval_owned_with_file_index` is the one public entry that hands the
path-context machinery an `OwnedValue` the caller built rather than one read
from a document, so it is the only way a bare `Float`, a NaN, or a
`NumberLiteral` longer than `REINDEX_LITERAL_LEN_CAP` (256 chars) can reach
`eval::eval_path_context_pipe_owned`. That door refuses the reindex bridge for
exactly those values (`reindex_bridge_is_identity`) and runs the pipe through
from a document, so it is the only way a NaN or a `NumberLiteral` longer than
`REINDEX_LITERAL_LEN_CAP` (256 chars) can reach
`eval::eval_path_context_pipe_owned`. (A bare `Float` used to be a third such
class; #2902 gave `to_json_for_reindex` a token spelling that survives the
reindex round trip intact, so a finite computed float is bridge-identity now
and takes the ordinary bridge instead of this door.) That door refuses the
reindex bridge for exactly those two remaining classes
(`reindex_bridge_is_identity`) and runs the pipe through
`eval_generic::eval_path_context_pipe_detached` instead, which never
serializes -- so `.[0] | parent`, `[.[0] | parent]` and `.[0] | parent | .[1]`
all hand the literal back exactly as the caller spelled it
Expand Down
39 changes: 31 additions & 8 deletions docs/compliance/yq/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -3896,11 +3896,11 @@ its text (`numeric_display_string`, so `==` and `tostring` agree about a compute
`True`/`yes`), `"~" == null` is `true` here (yq `false`), and a leading-zero, hex or
underscored integer compares by its resolved decimal text (`1 == 01` is `true` here,
`false` in yq). Every one of these already shows in `tostring`.
- **The reindex bridge re-spells a computed float**: `[(0.5+0.5)] | .[0] == 1` is `true`
in yq and `false` here, because the bridge serializes the computed `Float(1.0)` as
`1.0` and the text rule then sees `1.0` -- the same pre-existing artefact behind
`[(0.5+0.5)] | .[0] | tostring` printing `"1.0"` (yq `"1"`). `(0.5+0.5) == 1` itself,
which never crosses the bridge, is `true` in both.

A third entry used to sit here -- the reindex bridge re-spelling a computed float, so that
`[(0.5+0.5)] | .[0] == 1` was `false` (yq `true`) and `[(0.5+0.5)] | .[0] | tostring`
printed `1.0` (yq `1`). Fixed by [#2902](https://github.com/rust-works/succinctly/issues/2902):
the bridge now writes a bare `Float` as a token the reparse hands back as a bare `Float`.

Pinned by the `yq_text_equality_2785` module (`tests/yq_cli_tests.rs`), the
`scalar_text_equality_2785`/`scalar_wildcard_equality_2785` goldens, and the
Expand Down Expand Up @@ -4239,6 +4239,26 @@ output is unaffected, since neither appears in JSON
for the feature-level gaps (position builtins after DOM conversion; `file_index`/`key`/
`document_index` inside object literals or `any`/`all`).

## `tostring`/`tojson` render a container compactly ([#2902](https://github.com/rust-works/succinctly/issues/2902), found while verifying)

Real yq's `tostring` on a container is its YAML encoder (block style, `!!float` tags
where the spelling needs one), and its `tojson` is its indented JSON encoder, which ends
every string it produces -- scalar or container -- with a newline of its own. succinctly
renders both compactly and without that trailing newline. Captured live (v4.53.3, `null`
on stdin):

| filter | real yq | succinctly yq |
|--------------------------|------------------------|---------------|
| `[1.0] \| tostring` | `- 1.0` | `[1.0]` |
| `[0.5+0.5] \| tostring` | `- !!float 1` | `[1.0]` |
| `[1] \| tojson` | `[\n 1\n]\n` + `\n` | `[1]` |
| `"x" \| tojson` | `"x"\n` + `\n` | `"x"` |
| `(0.5+0.5) \| tojson` | `1.0\n` + `\n` | `1.0` |

Scalar `tostring` agrees in both (the #2902 fix above); the container spellings and the
trailing newline are pre-existing and not attempted there. The `tojson` tests in
`tests/yq_cli_tests.rs` compare trimmed output for this reason.

## Evaluator resource caps apply in yq mode too, and are uncatchable (#2132)

The five caps `succinctly jq` documents -- `MAX_RANGE`, `WHILE_UNTIL_MAX_STEPS`,
Expand Down Expand Up @@ -4365,9 +4385,12 @@ Until #2052 the flag validated through `serde_json::Value` and materialized thro
(see the jq-mode limitations doc for why) and moved yq's materialization onto the
`JsonIndex` + `to_owned_canonicalizing_numbers_at_depth` pair its `--input-format json` path
already uses. #978's convention is intact -- `--argjson` still discards a literal's source
spelling (`1.500` is `1.5`, `1.0` is `1.0`, `0099999999999999999999999` is `1e+23`) and
still does not preserve it the way `succinctly jq`'s own `--argjson` does (#1058 was
deliberately jq-mode-only).
spelling (`1.500` is `1.5`, `0099999999999999999999999` is `1e+23`) and still does not
preserve it the way `succinctly jq`'s own `--argjson` does (#1058 was deliberately
jq-mode-only). Since [#2902](https://github.com/rust-works/succinctly/issues/2902) that same
canonicalizer types a whole-valued float as an int, as real yq's own JSON decoder does
(`1.0` is `!!int` under `-p json`), so `--argjson x 1.0` and `1.e5` render as `1` and
`100000` where they rendered `1.0` and `100000.0` before.

Two renderings do move, both because `serde_json` is no longer the one producing them
(review of #2880 -- the PR text originally claimed nothing moved):
Expand Down
35 changes: 20 additions & 15 deletions src/bin/succinctly/yq_runner.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1686,21 +1686,26 @@ fn stream_yaml_sort_keys_alias_fallback<W: Write>(
/// RFC-8259 "null" substitution, wrong for this purely-internal round
/// trip), matching `eval_owned_input`'s identical reindex bridge for
/// `reduce`/`foreach` (#561, #472).
/// - Only the *fallback* arm — a plain, already-literal-less `Float` — is
/// `S`-gated, and `YqSemantics` there is actively wrong for this call
/// site specifically: `parse_input`'s `--input-format json` path already
/// collapses every number straight to a plain `Float`/`Int` via
/// `to_owned_canonicalizing_numbers` (#978, matching real yq's "a
/// JSON-sourced number never keeps its own spelling" convention) *before*
/// the value ever reaches here — forcing `YqSemantics`'s decimal point back onto
/// that already-canonicalized value reintroduced exactly the bug #978
/// fixed (`--slurp --input-format json '.'` on `{"a":1e2}` regressed from
/// `[{"a":100}]` to `[{"a":100.0}]`, caught by CI). `JqSemantics`'s bare
/// fallback (no forced point) is correct for both the JSON-canonicalized
/// case and the untouched-overflow-scalar case (the latter is already
/// lossy through this whole-document round trip regardless of the point —
/// confirmed live, real yq keeps an untouched i64-overflow scalar
/// byte-for-byte via `-i`, e.g. `99999999999999999999` verbatim, which
/// - The *fallback* arm -- a plain, already-literal-less `Float` -- used to
/// be `S`-gated, and `YqSemantics` there was actively wrong for this call
/// site: `parse_input`'s `--input-format json` path collapses every
/// number to a plain `Int`/`Float` via `to_owned_canonicalizing_numbers`
/// (#978, matching real yq's "a JSON-sourced number never keeps its own
/// spelling" convention) *before* the value reaches here, and forcing
/// yq's decimal point back onto it reintroduced the bug #978 fixed
/// (`--slurp --input-format json '.'` on `{"a":1e2}` regressed from
/// `[{"a":100}]` to `[{"a":100.0}]`, caught by CI). Since #2902 the
/// bridge writes a bare `Float` as a mode-independent token that
/// reparses to the same bare `Float`, so `S` no longer changes the
/// round trip at all; `JqSemantics` stays only because nothing here
/// evaluates under it. That same change removed the accident this route
/// used to rely on -- jq mode spelled a whole `Float(100.0)` as `100`,
/// so it came back an *integer* literal and printed `a: 100` like real
/// yq -- which is why `from_number_literal_plain` (the #978 canonicalizer)
/// now types a whole-valued JSON float as `Int` itself, as yq's own
/// decoder does. An untouched i64-overflow scalar is already lossy
/// through this whole-document round trip regardless (confirmed live:
/// real yq keeps `99999999999999999999` byte-for-byte via `-i`, which
/// this reindex-through-`f64` architecture cannot match either way).
fn evaluate_input(
input: &OwnedValue,
Expand Down
16 changes: 16 additions & 0 deletions src/jq/document.rs
Original file line number Diff line number Diff line change
Expand Up @@ -970,6 +970,22 @@ pub trait DocumentValue: Sized + Clone {
None
}

/// The value this number token carries if it is the reindex bridge's
/// computed-float token (`crate::json::validate::computed_float_token`,
/// #2902), `None` for every other value.
///
/// A materializer must consult this *before* [`number_literal`](Self::number_literal)
/// / [`as_i64`](Self::as_i64) / [`as_f64`](Self::as_f64): the token
/// deliberately fails the first two and decodes through the third, so a
/// chain that falls through to `as_f64` and then records document
/// provenance (`OwnedValue::from_document_float`) would re-bake the
/// computed value into a decimal literal past yq's threshold -- the very
/// re-spelling the token exists to prevent. Only a JSON document can
/// hold one (the bridge re-indexes as JSON), so the default is `None`.
fn bridge_computed_float(&self) -> Option<f64> {
None
}

/// Try to get as a string.
fn as_str(&self) -> Option<Cow<'_, str>>;

Expand Down
Loading
Loading