Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 167 additions & 0 deletions docs/compliance/jq/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -6288,6 +6288,173 @@ change (`eval_label`/`each_label`/`each_label_generic` and both owned-identity/p
`Label` arms would each need their non-matching-escape fallthrough routed through a resolvable
call), out of #2687's stated scope.

### A module `include` cycle is a compile error, where jq segfaults — accepted divergence, ADR-0018 rule 4 (#2865)

[#2865](https://github.com/rust-works/succinctly/issues/2865) made a module's own
`include`/`import` directives load transitively, which makes a cycle between two modules
reachable for the first time. Real jq 1.7.1 does not diagnose one at all — it recurses
until the process dies:

```console
$ cat ca.jq
include "cb";
def a: 1;
$ cat cb.jq
include "ca";
def b: 2;

$ jq -L . -n 'include "ca"; a'; echo "exit=$?"
exit=139 # SIGSEGV, nothing on stdout or stderr

$ succinctly jq -L . -n 'include "ca"; a'; echo "exit=$?"
jq: error: module cycle detected: ca -> cb -> ca

jq: 1 compile error
exit=3
```

A module that includes itself behaves the same way in jq (`exit=139`), and reports
`module cycle detected: selfinc -> selfinc` here.

This is the **cleanest** of ADR-0018 rule 4's carve-outs rather than a policy stretch:
the rule permits refusing the reference's behaviour where "matching would take the host
process down," and matching here means exactly a SIGSEGV. There is no reference *output*
to be faithful to — jq writes nothing to either stream — so the only open question was
which shape to leave through, and the answer is the one the other two compile-error kinds
already use (`jq: N compile error`, exit 3, per "Undefined functions and arity
mismatches" above).

Detection keys on the **resolved** file rather than the module path as written, so one
module reachable under two spellings still closes a cycle
(`alia.jq` containing `include "./alia"` reports `alia -> ./alia`); the chain in the
message keeps the spellings, since those are what the source actually says. It cannot use
the module memo cache as its guard: a module is absent from that cache for exactly as long
as its own dependencies are loading, which is precisely the window in which a cycle closes.

`test_module_cycle_is_a_compile_error_not_a_hang_2865` (`tests/jq_cli_tests.rs`) pins all
four shapes (two-module cycle, self-include, aliased spelling, `import`-side cycle).

### Seven module-scoping rules that *are* matched, and read as bugs (#2865)

Not divergences — recorded here because the next person to touch `ModuleLoader` will
otherwise read them as ones, and because the exclusions in `visible_deps_for` have no
other explanation. All captured live against jq 1.7.1, with `inner.jq` = `def g: 42;`:

1. **A dependency outranks the module's own same-name sibling.** With the module written
`include "inner"; def g: 7; def h: g;`, `h` answers **`42`** — not the `7` one line
above it. The dependency is bound innermost.
2. **...while that module still exports its own `g`**, which answers `7`.
3. **The same collision at the top level goes the opposite way.** `include "inner"; def
g: 7; g` answers `7`, because a filter's own defs bind at parse time, before the
module block is spliced in.
4. **A def's own recursive call binds to itself, not to a same-named dependency, per
(name, arity).** With `inner`'s `g/0` in scope, a module's
`def g: if . == 0 then "base" else (. - 1 | g) end;` still answers `"base"`, never
`42`. Arity-scoped: a dependency `g/1` alongside an own `g/0` leaves both reachable
from the same body (`["own0","dep1arg"]`).
5. **A parameter beats both.** `def f(g): g; def q: f(7);` answers `7` even with a
dependency or a sibling named `g` in scope, and `def f($g): [$g, g]` answers `[7,7]`
— a `$`-spelled parameter binds the bare call-site namespace too.

6. **A module's own def keeps the bindings it was written under**, whatever the def
that calls it declares. `def f: 1; def k: f; def h(f): k;` answers `1` for `h(99)` —
`k`'s `f` is the module's, not `h`'s parameter. `def h: "first"; def g: h; def h:
"second-" + g;` answers `"second-first"` — `g`'s `h` is the first one. `def a: length;
def h(length): a;` answers `2` for `h(9)` on `[1,2]` — `a`'s `length` is the builtin.
7. **...including when the name is defined nowhere.** `def a: b; def h(b): a;` is
`b/0 is not defined`, exit 3, not something `h`'s parameter can satisfy.

Rule 4 is why an exported def's body is *not* wrapped in a dependency matching its own
(name, arity), and rule 5 is why it is not wrapped in one matching any of its parameters —
both scoped to a name the body calls *directly*, since a dependency reached only through
another one is bound where that one was written and is not the def's own to shadow.
Rules 6 and 7 are why a module's own defs are not wrapped into each other at all: they are
emitted as siblings in the top-level chain, exactly as a filter's own defs are, and jq's
lexical rule relates them there. Nesting a copy of one inside another's body puts it under
scopes it was never written in, and rule 6's third row shows sealing cannot repair that —
a call to a builtin is free in every pre-bound form of the copy.
Rules 1-3 are why the wrap goes around each exported def's **body** rather than being
spliced into the module's exported chain.

One consequence worth stating, since it is the reason an exported body carries its
module's own earlier siblings as well as its dependencies: a def handed to another module
has to be **self-contained**. With `inner.jq` = `def g: 42; def k: g;` and `outer.jq` =
`include "inner"; def g: k;`, jq answers `42` — `k`'s `g` is `inner`'s. Leaving `k`'s `g`
to resolve outward into wherever `k` gets spliced would instead find `outer`'s own `g`,
which is `k`.

### Deeply chained modules compound in memory (#2955)

Binding copies the AST, so a chain of modules whose defs each call **more than one** def
from the level below grows exponentially with depth. jq binds symbolically and shares its
blocks, so it does not. Measured at #2865's own head (Apple M-series, release):

| chain | succinctly peak RSS | jq peak RSS |
|----------------------------------|---------------------|-------------|
| 6 levels x 40 defs, 1 call each | 10 MB | 2.5 MB |
| 8 levels x 6 defs, 3 calls each | 91 MB | 2.6 MB |
| 14 levels x 4 defs, 2 calls each | 361 MB | 2.6 MB |

The referenced-closure filter (jq's own `block_bind_referenced` rule, in
`visible_deps_for`) flattens the one-call-each shape completely — without it the
6 x 40 row was 359 MB rather than 10 MB — but it cannot flatten a genuinely wide closure,
because that closure is itself exponential. Every row above produces the **correct**
answer; this is a scalability limit of AST inlining, not a wrong result. It needs a
*chain of modules* to appear, and a wide one: only dependencies are wrapped into a body,
so a module with no `include`/`import` is bound exactly as cheaply as before #2865
however many of its own defs call each other (a 24-def Fibonacci module is 9 MB, against
`main`'s 9 MB), and a chain whose defs call one def apiece stays flat (21 levels of a
two-def module is 9 MB; a 7-level 40-def chain is 12 MB). Tracked as
[#2955](https://github.com/rust-works/succinctly/issues/2955), whose most promising fix is
splicing bound bodies by handle (the `Rc`-shaded opaque sub-expression #1371 already
introduced) instead of by clone.

### A wrapped dependency sits inside the including def's scope — no carve-out; recorded as a still-open gap (#2962)

A module's dependencies are bound by wrapping them around the body of each def that
reaches them, and that wrap nests **inside** the def's own `Expr::FuncDef` — so the def's
own name (for self-recursion) and its parameters are enclosing binders for every
dependency in the block. Real jq binds a module's block in its own scope and only then
links it, so nothing of the caller is ever in scope for it.

`visible_deps_for`'s two exclusions are a partial mitigation: they keep a def's own
recursion and its parameters working for names the body uses **directly**. They cannot
help a name a dependency reaches on its own, and the two cases pull opposite ways —
excluding strands the other dependency, keeping it would shadow the def's own binding.
Three shapes, all confirmed live against jq 1.7.1:

| fixtures | jq 1.7.1 | succinctly |
|----------|----------|------------|
| `inner` = `def c: 7; def g: c;`, `mid` = `include "inner"; def c: if . == 0 then g else (. - 1 \| c) end;`, then `0 \| c` | `7` | `g/0 exceeded maximum recursion depth`, exit 5 |
| `gb` = `def g: b;`, `hb` = `include "gb"; def h(b): g; def q: h(99);`, then `q` | `b/0 is not defined`, exit 3 | `99`, exit 0 |
| `inner3` = `def g: 42; def k: [g];`, `h3` = `include "inner3"; def h($g): [g, k]; def q: h(7);`, then `q` | `[7,[42]]` | `[7,[7]]` |

None of ADR-0018's four conditions covers this (the output is readable, nothing is
corrupted or discarded, and the process does not die), so per rule 4 it is recorded here
as a still-open gap rather than an accepted divergence. It is **not** a regression: every
shape needs a transitive `include`, which did not work at all before #2865 — `main`
answers `g/0 is not defined` / `k/0 is not defined` for each. Closing it needs either a
targeted rename of an excluded dependency (rewriting free calls to it inside the other
kept dependency bodies) or the sealed module scope of
[#2951](https://github.com/rust-works/succinctly/issues/2951), which subsumes it; the
second row additionally needs a module's own body resolved at load time, the way jq
reports it against the module's own file.
`test_dependency_capture_by_the_including_defs_scope_2962` (`tests/jq_cli_tests.rs`) pins
all three so a change to the mechanism cannot make them worse unnoticed.

### Two module-scope gaps that are genuinely open

Found while closing #2865, filed rather than recorded as divergences: a dependency named
after a builtin cannot shadow that builtin *inside* the module body, because a module's
own source is parsed with no shadow-candidate seeding
([#2950](https://github.com/rust-works/succinctly/issues/2950)); and a module body can see
names it should not — `~/.jq`'s defs, and sibling `include`d modules' defs in a
declaration-order-dependent way — because every module is inlined into one flat def chain
([#2951](https://github.com/rust-works/succinctly/issues/2951)). Data imports
(`import "f" as $d;`) are also still unimplemented: #2865 records the `$` on `Import::data`
and resolves the file so a typo is still jq's own `module not found`, but binding the
variable is [#2956](https://github.com/rust-works/succinctly/issues/2956).

## Provenance

| Artifact | Path |
Expand Down
17 changes: 17 additions & 0 deletions docs/reference/jq-language.md
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,23 @@ as opposed to a trailing same-line comment) are not implemented at all.
- [x] `~/.jq` auto-loading (file or directory)
- [x] `namespace::func` - Namespaced function calls
- [x] Parameterized functions in modules
- [x] Transitive `include`/`import` — a module's own directives are processed too (#2865)

A module's dependencies are bound into the bodies of the defs that module
exports, not spliced into its exported chain, which is what real jq does:
a transitively included name is visible *inside* the module and is **not**
re-exported to whoever included it, and it outranks a same-named sibling
def in that module (while the module still exports its own). A def's own
recursive call binds to itself rather than to a same-named dependency,
per (name, arity), and a module's own defs keep the bindings they were
written under however the def that calls them is declared. See
[jq Limitations](../compliance/jq/limitations.md#seven-module-scoping-rules-that-are-matched-and-read-as-bugs-2865)
for the full table and the two scoping gaps that remain open.

An `include` cycle is reported as `module cycle detected: a -> b -> a`
(exit 3) — a deliberate ADR-0018 rule-4 divergence, since real jq
segfaults instead; see
[jq Limitations](../compliance/jq/limitations.md#a-module-include-cycle-is-a-compile-error-where-jq-segfaults--accepted-divergence-adr-0018-rule-4-2865).

### Succinctly Extensions
These are succinctly-specific extensions not available in standard jq or yq:
Expand Down
Loading
Loading