jq: a module's own include directive is silently ignored (transitive include not processed)

[newhoggy]

Severity: Low

Summary

Found while investigating #2774 (module-sourced $__loc__ provenance). A
module loaded via include can itself contain its own include directive,
naming a third module. Real jq processes that nested include transitively,
making the third module's defs available to the second module's own body.
succinctly's ModuleLoader does not: it only calls extract_func_defs on a
loaded module's parsed Program::expr, silently dropping that Program's own
includes/imports lists.

Repro

Confirmed live against jq 1.7.1:

$ cat inner.jq
def g: 42;
$ cat outer.jq
include "inner";
def h: g;

$ jq            -L . -nc 'include "outer"; h'   # 42
$ succinctly jq -L . -nc 'include "outer"; h'   # jq: error: g/0 is not defined at <top-level>

Root cause

ModuleLoader::ensure_module_loaded (src/bin/succinctly/jq_runner.rs) does:

let program = jq::parse_program(&contents)...?;
Ok(entry.insert(extract_func_defs(&program.expr)))

jq::parse_program returns a full Program (with its own includes,
imports, and module fields, exactly like the top-level parse), but only
program.expr is inspected. A module's own include/import directives are
parsed successfully and then discarded -- h's reference to g is therefore
genuinely undefined by the time succinctly's resolve pass sees it, since g
was never inlined into outer.jq's own body at all.

Why this is separate from #2774

#2774 is about what path a module-sourced $__loc__ reports once a module
is loaded; this is about a module failing to load its own transitive
dependencies at all, a functional gap rather than a diagnostic-wording one.

Suggested fix direction

ensure_module_loaded/load_module need to recurse: after parsing a
module's Program, resolve and inline that module's own includes (bare
names, merged into its scope) and imports (namespaced) before extracting
its function defs -- essentially the same logic ModuleLoader::process_program
already does for the top-level program, applied recursively to each loaded
module. Watch for cycles (a.jq includes b.jq includes a.jq) -- confirm
what jq itself does there (a compile error, most likely) before choosing a
behavior, and consider loaded_modules' existing per-path cache as a natural
place to detect an in-progress load.

Found via adversarial review of #2774 (PR pending).

[newhoggy]

Triage plan (2026-09-13)

Premise confirmed on main @ db974b52e against the pinned oracle (/usr/bin/jq 1.7.1) and
target/release/succinctly. The issue's root-cause reading is exactly right — but its
suggested fix direction ("apply process_program recursively") would produce the wrong
answer, and the cycle question it flags for investigation turns out to need an ADR-0018 rule-4
decision. Details below.

Fixture set used for every row

$ cat inner.jq                 # def g: 42;
$ cat outer.jq                 # include "inner";  def h: g;

Oracle matrix (all rows captured live, jq 1.7.1 vs current main)

#	Case	jq 1.7.1	succinctly today
1	include "outer"; h — transitive include	42	g/0 is not defined
2	import "inner" as i; inside outer, then h	42	i::g/0 is not defined
3	include "outer"; g — does the transitive def leak to the includer?	compile error (no leak)	n/a (nothing loaded)
4	module has include "inner"; def g: 7; def h: g; — h	42 (the dep wins)	7
5	…and that module still exports its own g	7	7
6	…but a def's own self-recursion still binds to itself, not to the dep	"base"	—
7	shadowing is per (name, arity): dep g/1 + own g/0 → [h, i]	["own0","dep1arg"]	—
8	top level: include "inner"; def g: 7; g — opposite of row 4	7 (own def wins)	7 ✓
9	two includes in a module defining the same name	last-declared wins	—
10	sibling own defs inside a module bind lexically (def x:1;def y:x;def x:2;)	[1,2]	[1,2] ✓
11	nested include resolved relative to the including module's directory?	no — global search path only	—
12	2-level chain, and diamond (dia includes inner and outer)	works / 84	—
13	is ~/.jq visible inside a module body?	no — compile error	1234 (leaks — see F1)
14	cycle ca ⇄ cb, and self-include	SIGSEGV, exit 139	1 (dep silently dropped)
15	unreferenced dep def whose body calls an undefined function	no error	no error ✓
16	$__loc__ inside a transitively included def	the inner module's path	—

Rows 4/6/7/8 are the substance of this issue. jq's linker binds a module's references in the
order self-recursion → dependency → lexical sibling, and only then splices the module block
into the including program (whose own defs bind at parse time, hence row 8 going the other way).
A naive "inline the module's deps into the module's def chain, then export the chain" gets
rows 3, 4 and 6 wrong at once.

Root cause (confirmed)

ModuleLoader::ensure_module_loaded (src/bin/succinctly/jq_runner.rs) ends with

Ok(entry.insert(extract_and_stamp_func_defs(&program.expr, file_path)))

program.includes / program.imports are parsed and discarded. Note the entry() spelling
also structurally forbids the fix: it holds a mutable borrow of loaded_modules across the
body, so the loader cannot re-enter itself to load a dependency.

Plan

1. Make the loader re-entrant. Replace ensure_module_loaded's entry() with
   contains_key → load → insert → get(..).expect("just inserted"). The existing doc
   comment justifying entry() ("the contains_key spelling would need an unreachable
   expect") must be rewritten: recursion is now the overriding constraint, and that expect
   comes back.

2. Cycle guard. Add loading: Vec<String> to ModuleLoader; push the module path before
   resolving its dependencies, pop on both the Ok and Err exits. A path already on the stack
   is a new ModuleLoadError::Cycle { chain }, rendered by report_module_load_error as

   jq: error: module cycle detected: ca -> cb -> ca
   
   jq: 1 compile error
   

   exit 3 (exit_codes::COMPILE_ERROR), same shape as the existing NotFound arm. The
   memo cache is not a sufficient guard on its own — a module is not in loaded_modules
   while its own dependencies are loading, which is precisely when the cycle closes.
   This is an ADR-0018 rule-4 divergence under the explicit "matching would take the host
   process down" carve-out (row 14: real jq segfaults), so it is a clean grant, not a
   policy stretch — but it must still be recorded (see Docs).

3. Load the module's own deps, and wrap them into each exported def's body. In the module
   load path, after parsing:

   let own  = extract_and_stamp_func_defs(&program.expr, file_path);  // stamp FIRST -- see R2
   let deps = self.module_dep_defs(&program)?;                        // recursion happens here
   own.into_iter().map(|(name, params, body)| {
       let body = wrap_defs(body, deps_excluding_self(&deps, &name, params.len()));
       (name, params, body)
   }).collect()

   Wrapping the bodies rather than the chain is what buys rows 1/2 (dep visible inside the
   module), row 3 (dep not exported), row 4 (dep innermost, so it beats the module's own
   same-name sibling) and row 5 (the module's own def is still what gets exported) from one
   mechanism, with no special-casing.

   deps_excluding_self drops any dep whose (name, arity) equals the def being wrapped —
   that is row 6 (self-recursion binds to itself) and row 7 (arity-scoped). Without this filter
   recshadow.jq's exported g answers 42 where jq answers "base".

   module_dep_defs reuses the search path verbatim — row 11 says jq resolves a nested
   include against the global search path only, never $ORIGIN of the including module, so
   the simple implementation is also the faithful one, and a missing transitive dep already
   reports jq's byte-identical module not found: deep shape through the existing NotFound
   arm.

4. One definition of the wrap ordering. Factor process_program's three near-identical
   expr = Expr::FuncDef { .., then: expr } loops and the new call site into a shared
   wrap_defs, preserving the documented net effect (last-declared include innermost, hence
   winning — row 9, which holds inside a module too). Three copies of this ordering rule is
   exactly the "duplicated predicates diverge silently" trap CLAUDE.md calls out.

5. ~/.jq is deliberately not wrapped into module bodies (row 13). It leaks in today via
   the outer chain; closing that is F1 below, not this issue.

6. Size guard (conditional, measure before implementing). Per-body wrapping clones each dep
   body once per exported def: a module with N exported defs over M dep defs costs N×M def
   clones, multiplying down a chain. Build a synthetic 3-level chain (40 defs per level), time
   succinctly jq -L . -n 'include "top"; f' and check peak RSS. If it moves materially, adopt
   jq's own block_bind_referenced rule: wrap only deps in the transitive closure of names
   actually called from the body (Expr::FuncCall name, plus Expr::NamespacedCall as
   ns::name). Row 15 shows this is a pure optimization, not a fidelity requirement — an
   unreferenced dep with a broken body errors in neither tool.

Tests (tests/jq_cli_tests.rs, tempdir + -L, the file's existing module-test idiom)

One test per oracle row 1, 2, 3, 4+5 (same fixture), 6+7, 9, 11 (both the resolves-globally
and the not-found control), 12 (chain and diamond), 14 (cycle and self-include: exit 3,
chosen message, and it terminates), 16 ($__loc__ names the inner module's canonical path —
the #2774 regression this could silently break). Row 8 and row 10 already pass; add them as
locks since step 3/4 moves the code that decides them.

Docs

• docs/compliance/jq/limitations.md, new ### under Deliberate divergences (ADR-0018
  rule 4): the cycle error, with the jq-segfaults transcript as the carve-out evidence.
• The same section (or a short note next to it) should record rows 4/6/8 as matched quirks —
  they read as bugs to the next person who touches the loader, and the self-recursion filter in
  step 3 has no other explanation.

Risks

• R1 — borrow rework. Step 1 reintroduces an expect on the re-lookup; unavoidable, needs
  the doc comment updated rather than left contradicting the code.
• R2 — $__loc__ stamping order. stamp_loc_file (src/jq/walk.rs) overwrites file
  unconditionally, so wrapping must happen after stamping or the inner module's $__loc__
  silently acquires the outer module's path (row 16 regresses, breaking jq: $__loc__ inside an included module reports the top-level filter's file, not the module's #2774). Consider making
  it first-stamp-wins as defence in depth, with a walk.rs unit test.
• R3 — shadow candidates. unqualified_def_names must not gain transitive names: they are
  not visible unqualified at the top level (row 3), so yq: jq: a def loaded via include/import/~/.jq cannot shadow a builtin (#2036 follow-up) #2395's seeding is unchanged. Worth an
  explicit test so a later "helpful" edit doesn't add them.
• R4 — deep chains. Legit deep chains now recurse through ensure_module_loaded; frames are
  small and the step-2 guard bounds the pathological case, so no depth cap is proposed.

Follow-ups found while planning (not in scope; say the word and I'll file them)

• F1 — module scope leaks the other way. Inside a module body, succinctly resolves names it
  should not see: ~/.jq's defs (row 13), and sibling included modules' defs in a
  declaration-order-dependent way — include "sibB"; include "sibA"; sa answers 77 where jq
  reports sb/0 is not defined at .../sibA.jq, and reversing the two includes makes
  succinctly agree with jq by accident. Closing this needs a sealed module scope, which AST
  inlining cannot express without name mangling or a resolver-side boundary — a design change,
  likely Fable. The fix here is forward-compatible with either.
• F2 — a module's own source is parsed by plain jq::parse_program, with no
  shadow-candidate seeding from its deps' def names, so a dep named after a builtin/keyword
  (limit, range) cannot shadow inside the module the way yq: jq: a def loaded via include/import/~/.jq cannot shadow a builtin (#2036 follow-up) #2395 makes it shadow at top level.
  Pre-existing, widened slightly in reach by this fix.

Model class: Opus (unchanged)

Localised to one file and one struct, which reads Sonnet-ish — but the work is not mechanical:
the semantics had to be re-derived from a 16-row oracle matrix in which three rows contradict
the obvious implementation (rows 3, 4 and 6 all break the issue's own suggested fix
direction), the cycle case needs an ADR-0018 rule-4 ruling plus a limitations entry, and the
sizing call in step 6 needs a measurement rather than a guess. That is squarely the
"semantics re-derived from an oracle matrix / ADR-0018 fork resolution" band. Not Fable: no
ADR, no new representation, no cross-cutting invariant — F1, the sealed-scope sibling, is the
Fable-class piece and is deliberately carved out.

[newhoggy]

🔧 Claimed by agent Miles — branch issue-2865-transitive-include

[newhoggy]

Follow-ups filed while implementing this, all cross-linked from PR #2954:

• jq: a module's own source is parsed with no shadow-candidate seeding, so a transitively included def can't shadow a builtin inside that module #2950 — a module's own source is parsed with no shadow-candidate seeding, so a dependency named after a builtin cannot shadow it inside the module body (the module-level twin of yq: jq: a def loaded via include/import/~/.jq cannot shadow a builtin (#2036 follow-up) #2395).
• jq: a module body sees names it should not — ~/.jq's defs, and sibling included modules' defs in declaration order #2951 — a module body sees ~/.jq's defs and sibling modules' defs, which real jq keeps out; needs a real scope boundary.
• perf(jq): transitively included modules compound in memory when a module's defs each call more than one def below #2955 — a chain of modules whose defs each call more than one def below compounds in memory, because binding copies the AST where jq shares blocks.
• jq: data imports (import "f" as $d;) are unimplemented — the $ is dropped at parse time, so they resolve as module imports and fail #2956 — data imports (import "f" as $d;) are unimplemented; the $ was dropped at parse time, which this PR fixes by recording Import::data.
• jq: rewrite_namespaced_calls skips computed destructuring keys, so . as {(m::f): $v} fails with "module 'm' not loaded" #2957 — rewrite_namespaced_calls (and walk::uses_input_builtins) skip computed destructuring keys, the same stale-Pattern assumption this PR fixed in its own dependency walk.
• jq: a wrapped module dependency sits inside the including def's scope, so that def's own name or parameters can capture names free in it #2962 — the boundary of this PR's own mechanism: a wrapped dependency sits inside the including def's scope, so that def's name or parameters can capture a name left free in the dependency. Recorded in docs/compliance/jq/limitations.md.