fix(query): size registration to the host, private spill dir, no root, no extension installs at query time

[laithalsaadoon]

Closes the five open findings of the MicroVM defensive review of main 4571e14 (reach-ranked findings 1 to 5) and pins findings 6 and 7 (closed by #8) with their exact reproductions.

Finding 1: query could not register on a host with under ~1.9 GiB RAM per core

Cause, measured: DuckDB's eager read_json readers reserve about 2x maximum_object_size per thread, so the 1 GiB constant cost 2 GiB a thread. On this host SET threads=4; SET memory_limit='6GB' then register() failed (6.0 GiB/5.5 GiB used), as did 16 threads under 25 GB; 16 MiB read the same rows in 0.10 s. Two or more edges.jsonl files are enough to trigger it (one file registers on main, two don't).

Fix: maximum_object_size is sized from the largest file each reader opens (plus a quarter and 1 MiB, floored at DuckDB's 16 MiB default, capped at 1 GiB). query applies memory_limit, threads and temp_directory BEFORE register(). The cap derives from available RAM (MemAvailable, 80% ceilings, 8 GiB target, 512 MiB floor) and the thread count from the cap (one per 2 GiB, capped at sched_getaffinity). ATIF_SQL_QUERY_MEMORY_LIMIT and ATIF_SQL_QUERY_THREADS override both; malformed values exit 64. _query_memory_limit_bytes is the same function re-derived (no duplicate).

Proof on the 300-session frozen corpus (/tmp/findings-corpus, 2.1 GB, LITELLM_LOCAL_MODEL_COST_MAP=true PYTHONHASHSEED=0), 3 reps each, /usr/bin/time:

run	count	tool_rank	tokens_by_source	RSS
before, unlimited	0.98 / 0.92 / 0.92 s	0.96 / 0.90 / 0.89 s	0.90 / 0.92 / 0.88 s	479 to 492 MB
before, 6GB + 4 threads	exit 70 out of memory	exit 70	exit 70	n/a
after, unlimited	0.77 / 0.74 / 0.73 s	0.73 / 0.78 / 0.75 s	0.77 / 0.77 / 0.79 s	548 to 565 MB
after, 6GB + 4 threads	0.80 / 0.80 / 0.78 s	0.75 / 0.76 / 0.75 s	0.79 / 0.78 / 0.78 s	426 to 440 MB

All six after-fix outputs are byte-identical to the before-fix unlimited outputs (cmp). The unlimited case got faster because the lance extension is no longer loaded when no store exists (0.2 s). Tests: TestQueryResources::test_registration_fits_under_limits_that_used_to_oom[4-6GB|16-8GB], TestRegistrationUnderHostLimits (registry), TestObjectSizeBound, the cap/thread/parse/env-override tests.

Finding 2: root could overwrite a granted parquet with COPY ... (USE_TMP_FILE false)

Investigated as asked: DuckDB 1.5.5 has no read-only grant. duckdb_settings() lists no write switch, and a read_parquet relation bound before enable_external_access=false is refused at query time without a grant, so the per-file grants must stay and they're read-write. Two layers close it:

• query, search and analyze refuse uid 0 with exit 77 root_refused unless ATIF_SQL_ALLOW_ROOT=1 (a warning is logged).
• query refuses every file-facing statement kind before executing anything, using DuckDB's own extract_statements on the hardened connection: COPY, COPY_DATABASE, EXPORT, ATTACH, DETACH, INSTALL, LOAD, PREPARE, EXECUTE (an allowlist, so a new kind fails closed). Exit 70, kind sandbox_refused, for any uid; a batch containing one runs nothing.

Probe d2 on the real corpus: before exit 0 as root / exit 70 IO Error as uid 1000; after exit 70 sandbox_refused for any uid and steps.parquet sha unchanged. Tests: TestQueryStatementGate, TestQueryRefusesRoot, TestQuerySandboxWrites::test_use_tmp_file_false_over_a_granted_parquet_is_refused_for_any_uid, and TestHardenedConnectionLayer proves DuckDB's own layer separately and pins the read-write grant fact the gate exists for.

Finding 3: caller SQL could create and read files in <corpus>/.duckdb_tmp, persisting

Fix: a per-process tempfile.mkdtemp (0700) under the system temp dir is the spill directory and the sole allowed_directories entry; removed on every exit path after the connection closes. Nothing under the corpus is writable. Probe d3 (COPY into a pre-created .duckdb_tmp): before exit 0 and the file persisted; after exit 70 sandbox_refused, tree digest unchanged. Tests: TestQuerySpillDirectory (in-process, plus a real python -m atif_cli process asserting the corpus tree hash is unchanged, no .duckdb_tmp appears, and no spill dir outlives the process).

Finding 4: query downloaded a 242 MB extension over the network during registration

Fix: register_vss never installs. It LOADs lance only when duckdb_extensions() says it's installed, skips the load entirely when no store directory exists, and otherwise binds message_embeddings empty with a warning. query sets autoinstall_known_extensions=false and autoload_known_extensions=false before registration (both exist in duckdb 1.5.5). Installing is explicit: atif-sql embed --install-extension, or a real embed run. atif-sql status reports vector_search as ready, no_store or extension_missing; search exits 78 extension_missing instead of pretending the store is empty. Tests: TestQueryNeverInstallsExtensions (store present, empty extension_directory, recorded statements contain no INSTALL, directory stays empty), TestVssNeverInstalls (registry), embed install tests.

Finding 5: sandbox settings and the session inventory are readable from caller SQL

Assessed: DuckDB cannot hide duckdb_settings() / current_setting(), and the grants must be per file for lazy reads to work, so the inventory is a property of the design. Documented as accepted in query's docstring, docs/reference/cli.md, and SECURITY.md: the caller is the local user who can list the corpus and SELECT session_id FROM sessions anyway, and query is not a privilege boundary.

Findings 6 and 7 (closed by #8), pinned

TestFilenamesThatOnceDestroyedTheCorpus: ..jsonl and *.jsonl over three materialized victims are rejected (('*', '.')) and the tree is untouched. TestGlobShapedNamesDoNotMultiplyRows: session dirs named * and ??? count each session once on both read paths and grant no glob.

Fails on main, passes here

A standalone reproduction file with six tests was run on a scratch checkout of origin/main (4571e14): 6 failed there, 6 passed on this branch. mise run check green (1473 passed, 1 skipped; up from about 1399), mise run docs:gate green.