Archive compiled CREATE and refresh SQL in metadata tables (#83)

[ila]

Implements #83 (automatic archive of compiled SQL in metadata tables). Draft: not ready to merge until two independent feature reviews, any repairs, exact-revision validation and all CI workflows are done.

What it does

• Always-on archive. Every CREATE MATERIALIZED VIEW and executed refresh program is stored in openivm_compiled_programs (one row per version) and openivm_compiled_statements (statements in execution order). The tables sit in the view's metadata catalog/schema, are created through the existing metadata init/migration path, and are independent of openivm_files_path. File export still works and can be used alongside the archive.
• Versioning. Programs are keyed by the qualified MV identity (view_name key plus catalog, schema and SQL name) and by operation (create/refresh). A new version is stored only when the complete statement list differs verbatim from the latest stored version. Nothing is normalized; all versions are kept, including across DROP and recreate. Refresh ids match openivm_refresh_profile.refresh_id.
• Outcomes.
  • committed: the transaction committed.
  • attempted: the run failed and rolled back.
  • unknown: a cross-catalog refresh failed after data statements ran, or COMMIT itself failed.
  • Compile-only calls (openivm_compile_with_facts) are never archived.
• Transaction boundaries.
  • Native CREATE and autocommit refresh: the archive write is inside the operation's transaction. If the write fails, the operation rolls back with an error.
  • Explicit transactions: the write is part of the caller's program, so it rolls back with the caller.
  • Failed autocommit refresh: one separate metadata transaction records attempted/unknown. If that write also fails, the error says so.
  • DuckLake refresh and CREATE OR REPLACE: one metadata transaction after the commit. If it fails, the operation returns an error that says it committed, never plain success.
• Batching. Writes are batched per operation; there is never a commit per statement. Connection-executed writes use bound parameters, because DuckDB parses long SQL literals slowly (~5 ms per 7 KB).

Behaviour to be aware of

• Native incremental refresh programs embed the delta cutoff as a timestamp literal, and DuckLake programs embed snapshot ids. Without normalization, practically every successful incremental refresh stores a new version, about 7 KB in the benchmark workload. Retention is unbounded by design; prune with SQL as documented.
• Unchanged incremental programs do recur when the state is unchanged, for example a retry after a failed attempt. Full-recompute programs also repeat, reusing one version.

Tests (all through the real archive paths)

All on the final binary:

• metadata.test: 396 assertions.
• transactional_lifecycle.test: 546 assertions.
• ducklake_deltas.test: 238 assertions.

They cover CREATE, repeated refreshes, an unchanged program run twice (one version, text unchanged), changed programs (new version, with text and order asserted), full refresh, skipped refresh, compile-only, file export on/off, qualified and attached-catalog names, failure/rollback, explicit transactions, archive-write failures (native, DuckLake refresh, DuckLake replace), reopen, migration, and DROP/recreate.

CI on head 8cc4f984: all workflows pass (Linux amd64/arm64, macOS amd64/arm64, Windows MSVC/MinGW, Wasm ×3, Format, Tidy). The first CI run (47d82787) failed only on Windows, at one new assertion: the reference file is written in text mode (CRLF), so the comparison now normalizes the file's newlines. The archived text is unchanged.

Local full suite on the final revision: 91/92 test files passed (14,214 assertions); the remaining file, refresh_timezone.test, was skipped because it requires icu. Both integration scripts pass. It ran per file in batches because one monolithic run exceeded the session's 2-hour limit on a loaded machine.

Benchmark (loaded machine, noisy)

• Workload: 200 × (insert 100 rows + refresh), interleaved against the previous-commit binary.
• Wall time: mean 23.2 s vs 26.7 s (about +15%). The profiler shows the archive step at 8–12 ms per refresh, out of 110–120 ms.
• window_running_aggregate_incremental.test: CPU time 22.4 s vs 27.8 s; wall time unchanged.

Found, not fixed (out of scope)

Dropping and recreating an aggregate MV after a database reopen fails with Could not commit creation of dependency ... openivm_index has been deleted. It reproduces identically on the baseline binary.

🤖 Generated with Claude Code

Reviewed head: 2c94d92; base: 9f65757
Independent review and configured validation passed.

Performance comparisons: not applicable to this non-performance task.

Required runtime checks incomplete: timeout is non-blocking, not a pass.

• sf10-validation: passed; limit 30000s; evidence: https://github.com/mdrakiburrahman/ivm-bench/actions/runs/38638965642; https://api.github.com/repos/mdrakiburrahman/ivm-bench/actions/artifacts/11930441970/zip; sha256:7d1974926c27df60c775e00cd79ed24ee05e25b9ee1ce360098425b322a86f27; pinned models 49 in each of three batches; archive /Users/ila/Code/agent-mistress/.runtime/gci-artifacts/run-38638965642-wzsm4o9d/results.zip
• rewriter: timeout — incomplete, not passed; limit 120s; evidence: /Users/ila/Code/agent-mistress/.runtime/runs/15f6794a99d2-v5ud7syb/result.json;

Two independent fresh reviewer sessions passed this exact revision.
Validation scope: configured command only; no additional runtime coverage claimed
Validation command: ["/Users/ila/Code/agent-mistress/.venv/bin/python", "/Users/ila/Code/agent-mistress/scripts/general_validation.py", "9f657576d1e47440d7e62de1befd47bf76b4a83e"]

Fixes #83