Archive compiled CREATE and refresh SQL in metadata tables (#83)

CMakeLists.txt

@@ -13,6 +13,7 @@ include_directories(src/include ${LPTS_DIR}/src/include ${LPTS_DIR}/third_party/
 set(EXTENSION_SOURCES
     src/openivm_extension.cpp
     src/compile_facts.cpp
+    src/core/compiled_sql_archive.cpp
     src/core/ivm_delta_model.cpp
     src/core/ivm_view_classifier.cpp
     src/core/parser.cpp
@@ -135,6 +136,10 @@ if(NOT EMSCRIPTEN)
 
     add_executable(scheduler_catalog_test test/integration/scheduler_catalog_test.cpp)
     target_link_libraries(scheduler_catalog_test ${OPENIVM_LINK_LIBS})
+
+    add_executable(benchmark_metadata_test test/integration/benchmark_metadata_test.cpp)
+    target_link_libraries(benchmark_metadata_test ${OPENIVM_LINK_LIBS})
+    target_include_directories(benchmark_metadata_test PUBLIC benchmark/src)
 endif()
 
 # Benchmark tools are currently supported on Unix native platforms.

Makefile

@@ -21,3 +21,11 @@ test_release_internal: test_scheduler_catalog
 test_scheduler_catalog:
 	cmake --build build/release --target scheduler_catalog_test
 	python3 test/integration/test_scheduler_catalog.py ./build/release/extension/openivm/scheduler_catalog_test
+
+.PHONY: test_benchmark_metadata
+
+test_release_internal: test_benchmark_metadata
+
+test_benchmark_metadata:
+	cmake --build build/release --target benchmark_metadata_test
+	dir=$$(mktemp -d) && ./build/release/extension/openivm/benchmark_metadata_test $$dir | grep -qx PASS; status=$$?; rm -rf $$dir; exit $$status

README.md

@@ -183,13 +183,15 @@ MVs can be created using any SQL construct. Unsupported operators automatically
   and set `openivm_files_path` before CREATE or refresh. Then call
   `PRAGMA openivm_files('dl.observation.product_summary')` to inspect exact paths.
   This pragma reports files; it does not generate them.
-- **Can compiled SQL be stored in a table?** Yes: select from
-  `openivm_compile_with_facts` into your own table. Automatic archival in OpenIVM
-  metadata is not implemented. Compiled SQL contains state-specific cutoffs;
-  compile again for later refreshes.
+- **Can compiled SQL be stored in a table?** It is archived automatically: every
+  CREATE and executed refresh program is kept in `openivm_compiled_programs` and
+  `openivm_compiled_statements`, with a new version only when the SQL changes. File
+  export stays independent. Compiled SQL contains state-specific cutoffs; it is an
+  inspection record, not a reusable refresh procedure.
 
 See [schema and metadata examples](docs/ducklake.md#schemas-metadata-and-internal-tables),
 [exporting and inspecting SQL files](docs/build/building.md#inspect-the-generated-sql),
+[the compiled SQL archive](docs/build/building.md#archived-compiled-sql),
 and [saving compiled SQL as rows](docs/build/building.md#inspect-or-save-compiled-sql-without-files).
 
 ## Documentation

benchmark/src/benchmark_metadata.hpp

@@ -0,0 +1,85 @@
+#pragma once
+
+// Helpers for benchmarks that read openivm_views. view_name holds an encoded internal key; the
+// SQL-facing name lives in view_sql_name, so a SQL name must be resolved to its key first.
+
+#include "duckdb.hpp"
+#include "duckdb/main/connection.hpp"
+#include "duckdb/parser/keyword_helper.hpp"
+#include "core/refresh_metadata.hpp"
+
+#include <string>
+#include <vector>
+
+namespace openivm_bench {
+
+struct ViewTypeLookup {
+	bool found = false;
+	int64_t type = 0;
+	std::string key;
+	std::string error;
+};
+
+// Resolves the MV named `sql_name` in the connection's current catalog/schema to its metadata key
+// via RefreshMetadata::FindViewKey, then reads its refresh type from the native catalog's metadata.
+// `con` may point at a DuckLake catalog (USE dl.main); metadata always lives in `native_catalog`.
+inline ViewTypeLookup LookupViewType(duckdb::DuckDB &db, duckdb::Connection &con, const std::string &native_catalog,
+                                     const std::string &sql_name) {
+	ViewTypeLookup out;
+	try {
+		auto loc = con.Query("SELECT current_database(), current_schema()");
+		if (!loc || loc->HasError() || loc->RowCount() == 0) {
+			out.error = loc ? loc->GetError() : "no result";
+			return out;
+		}
+		duckdb::Connection meta_con(db);
+		duckdb::RefreshMetadata::UseCatalog(*meta_con.context, meta_con, native_catalog);
+		duckdb::RefreshMetadata metadata(meta_con);
+		out.key = metadata.FindViewKey(loc->GetValue(0, 0).ToString(), loc->GetValue(1, 0).ToString(), sql_name);
+	} catch (std::exception &e) {
+		out.error = e.what();
+		return out;
+	}
+	if (out.key.empty()) {
+		return out;
+	}
+	auto result = con.Query("SELECT type FROM " + duckdb::KeywordHelper::WriteOptionallyQuoted(native_catalog) +
+	                        ".main.openivm_views WHERE view_name = " + duckdb::Value(out.key).ToSQLString());
+	if (!result || result->HasError()) {
+		out.error = result ? result->GetError() : "no result";
+		return out;
+	}
+	if (result->RowCount() > 0) {
+		out.type = result->GetValue(0, 0).GetValue<int64_t>();
+		out.found = true;
+	}
+	return out;
+}
+
+struct LeftoverView {
+	std::string key;
+	std::string catalog;
+	std::string schema;
+};
+
+// Lists metadata rows whose SQL-facing name matches `sql_like` (a LIKE pattern), with the catalog/schema
+// that hold their data table. Must run before DROP VIEW removes the rows.
+inline std::vector<LeftoverView> ListLeftoverViews(duckdb::Connection &con, const std::string &native_catalog,
+                                                   const std::string &sql_like) {
+	std::vector<LeftoverView> out;
+	auto meta = con.Query("SELECT view_name, view_catalog, view_schema FROM " +
+	                      duckdb::KeywordHelper::WriteOptionallyQuoted(native_catalog) +
+	                      ".main.openivm_views WHERE COALESCE(view_sql_name, view_name) LIKE " +
+	                      duckdb::Value(sql_like).ToSQLString());
+	if (meta && !meta->HasError()) {
+		for (duckdb::idx_t r = 0; r < meta->RowCount(); r++) {
+			auto cat = meta->GetValue(1, r);
+			auto sch = meta->GetValue(2, r);
+			out.push_back({meta->GetValue(0, r).ToString(), cat.IsNull() ? native_catalog : cat.ToString(),
+			               sch.IsNull() ? "main" : sch.ToString()});
+		}
+	}
+	return out;
+}
+
+} // namespace openivm_bench

benchmark/src/daemon_benchmark.cpp

@@ -871,10 +871,17 @@ static int RunSubtest(const string &subtest, int duration_s, const string &db_pa
 				con.Query("DROP TABLE IF EXISTS " + full);
 			}
 		}
-		con.Query("DELETE FROM openivm_views WHERE view_name LIKE 'mv_%'");
-		con.Query("DELETE FROM openivm_delta_tables WHERE view_name LIKE 'mv_%'");
-		con.Query("DELETE FROM openivm_refresh_history WHERE view_name LIKE 'mv_%'");
-		con.Query("DELETE FROM openivm_refresh_hooks WHERE view_name LIKE 'mv_%'");
+		// view_name holds an encoded internal key; match the SQL-facing name (view_sql_name) too,
+		// and delete dependent metadata before the openivm_views rows that identify it.
+		const string stale_keys = "SELECT view_name FROM openivm_views WHERE COALESCE(view_sql_name, view_name) LIKE "
+		                          "'mv\\_%' ESCAPE '\\'";
+		for (const char *dependent :
+		     {"openivm_delta_tables", "openivm_refresh_history", "openivm_refresh_hooks"}) {
+			con.Query(string("DELETE FROM ") + dependent + " WHERE view_name LIKE 'mv_%' OR view_name IN (" +
+			          stale_keys + ")");
+		}
+		con.Query("DELETE FROM openivm_views WHERE view_name LIKE 'mv_%' OR COALESCE(view_sql_name, view_name) LIKE "
+		          "'mv\\_%' ESCAPE '\\'");
 
 		// DuckLake attach for S9/S10/S11
 		if (subtest == "S9" || subtest == "S10" || subtest == "S11") {
@@ -1082,8 +1089,9 @@ static int RunSubtest(const string &subtest, int duration_s, const string &db_pa
 					// vs the refresh applied something wrong.
 					string diag;
 					auto dt_r = fin_con.Query(
-					    "SELECT table_name, last_update FROM openivm_delta_tables WHERE view_name = '" +
-					    v.name + "'");
+					    "SELECT table_name, last_update FROM openivm_delta_tables WHERE view_name IN "
+					    "(SELECT view_name FROM openivm_views WHERE lower(COALESCE(view_sql_name, view_name)) = "
+					    "lower(" + duckdb::Value(v.name).ToSQLString() + "))");
 					if (dt_r && !dt_r->HasError()) {
 						for (idx_t di = 0; di < dt_r->RowCount(); di++) {
 							string tn = dt_r->GetValue(0, di).ToString();

benchmark/src/rewriter_benchmark.cpp

@@ -9,6 +9,8 @@
 #include "duckdb/main/connection.hpp"
 #include "duckdb/common/printer.hpp"
 #include "core/openivm_extension.hpp"
+#include "core/refresh_metadata.hpp"
+#include "benchmark_metadata.hpp"
 #include "tpcc_helpers.hpp"
 #include "tpcdi_helpers.hpp"
 
@@ -598,6 +600,9 @@ static void ChildWorkerMain(int read_fd, int write_fd, const string &db_path, co
 		// DuckLake MVs (created with USE dl.main) land in dl.main.*, so we use catalog-qualified
 		// drops via information_schema which spans all catalogs.
 		{
+			// Metadata is keyed by an encoded internal key, not the SQL-facing mv_q* name. Record the
+			// leftover keys (and their data-table locations) before any DROP VIEW removes the rows.
+			auto leftover_keys = openivm_bench::ListLeftoverViews(con, native_catalog, "mv_q%");
 			// Drop views from all catalogs
 			auto leftover = con.Query("SELECT table_catalog, table_name FROM information_schema.views "
 			                          "WHERE table_name LIKE 'mv_q%' AND table_schema = 'main'");
@@ -619,9 +624,21 @@ static void ChildWorkerMain(int read_fd, int write_fd, const string &db_path, co
 					con.Query("DROP TABLE IF EXISTS " + cat + ".main." + tn);
 				}
 			}
-			// Clean metadata tables (always in native catalog, unqualified)
-			con.Query("DELETE FROM openivm_views WHERE view_name LIKE 'mv_q%'");
-			con.Query("DELETE FROM openivm_delta_tables WHERE view_name LIKE 'mv_q%'");
+			// Data tables are named after the internal key, so the LIKE patterns above miss them.
+			for (auto &lv : leftover_keys) {
+				con.Query("DROP TABLE IF EXISTS " + duckdb::KeywordHelper::WriteOptionallyQuoted(lv.catalog) + "." +
+				          duckdb::KeywordHelper::WriteOptionallyQuoted(lv.schema) + "." +
+				          duckdb::KeywordHelper::WriteOptionallyQuoted("openivm_data_" + lv.key));
+				string key_literal = duckdb::Value(lv.key).ToSQLString();
+				con.Query("DELETE FROM " + quoted_native_catalog + ".main.openivm_delta_tables WHERE view_name = " +
+				          key_literal);
+				con.Query("DELETE FROM " + quoted_native_catalog + ".main.openivm_views WHERE view_name = " +
+				          key_literal);
+			}
+			// Clean metadata tables (always in native catalog) for legacy rows keyed by the SQL name
+			con.Query("DELETE FROM " + quoted_native_catalog + ".main.openivm_views WHERE view_name LIKE 'mv_q%'");
+			con.Query("DELETE FROM " + quoted_native_catalog +
+			          ".main.openivm_delta_tables WHERE view_name LIKE 'mv_q%'");
 		}
 
 		int delta_idx = 0;
@@ -744,15 +761,15 @@ static void ChildWorkerMain(int read_fd, int write_fd, const string &db_path, co
 						// Qualify with native_catalog so the lookup works both when the active catalog
 						// is a DuckLake catalog (USE dl.main) and when the DB is file-based (catalog
 						// name = filename, never "memory").
-						auto check_result = con.Query("SELECT type FROM " + quoted_native_catalog +
-						                              ".main.openivm_views WHERE view_name = '" + mv_name + "'");
-						if (check_result && !check_result->HasError() && check_result->RowCount() > 0) {
-							int64_t refresh_type = check_result->GetValue(0, 0).GetValue<int64_t>();
-							is_incremental = (refresh_type != 3) ? 1 : 0;
+						// openivm_views.view_name holds an encoded internal key; the SQL-facing name is in
+						// view_sql_name. Resolve the qualified identity to its key via RefreshMetadata.
+						auto lookup = openivm_bench::LookupViewType(db, con, native_catalog, mv_name);
+						if (lookup.found) {
+							is_incremental = (lookup.type != 3) ? 1 : 0;
 						} else {
 							error = "OpenIVM metadata lookup failed for " + mv_name;
-							if (check_result && check_result->HasError()) {
-								error += ": " + check_result->GetError();
+							if (!lookup.error.empty()) {
+								error += ": " + lookup.error;
 							}
 							phase_reached = PHASE_MV_CREATION_FAILED;
 						}

docs/build/building.md

@@ -200,6 +200,79 @@ that skips empty deltas may not compile a new file. Setting the path after CREAT
 does not retroactively create the setup files; do not replace an existing MV just
 to obtain an inspection file.
 
+### Archived compiled SQL
+
+OpenIVM always archives the programs it runs for `CREATE MATERIALIZED VIEW` and
+refresh in two tables next to the view's `openivm_views` metadata (for DuckLake
+views, the native frontend database's `main` schema), independently of
+`openivm_files_path`:
+
+```sql
+SELECT view_catalog, view_schema, view_sql_name, operation, version,
+       statement_count, last_outcome, committed_count
+FROM openivm_compiled_programs ORDER BY view_name, operation, version;
+
+SELECT s.stmt_order, s.sql
+FROM openivm_compiled_statements s
+JOIN openivm_compiled_programs p USING (view_name, operation, version)
+WHERE p.view_sql_name = 'regional_totals' AND p.operation = 'refresh'
+  AND p.version = (SELECT max(version) FROM openivm_compiled_programs
+                   WHERE view_name = p.view_name AND operation = 'refresh')
+ORDER BY s.stmt_order;
+```
+
+- **Identity.** `view_name` is the internal key also used by `openivm_views`;
+  `view_catalog`, `view_schema` and `view_sql_name` give the qualified MV name.
+  `operation` is `create` or `refresh`. `compilation_id` identifies the operation that
+  first stored a version; `last_compilation_id` is the latest operation that ran
+  the same program. For refreshes outside an explicit transaction this is the
+  `refresh_id` in `openivm_refresh_profile` when profiling is enabled; refreshes
+  inside an explicit transaction are not profiled.
+- **Versions.** A program is the complete statement list, in execution order. A
+  new version is stored only when that list differs from the view's latest stored
+  version for the same operation. The comparison is exact: delta cutoffs, snapshot
+  IDs and catalog references are not normalized away. A refresh inside an explicit
+  transaction, for example, stamps its own timestamp, and DuckLake refreshes embed
+  snapshot IDs, so these usually store a new version. Refreshes skipped because
+  deltas are empty compile nothing and store nothing. If a DuckLake refresh rolls
+  back its first delete attempt and retries with ranked deletion, only the retried
+  program that committed is stored. For snapshot-published DuckLake views, the
+  executed publication statement is stored between the data statements and the
+  watermark update. Statements OpenIVM derives while it runs, such as DuckLake
+  delta-table definitions and pinned snapshot publications, are stored as the SQL
+  that actually ran. Cleanup of consumed delta rows after a refresh commits is
+  housekeeping and is not part of the stored program.
+- **Outcomes.** `last_outcome` describes the latest operation that ran the version:
+  `committed` (its transaction committed), `attempted` (it failed and rolled back,
+  so nothing was committed) or `unknown` (a cross-catalog refresh failed after its
+  first statements ran, or `COMMIT` itself failed, so some effects may have
+  committed). This includes a DuckLake refresh whose data statements roll back,
+  because its native in-progress marker has already committed.
+  `committed_count` counts committed runs. Compile-only calls such as
+  `openivm_compile_with_facts` are never archived, so a stored program has always
+  been executed.
+- **Transactions.** Native CREATE and refresh write the archive in their own
+  transaction, so it is visible only if they commit. A failed or rolled-back native
+  operation in an explicit transaction leaves no row. An autocommit refresh that
+  fails is recorded as `attempted` (or `unknown`) in a separate metadata
+  transaction. DuckLake data cannot share a transaction with native metadata, so
+  cross-catalog operations write the archive on separate connections: refreshes
+  in one metadata transaction after they commit (or fail), even inside an explicit
+  transaction, and DuckLake CREATE with its native metadata.
+- **Archive failures.** OpenIVM does not report success if the archive cannot be
+  written. For native operations the archive write is part of the transaction, so
+  a failed write rolls back the CREATE or refresh with an error. If a
+  cross-catalog refresh or DuckLake `CREATE OR REPLACE` commits but its archive
+  write fails, it returns an error that says it committed. If recording a failed refresh also fails, both
+  errors are reported.
+- **Retention.** All versions are kept until you delete them. History survives
+  `DROP VIEW` and `CREATE OR REPLACE`, and a recreated view continues the same
+  version sequence. To prune, delete matching
+  `(view_name, operation, version)` rows from both tables.
+
+Archived SQL is for inspection and auditing. Later refreshes always compile against
+current state; do not run an archived program to refresh a view.
+
 ### Inspect or save compiled SQL without files
 
 The compiler also returns SQL as rows. This works without `openivm_files_path`:
@@ -217,7 +290,7 @@ SELECT * FROM openivm_compile_with_facts(
 
 This compiles a refresh without executing it or consuming the pending changes.
 The saved table is an ordinary user table, not automatically maintained OpenIVM
-metadata. The generated program depends on current state, including delta cutoffs
+metadata, and compile-only calls do not add to the compiled SQL archive. The generated program depends on current state, including delta cutoffs
 and snapshot IDs. Compile again for a later refresh; do not treat an old saved
 program as a reusable refresh procedure. Use `PRAGMA refresh` for normal execution.
 

docs/internals/parser.md

@@ -106,7 +106,7 @@ constrained temporary shadow tables for compilation. DuckLake and other
 cross-catalog lifecycles use staged execution because DuckDB cannot commit writes to
 two attached catalogs in one transaction.
 
-1. **System tables**: `openivm_views`, `openivm_delta_tables`, `openivm_mv_dependencies`, `openivm_refresh_hooks`, `openivm_refresh_history`, and `openivm_refresh_profile`, created once in `main` of the native default database. In autocommit mode this runs in a short, serialized setup transaction; see [Concurrency](concurrency.md#locking).
+1. **System tables**: `openivm_views`, `openivm_delta_tables`, `openivm_mv_dependencies`, `openivm_refresh_hooks`, `openivm_refresh_history`, `openivm_refresh_profile`, and the compiled SQL archive (`openivm_compiled_programs`, `openivm_compiled_statements`), created once in `main` of the native default database. In autocommit mode this runs in a short, serialized setup transaction; see [Concurrency](concurrency.md#locking).
 2. **Metadata inserts**: Registers the internal key, SQL name, location, query string, type, and source table mappings.
 3. **MV table**: `CREATE TABLE openivm_data_<internal_key> AS <query>` to materialize the initial result.
 4. **Published table and view**: `openivm_visible_<internal_key>` holds the visible rows, `openivm_delta_openivm_visible_<internal_key>` their signed changes, and the SQL view `<view_name>` selects from the published table.