Feature: configurable metadata databases with a path to remote SQL storage

.github/workflows/RemoteMetadata.yml

@@ -0,0 +1,49 @@
+#
+# Validates OpenIVM metadata stored in a remote PostgreSQL catalog (issue #84) against a
+# disposable PostgreSQL service container. No external or managed database is contacted.
+#
+name: Remote Metadata Integration
+on:
+  push:
+  pull_request:
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref || '' }}-${{ github.base_ref || '' }}-${{ github.ref != 'refs/heads/main' && github.sha || '' }}
+  cancel-in-progress: true
+
+jobs:
+  remote-metadata:
+    name: OpenIVM metadata in PostgreSQL
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_USER: openivm
+          POSTGRES_PASSWORD: openivm
+          POSTGRES_DB: openivm_test
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U openivm -d openivm_test"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 20
+    env:
+      GEN: ninja
+      OPENIVM_POSTGRES_DSN: host=localhost port=5432 user=openivm password=openivm dbname=openivm_test
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          submodules: recursive
+
+      - name: Install build dependencies
+        run: sudo apt-get update && sudo apt-get install -y ninja-build ccache
+
+      - name: Build
+        run: make release
+
+      - name: Remote metadata integration test
+        run: python3 test/integration/test_remote_metadata.py ./build/release/duckdb --require

CLAUDE.md

@@ -150,6 +150,9 @@ MIN/MAX and ARG_MIN/ARG_MAX usually force group-recompute unless insert-only fas
 | `openivm_adaptive_backoff` | BOOLEAN | `true` | Auto-increase refresh interval when refresh exceeds interval |
 | `openivm_disable_daemon` | BOOLEAN | `false` | Disable the background refresh daemon |
 | `openivm_files_path` | VARCHAR | — | Path for compiled query reference files |
+| `openivm_metadata_catalog` | VARCHAR | `""` | Database-wide attached catalog (native DuckDB or `TYPE postgres`) for OpenIVM metadata; empty keeps it next to each native view catalog. See `docs/internals/metadata-placement.md` |
+| `openivm_metadata_schema` | VARCHAR | `"main"` | Schema for OpenIVM metadata in the metadata catalog (created on first use) |
+| `openivm_metadata_lease_seconds` | BIGINT | `600` | Refresh lease for clients sharing a remote metadata catalog |
 | `openivm_cost_decay` | DOUBLE | `0.9` | Decay factor for learned cost model regression (0.0–1.0) |
 | `openivm_skip_empty_deltas` | BOOLEAN | `true` | Skip refresh or join terms when deltas are empty |
 | `openivm_ducklake_nterm` | BOOLEAN | `true` | N-term telescoping for DuckLake joins (vs 2^N-1) |

CMakeLists.txt

@@ -24,6 +24,7 @@ set(EXTENSION_SOURCES
     src/core/parser_sql_extractors.cpp
     src/core/sql_utils.cpp
     src/core/time_travel_pins.cpp
+    src/core/metadata_location.cpp
     src/core/refresh_metadata.cpp
     src/core/published_view.cpp
     src/core/incremental_checker.cpp

Makefile

@@ -22,6 +22,15 @@ test_scheduler_catalog:
 	cmake --build build/release --target scheduler_catalog_test
 	python3 test/integration/test_scheduler_catalog.py ./build/release/extension/openivm/scheduler_catalog_test
 
+.PHONY: test_remote_metadata
+
+# Skipped unless OPENIVM_POSTGRES_DSN names a disposable local PostgreSQL database
+# (see docs/build/testing.md). CI runs it with --require against a service container.
+test_release_internal: test_remote_metadata
+
+test_remote_metadata:
+	python3 test/integration/test_remote_metadata.py ./build/release/duckdb
+
 .PHONY: test_benchmark_metadata
 
 test_release_internal: test_benchmark_metadata

README.md

@@ -119,6 +119,9 @@ MVs can be created using any SQL construct. Unsupported operators automatically
 | `openivm_profile_refresh` | BOOLEAN | `false` | Record per-step refresh timings in `openivm_refresh_profile` | [Automatic refresh](docs/refresh/automatic-refresh.md) |
 | `openivm_profile_retention_days` | BIGINT | `31` | Delete profile rows older than this many days when profiling writes new rows | [Automatic refresh](docs/refresh/automatic-refresh.md) |
 | `openivm_files_path` | VARCHAR | — | Directory for compiled SQL reference files (off when unset) | [Build: inspect SQL](docs/build/building.md#inspect-the-generated-sql) |
+| `openivm_metadata_catalog` | VARCHAR | `""` | Database-wide attached catalog for OpenIVM metadata (native DuckDB, or PostgreSQL via `TYPE postgres`); empty keeps metadata next to each native view catalog | [Metadata placement](docs/internals/metadata-placement.md) |
+| `openivm_metadata_schema` | VARCHAR | `main` | Schema for OpenIVM metadata in the metadata catalog | [Metadata placement](docs/internals/metadata-placement.md) |
+| `openivm_metadata_lease_seconds` | BIGINT | `600` | Refresh lease for clients sharing a PostgreSQL metadata catalog | [Metadata placement](docs/internals/metadata-placement.md) |
 | `openivm_explain_initial_load` | BOOLEAN | `false` | Print the `CREATE MATERIALIZED VIEW` initial-load SQL and its `EXPLAIN` plans | [Build: inspect SQL](docs/build/building.md#inspect-the-generated-sql) |
 | `openivm_explain_initial_load_only` | BOOLEAN | `false` | With `openivm_explain_initial_load`, print the diagnostic without creating the MV | — |
 | `openivm_input_dialect` | VARCHAR | `duckdb` | Dialect of incoming `CREATE MATERIALIZED VIEW` bodies: `duckdb` or `spark` (e.g. `VERSION AS OF`) | [Parser](docs/internals/parser.md#input-dialect) |

benchmark/src/benchmark_metadata.hpp

@@ -21,18 +21,19 @@ struct ViewTypeLookup {
 };
 
 // Resolves the MV named `sql_name` in the connection's current catalog/schema to its metadata key
-// via RefreshMetadata::FindViewKey, then reads its refresh type from the native catalog's metadata.
-// `con` may point at a DuckLake catalog (USE dl.main); metadata always lives in `native_catalog`.
+// via RefreshMetadata::FindViewKey, then reads its refresh type from the metadata location that
+// RefreshMetadata::UseCatalog selects (`native_catalog`.main unless openivm_metadata_catalog/_schema
+// or a durable marker place it elsewhere). `con` may point at a DuckLake catalog (USE dl.main).
 inline ViewTypeLookup LookupViewType(duckdb::DuckDB &db, duckdb::Connection &con, const std::string &native_catalog,
                                      const std::string &sql_name) {
 	ViewTypeLookup out;
+	duckdb::Connection meta_con(db);
 	try {
 		auto loc = con.Query("SELECT current_database(), current_schema()");
 		if (!loc || loc->HasError() || loc->RowCount() == 0) {
 			out.error = loc ? loc->GetError() : "no result";
 			return out;
 		}
-		duckdb::Connection meta_con(db);
 		duckdb::RefreshMetadata::UseCatalog(*meta_con.context, meta_con, native_catalog);
 		duckdb::RefreshMetadata metadata(meta_con);
 		out.key = metadata.FindViewKey(loc->GetValue(0, 0).ToString(), loc->GetValue(1, 0).ToString(), sql_name);
@@ -43,8 +44,8 @@ inline ViewTypeLookup LookupViewType(duckdb::DuckDB &db, duckdb::Connection &con
 	if (out.key.empty()) {
 		return out;
 	}
-	auto result = con.Query("SELECT type FROM " + duckdb::KeywordHelper::WriteOptionallyQuoted(native_catalog) +
-	                        ".main.openivm_views WHERE view_name = " + duckdb::Value(out.key).ToSQLString());
+	auto result =
+	    meta_con.Query("SELECT type FROM openivm_views WHERE view_name = " + duckdb::Value(out.key).ToSQLString());
 	if (!result || result->HasError()) {
 		out.error = result ? result->GetError() : "no result";
 		return out;

docs/build/building.md

@@ -203,9 +203,11 @@ to obtain an inspection file.
 ### Archived compiled SQL
 
 OpenIVM always archives the programs it runs for `CREATE MATERIALIZED VIEW` and
-refresh in two tables next to the view's `openivm_views` metadata (for DuckLake
-views, the native frontend database's `main` schema), independently of
-`openivm_files_path`:
+refresh in two tables next to the view's `openivm_views` metadata (by default the
+view's native catalog, or for DuckLake views the native frontend database's `main`
+schema; `openivm_metadata_catalog`/`openivm_metadata_schema` move them together with
+the rest of the metadata, see [metadata placement](../internals/metadata-placement.md)),
+independently of `openivm_files_path`:
 
 ```sql
 SELECT view_catalog, view_schema, view_sql_name, operation, version,
@@ -245,9 +247,10 @@ ORDER BY s.stmt_order;
 - **Outcomes.** `last_outcome` describes the latest operation that ran the version:
   `committed` (its transaction committed), `attempted` (it failed and rolled back,
   so nothing was committed) or `unknown` (a cross-catalog refresh failed after its
-  first statements ran, or `COMMIT` itself failed, so some effects may have
-  committed). This includes a DuckLake refresh whose data statements roll back,
-  because its native in-progress marker has already committed.
+  first statements ran outside a transaction, or `COMMIT` itself failed, so some
+  effects may have committed). This includes a DuckLake refresh whose data statements
+  roll back, because its in-progress marker in the metadata location has already
+  committed.
   `committed_count` counts committed runs. Compile-only calls such as
   `openivm_compile_with_facts` are never archived, so a stored program has always
   been executed.

docs/build/testing.md

@@ -52,6 +52,29 @@ All tests live in `test/sql/*.test` using DuckDB's SQLLogicTest format.
 | `compile_refresh.test` | `openivm_compile_with_facts` compile-only refresh |
 | `time_travel.test`, `time_travel_ducklake.test` | Time-travel pins |
 | `ducklake_*.test` | The same operator families over DuckLake sources |
+| `metadata_location.test` | Metadata in an attached catalog/schema: placement, crash recovery, explicit transactions, reopen, read-only/missing catalogs, drop, replace |
+| `metadata_location_ducklake.test` | DuckLake views with metadata in an attached catalog: frontend marker, reopen discovery, crash recovery (including chained views), concurrent source commit |
+| `ducklake_chained_recovery.test` | Chained DuckLake views with default metadata: an interrupted upstream refresh neither loses nor doubles downstream changes |
+| `ducklake_concurrent_source_commit.test` | DuckLake source changes committed by another connection during a refresh: pinned watermark, retry, no lost or doubled changes |
+
+## Remote metadata integration test
+
+`test/integration/test_remote_metadata.py` stores OpenIVM metadata in PostgreSQL and runs
+several DuckDB processes against it. It is skipped unless `OPENIVM_POSTGRES_DSN` names a
+disposable database; CI starts a PostgreSQL 16 service container
+(`.github/workflows/RemoteMetadata.yml`). Never point it at a shared server: each run creates
+and drops its own schemas. To run it locally:
+
+```bash
+docker run --rm -d --name openivm-pg -p 5432:5432 \
+    -e POSTGRES_USER=openivm -e POSTGRES_PASSWORD=openivm -e POSTGRES_DB=openivm_test postgres:16
+export OPENIVM_POSTGRES_DSN='host=localhost port=5432 user=openivm password=openivm dbname=openivm_test'
+python3 test/integration/test_remote_metadata.py ./build/release/duckdb --require
+docker stop openivm-pg
+```
+
+The test installs the `postgres`, `ducklake` and `sqlite` extensions, so it needs network
+access the first time.
 
 ## Verification Pattern
 

docs/concurrency.md

@@ -16,7 +16,9 @@ day to day. For the mechanics behind this (the mutation gate, cursor bookkeeping
 | Writing to a table no MV depends on | Not affected by OpenIVM. |
 | Two sessions creating the same MV | Exactly one succeeds and metadata stays consistent; the other fails with a write-write conflict naming an internal table ([#91](https://github.com/ila/openivm/issues/91)). |
 | `DROP VIEW` or `ALTER TABLE` during a refresh | Waits for the refresh to finish, then runs. |
-| Writers outside this DuckDB process (e.g. other DuckLake clients) | Not blocked by OpenIVM; their changes are picked up by the next refresh. |
+| Writers outside this DuckDB process (e.g. other DuckLake clients) | Not blocked by OpenIVM. A change committed before a refresh starts is applied by it; one committed while it runs is applied by it or by the next refresh, never lost or applied twice (see [below](#ducklake-sources-changing-during-a-refresh)). |
+| Refresh with metadata in another database (`openivm_metadata_catalog`) | Committed in three autocommit steps; a crash between them makes the next refresh a full recompute. Rejected inside explicit transactions. See [metadata placement](internals/metadata-placement.md). |
+| Two processes refreshing one view through shared PostgreSQL metadata | Serialized by a refresh lease; the second fails with `being refreshed by another OpenIVM client` and can retry. |
 
 A table is *tracked* when at least one materialized view reads from it.
 
@@ -45,6 +47,27 @@ Two consequences matter in practice:
 
 Keep transactions that touch tracked tables short, and commit before doing slow work in the same session.
 
+## DuckLake sources changing during a refresh
+
+The gate cannot stop other processes (or other DuckLake clients) from committing to a DuckLake source while a
+refresh reads it. Before compiling, a refresh of a view with DuckLake sources records the current snapshot of every
+attached DuckLake catalog. Its
+reads all happen at that snapshot or later, so once it has run it checks whether any of the view's source tables
+changed after the recorded snapshot. The refresh's own writes to its backing, delta and auxiliary tables are not
+sources and do not count.
+
+- **No source changed:** every read saw the recorded snapshot. The view's watermark becomes the snapshot up to which
+  the check ran, which has the same source state. A change committed after the check has a higher snapshot id and is
+  applied by the next refresh, including one that lands between a transactional data commit and the watermark write.
+- **A source changed:** the refresh may have read part of that change. A native view's data transaction is rolled
+  back before it commits, and so is a DuckLake view's when its metadata is placed in a configured metadata catalog or
+  when downstream views read its changes. Other DuckLake views have already committed their data statements one by
+  one, so their watermarks are not written and the interruption marker stays set. Either way the refresh retries at
+  once; for a view whose marker stays set the retry is a full recompute.
+
+After three attempts that each saw a source change, the refresh fails with `changed while refreshing`. Nothing is
+lost: the view keeps its last stored watermark, or stays marked for a full recompute by the next refresh.
+
 ## Scheduled refresh needs a running process
 
 `REFRESH EVERY` is served by a background thread inside the process that has the database open with OpenIVM loaded.
@@ -101,6 +124,10 @@ run any statement. See [refresh hooks](refresh_hooks.md).
 |---|---|
 | Refresh, DML on a tracked table or the daemon hangs | Another session holds the gate: look for an open transaction that wrote to a tracked table, and commit or roll it back. |
 | `Warning: recovering '<view>' from interrupted refresh via full recompute` | A previous refresh stopped mid-way; the full recompute restores the view. See [crash safety](refresh/automatic-refresh.md#crash-safety). |
+| `cannot run inside an explicit transaction` | The view's metadata is in another database; run the statement in autocommit mode. |
+| `OpenIVM metadata catalog '<name>' ... is not attached` or `... which is not attached` | Attach the metadata catalog under the recorded name. Base-table changes are still captured meanwhile. |
+| `being refreshed by another OpenIVM client` | Another process holds the view's refresh lease; retry. A crashed client's lease expires after `openivm_metadata_lease_seconds`. |
+| `DuckLake source table '<table>' changed while refreshing '<view>' in each of 3 attempts` | Another client kept committing to a source during every attempt. No change was lost; refresh again once the writes pause. See [DuckLake sources changing during a refresh](#ducklake-sources-changing-during-a-refresh). |
 | `OpenIVM refresh committed; external delta cleanup deferred` | Consumed deltas in another attached database couldn't be deleted after commit. The view is correct; the rows are retried on a later refresh. |
 | `... does not exist in IVM metadata` right after `CREATE` | The statements were sent in one query string; see [batching statements](#batching-statements). |
 | A scheduled view stopped updating | Check that a process with the database open is still running, then `PRAGMA refresh_status('<view>')` for the effective interval and last refresh. |

docs/ducklake.md

@@ -79,7 +79,7 @@ ORDER BY database_name, schema_name, table_name;
 ```
 
 For DuckLake MVs, OpenIVM's control tables (`openivm_views`,
-`openivm_delta_tables`, dependency and refresh-history tables) already live in
+`openivm_delta_tables`, dependency and refresh-history tables) live by default in
 `main` of the native DuckDB frontend database, not as Parquet tables in DuckLake.
 Start the CLI with a persistent frontend file to retain them between sessions:
 
@@ -90,9 +90,36 @@ Start the CLI with a persistent frontend file to retain them between sessions:
 Then attach the lake and create views in its catalog as in the quick start. After
 reopening the same frontend file, attach the same lake under the same catalog name
 before refreshing. Starting the CLI without a filename uses an in-memory frontend;
-its OpenIVM metadata will not survive closing the process. There is currently no
-setting to relocate OpenIVM control tables into an `openivm` schema. Do not move
-those tables manually.
+its OpenIVM metadata will not survive closing the process. Do not move the control
+tables manually.
+
+To keep the control tables elsewhere, select a metadata catalog and schema before
+creating views. The catalog can be another attached DuckDB file or a PostgreSQL
+database attached with the `postgres` extension:
+
+```sql
+ATTACH 'dbname=openivm host=localhost' AS control (TYPE postgres);
+SET openivm_metadata_catalog = 'control';
+SET openivm_metadata_schema = 'openivm';
+CREATE MATERIALIZED VIEW dl.main.product_summary AS ...;
+```
+
+Backing and delta tables still live in the lake. DuckLake catalogs cannot hold the
+location marker, so OpenIVM records it in the frontend database: a persistent frontend
+file finds the metadata again after reopening, while a process with an in-memory
+frontend sets these two settings at startup, before refreshing. Views cannot be created
+inside a PostgreSQL metadata catalog. DuckDB cannot commit the lake and the metadata catalog in one
+transaction, so refresh uses a crash-safe three-step protocol whose data step is one
+lake transaction (downstream views never see a partly refreshed backing table), and
+lifecycle statements
+and refreshes inside explicit transactions are rejected for these views. The PostgreSQL
+metadata integration test runs against a local PostgreSQL 16 in CI; managed providers
+are untested, and a client process running OpenIVM is still required. A source change
+another client commits while a refresh runs is never lost: the stored watermark is a
+snapshot with the source state the refresh read, and a refresh that saw a source change retries (see
+[concurrency](concurrency.md#ducklake-sources-changing-during-a-refresh)). See
+[metadata placement](internals/metadata-placement.md) for discovery, recovery and
+multi-client semantics.
 
 For native sources in another attached DuckDB database, refresh commits the MV
 and its watermark together, then cleans up external source deltas. Rollback