From 00b8d944812241279b0d0e4621a7cb0991bd2673 Mon Sep 17 00:00:00 2001 From: azim afroozeh <13484327+azimafroozeh@users.noreply.github.com> Date: Mon, 5 Oct 2026 01:30:28 +0200 Subject: [PATCH] feat(storage): upgrade format 8 and 9 graphs to format 14 --- .github/workflows/ci.yml | 37 +- changelog.d/history-release.breaking.md | 4 +- .../schema-contract-inlining.breaking.md | 4 +- .../storage-upgrade-v13-v14.breaking.md | 30 +- crates/omnigraph-catalog/src/legacy.rs | 511 +++-- crates/omnigraph-catalog/src/legacy/census.rs | 4 +- crates/omnigraph-catalog/src/migrations.rs | 162 +- crates/omnigraph-catalog/src/tests.rs | 391 +++- crates/omnigraph-cli/src/cli.rs | 4 +- .../tests/crossversion_upgrade.rs | 852 +++++-- crates/omnigraph/src/db/catalog_tests.rs | 7 +- crates/omnigraph/src/db/omnigraph.rs | 6 +- crates/omnigraph/src/db/upgrade.rs | 715 ++++-- .../omnigraph/src/db/upgrade/root_schema.rs | 198 ++ crates/omnigraph/src/db/upgrade/tests.rs | 1983 ++++++++++++++--- docs/dev/blob.md | 2 +- docs/dev/ci.md | 28 +- docs/dev/invariants.md | 2 +- docs/dev/recovery.md | 36 +- docs/dev/testing.md | 59 +- docs/dev/versioning.md | 114 +- docs/rfcs/0064-explicit-storage-upgrades.md | 25 + docs/rfcs/0068-graph-commit-record.md | 13 + docs/user/cli/reference.md | 12 +- docs/user/deployment.md | 2 +- docs/user/operations/troubleshooting.md | 2 +- docs/user/operations/upgrade.md | 82 +- scripts/check-storage-upgrade-ci.py | 115 +- skills/omnigraph/SKILL.md | 13 +- skills/omnigraph/references/commands.md | 13 +- skills/omnigraph/references/migrations.md | 44 + 31 files changed, 4594 insertions(+), 876 deletions(-) create mode 100644 crates/omnigraph/src/db/upgrade/root_schema.rs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f0bbf9ba7..d2469630e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -771,7 +771,42 @@ jobs: - name: Check storage compatibility gate configuration run: python3 scripts/check-storage-upgrade-ci.py --self-test - # The genuine stamp-13 predecessor for the one in-place route + # The genuine predecessors of the release route (`omnigraph upgrade`, + # storage format 8 or 9 to 14): v0.11.0 writes stamp 9 and converts a + # v0.10.0 graph to stamp 8. The release assets are trusted exactly as + # the `test` job's v0.9.0 and v0.10.0 installs trust them: `install.sh` + # verifies the archive against the `.sha256` published beside it, and + # no digest is pinned here. + - name: Install released v0.10.0 and v0.11.0 CLIs + run: | + set -euo pipefail + v6_dir="$RUNNER_TEMP/omnigraph-v010" + v011_dir="$RUNNER_TEMP/omnigraph-v011" + # The installer downloads the official archive and verifies its + # SHA256 before extraction. An exact VERSION never falls back to edge. + # A failed download leaves nothing behind, so each release gets three + # attempts before this required context goes red. + install_release() { + for attempt in 1 2 3; do + REPO_SLUG=ModernRelay/omnigraph VERSION="$1" INSTALL_DIR="$2" \ + bash scripts/install.sh && return 0 + echo "install of $1 failed on attempt $attempt/3; retrying" + sleep 10 + done + return 1 + } + install_release v0.10.0 "$v6_dir" + test -x "$v6_dir/omnigraph" + [[ "$("$v6_dir/omnigraph" --version)" == "omnigraph 0.10.0" ]] \ + || { echo "::error::the storage upgrades of a 0.10.0 graph require the genuine v0.10.0 CLI"; exit 1; } + install_release v0.11.0 "$v011_dir" + test -x "$v011_dir/omnigraph" + [[ "$("$v011_dir/omnigraph" --version)" == "omnigraph 0.11.0" ]] \ + || { echo "::error::the release storage upgrade requires the genuine v0.11.0 CLI"; exit 1; } + echo "OMNIGRAPH_V6_BIN=$v6_dir/omnigraph" >> "$GITHUB_ENV" + echo "OMNIGRAPH_V011_BIN=$v011_dir/omnigraph" >> "$GITHUB_ENV" + + # The genuine stamp-13 predecessor for the main-build in-place route # (`omnigraph upgrade`, storage format 13 to 14). Keep this immutable: # HEAD^ is not a stable format reference, and the last main commit that # wrote stamp 13 is the source the journey converts. diff --git a/changelog.d/history-release.breaking.md b/changelog.d/history-release.breaking.md index 67e5bcb3e..c24d9f0e0 100644 --- a/changelog.d/history-release.breaking.md +++ b/changelog.d/history-release.breaking.md @@ -2,8 +2,8 @@ commits and releases older ones into immutable Lance files under `__history`, so a publish no longer rewrites the branch's whole history. Graph commit ids take the form `hb1...`. This is - storage format 14: a standalone format 13 graph is converted by the offline - `omnigraph upgrade`, and a graph at any older format is rebuilt by export, + storage format 14: a standalone format 8, 9 or 13 graph is converted by the + offline `omnigraph upgrade`, and a graph at any other format is rebuilt by export, `init` and `load --mode overwrite`, as described in the [upgrade guide][history-release-upgrade]. A session may lower the release budget with `set history_release_bytes = ;` (`1024..=262144`, default diff --git a/changelog.d/schema-contract-inlining.breaking.md b/changelog.d/schema-contract-inlining.breaking.md index fb0b6f38f..bc4e498dd 100644 --- a/changelog.d/schema-contract-inlining.breaking.md +++ b/changelog.d/schema-contract-inlining.breaking.md @@ -1,8 +1,8 @@ - Schema apply and system-column upgrades now publish the schema source, compiled schema and identity atomically with their table references in the catalog. Ordinary open serves storage format 14 only and refuses every other - format. Convert a standalone format 13 graph with the offline - `omnigraph upgrade`; rebuild an older graph by export, `init` and + format. Convert a standalone format 8, 9 or 13 graph with the offline + `omnigraph upgrade`; rebuild a graph at any other format by export, `init` and `load --mode overwrite`, as described in the [upgrade guide][schema-contract-inlining-upgrade]. Fresh graphs no longer create separate root schema files. diff --git a/changelog.d/storage-upgrade-v13-v14.breaking.md b/changelog.d/storage-upgrade-v13-v14.breaking.md index 52751d9c9..236e9d161 100644 --- a/changelog.d/storage-upgrade-v13-v14.breaking.md +++ b/changelog.d/storage-upgrade-v13-v14.breaking.md @@ -1,8 +1,9 @@ -- `omnigraph upgrade` now converts a standalone storage format 13 graph - (written by main development builds from 2026-10-01 to 2026-10-04) to - format 14 in place, keeping its branches, commit ids and commit history; it - no longer only reports. The upgrade is offline: stop every process using - the graph, retain a verified backup of the whole root, run +- `omnigraph upgrade` now converts a standalone graph to storage format 14 + in place, keeping its branches, commit ids and commit history; it no longer + only reports. It accepts two sources: formats 8 and 9, written by release + 0.11.x, and format 13, written by main development builds from 2026-10-01 + to 2026-10-04. The upgrade is offline: stop every process using the graph, + retain a verified backup of the whole root, run `omnigraph upgrade --check`, then `omnigraph upgrade `. An interrupted run leaves the graph refused by normal open until the same command is run again. Cluster-managed graphs and every other format are @@ -10,8 +11,23 @@ commits are kept under `__history/legacy/`: a 14 build older than this change reports pre-upgrade commits as not found by id and refuses full-history reads on an upgraded root, so upgrade every binary before the - graph. The JSON report gains a `work` object and now reports - `check_passed` and `completed`. See the + graph. For a 0.11.x graph the schema is read from the three schema objects + at the graph root, which the upgrade leaves in place and which go stale at + the next schema apply; the objects must match the tables, so a stale copy + restored from an older backup is refused before any write; every + pre-upgrade commit is recorded under the one schema live at the upgrade; a + `.staging` schema object or a recovery file left by 0.11.x is refused until + a 0.11.x read-write open (`omnigraph snapshot`) resolves it, while a + `__schema_apply_lock__` branch left alone by a killed 0.11.x schema apply is + handled by the upgrade and is not a live branch afterwards; and on a root + where 0.11.x `cleanup` or `schema apply --allow-data-loss` ran, the first + `cleanup` after the upgrade is `cleanup --keep 1 --confirm` alone. The JSON + report gains a `work` object, names its route per source + (`history-lance-files-v8-to-v14`, `history-lance-files-v9-to-v14`, + `history-lance-files-v13-to-v14`), names `history-lance-files-to-v14` as + `recovery.failed_handler` when it stops before any route is known (an + unreadable pending marker, or recovery files on a format no route + converts), and now reports `check_passed` and `completed`. See the [upgrade guide][storage-upgrade-v13-v14-guide]. [storage-upgrade-v13-v14-guide]: ../docs/user/operations/upgrade.md#storage-upgrade diff --git a/crates/omnigraph-catalog/src/legacy.rs b/crates/omnigraph-catalog/src/legacy.rs index 703f0a1f7..d4cad7833 100644 --- a/crates/omnigraph-catalog/src/legacy.rs +++ b/crates/omnigraph-catalog/src/legacy.rs @@ -2,9 +2,11 @@ //! from, and the test fixture writer of those stamps. The reader is handed a //! dataset checked out at one version and never writes. Everything a source //! stamp decides (which versions it admits, its row shapes, its key grammar) -//! sits behind [`LegacyManifestSource`], so a reader of another source stamp -//! replaces [`Stamp13Source`] alone. The census, the plan and the conversion -//! rows derived from what the reader returns are source-independent. +//! sits behind [`LegacyManifestSource`]. Its two implementors are +//! [`Stamp13Source`], whose contract is a `__manifest` row, and +//! [`RootContractSource`] for stamps 8 and 9, whose contract the caller read +//! from the graph root. The census, the plan and the conversion rows derived +//! from what the reader returns are source-independent. mod census; mod layout; @@ -17,7 +19,6 @@ pub use census::{ }; use std::collections::{BTreeMap, BTreeSet, HashMap}; -use std::ops::RangeInclusive; use std::sync::Arc; use arrow_array::{Array, LargeStringArray, RecordBatch, StringArray, UInt64Array}; @@ -56,14 +57,18 @@ pub const OBJECT_TYPE_TABLE_TOMBSTONE: &str = "table_tombstone"; /// The head pointer of one branch, keyed `graph_head:`. pub const OBJECT_TYPE_GRAPH_HEAD: &str = "graph_head"; -/// The stamp this route converts from: a live ref's head carries it. -const SOURCE_STAMP: u32 = SCHEMA_CONTENT_STAMP; -/// The stamps a retired ref's head may carry: retirement metadata arrived at 8. -const RETIRED_HEAD_STAMPS: RangeInclusive = 8..=SOURCE_STAMP; -/// The stamps a retained version may carry: the oldest a supported route left behind is 6. -const VERSION_STAMPS: RangeInclusive = 6..=SOURCE_STAMP; +/// The oldest stamp a retired ref's head may carry: retirement metadata arrived at 8. +const OLDEST_RETIRED_HEAD_STAMP: u32 = 8; +/// The oldest stamp a retained version may carry: the oldest a supported route left behind is 6. +const OLDEST_VERSION_STAMP: u32 = 6; /// What an unstamped bootstrap version admits as: the stamp an absent key means (`guard_stamp`). const ABSENT_STAMP: u32 = 1; +/// The stamps [`RootContractSource`] reads a live head at: the upgrade sources +/// below the one whose contract is a row. +const ROOT_CONTRACT_STAMPS: [u32; 2] = { + let [oldest, older, _] = crate::migrations::UPGRADE_SOURCE_FORMATS; + [oldest, older] +}; /// What the upgrade reads a `__manifest` version as. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -79,24 +84,30 @@ pub enum SourceRole { } impl SourceRole { - fn admits(self, stamp: Option, version: u64) -> bool { + /// Whether a version stamped `stamp` reads as this role under a source + /// whose live heads are stamped `live`: no version of a root is stamped + /// above its live heads. + fn admits(self, stamp: Option, version: u64, live: u32) -> bool { match (self, stamp) { - (Self::LiveHead, Some(stamp)) => stamp == SOURCE_STAMP, - (Self::RetiredHead, Some(stamp)) => RETIRED_HEAD_STAMPS.contains(&stamp), - (Self::Version, Some(stamp)) => VERSION_STAMPS.contains(&stamp), + (Self::LiveHead, Some(stamp)) => stamp == live, + (Self::RetiredHead, Some(stamp)) => (OLDEST_RETIRED_HEAD_STAMP..=live).contains(&stamp), + (Self::Version, Some(stamp)) => (OLDEST_VERSION_STAMP..=live).contains(&stamp), (Self::Version, None) => matches!(version, 1 | 2), (Self::LiveHead | Self::RetiredHead, None) => false, } } - fn admitted(self) -> &'static str { + fn admitted(self, live: u32) -> String { match self { - Self::LiveHead => "the head of a live branch must be stamped v13", - Self::RetiredHead => "the head of a retired branch must be stamped v8 to v13", - Self::Version => { - "a retained version must be stamped v6 to v13, or be the unstamped version 1 \ - or 2 an older init wrote" - } + Self::LiveHead => format!("the head of a live branch must be stamped v{live}"), + Self::RetiredHead => format!( + "the head of a retired branch must be stamped v{OLDEST_RETIRED_HEAD_STAMP} to \ + v{live}" + ), + Self::Version => format!( + "a retained version must be stamped v{OLDEST_VERSION_STAMP} to v{live}, or be \ + the unstamped version 1 or 2 an older init wrote" + ), } } } @@ -118,7 +129,8 @@ pub struct LegacyCommit { /// Everything one scan of a ref's head holds: every commit and head row, /// every registration and tombstone keyed by `(identity, clock)`, the contract -/// row (`None` below stamp 13) and the count of rows scanned. A stamp-13 +/// (the head's row at stamp 13, the root contract under +/// [`RootContractSource`], else `None`) and the count of rows scanned. A stamp-13 /// overwrite rewrites every row and never removes a registration, so a head /// holds every pin and drop written on its lineage before it. #[derive(Debug, Clone, PartialEq, Eq)] @@ -134,8 +146,8 @@ pub struct HeadScan { } /// The table membership, aliases and paths of one version, in identity order, -/// and its contract (`None` below stamp 13 or when the version holds no -/// `schema_contract` row). +/// and its contract: the version's `schema_contract` row at stamp 13, `None` +/// when it holds none and under `RootContractSource` for every version. #[derive(Debug, Clone, PartialEq, Eq)] pub struct VersionSchema { pub tables: Vec, @@ -173,58 +185,64 @@ pub struct Stamp13Source; #[async_trait::async_trait] impl LegacyManifestSource for Stamp13Source { fn admits(&self, dataset: &Dataset, role: SourceRole) -> Result { - let version = dataset.version().version; - let metadata = &dataset.schema().metadata; - let pending = metadata.get(UPGRADE_PENDING_KEY); - if pending.is_some() && role != SourceRole::Version { - return Err(OmniError::manifest(format!( - "__manifest version {version} carries a pending storage conversion \ - ({UPGRADE_PENDING_KEY}); it is not a v13 source" - ))); - } - let stamp = match metadata.get(INTERNAL_SCHEMA_VERSION_KEY) { - None => None, - Some(value) => Some(value.parse::().map_err(|_| { - OmniError::manifest(format!( - "__manifest version {version} carries the internal-schema stamp '{value}', \ - which is not a version number" - )) - })?), - }; - if let Some(intent) = pending { - return earlier_route_stamp(dataset, stamp, intent); - } - if !role.admits(stamp, version) { - let found = stamp.map_or_else(|| "unstamped".to_string(), |stamp| format!("v{stamp}")); + admit(dataset, role, SCHEMA_CONTENT_STAMP) + } + + async fn scan_head(&self, dataset: &Dataset) -> Result { + scan_head_as(dataset, SCHEMA_CONTENT_STAMP).await + } + + async fn version_schema( + &self, + dataset: &Dataset, + previous: Option<&VersionSchema>, + ) -> Result { + version_schema_as(dataset, previous, SCHEMA_CONTENT_STAMP).await + } +} + +/// The source of the 8 → 14 and 9 → 14 routes: flat rows under the stamp-7 +/// key grammar and no `schema_contract` row in any version. The contract is +/// the one the caller read from the graph root. +#[derive(Debug, Clone)] +pub struct RootContractSource { + stamp: u32, + contract: SchemaContractRow, +} + +impl RootContractSource { + /// The source of a root whose live heads are stamped `stamp` (8 or 9), + /// under the contract read from its root. + pub fn new(stamp: u32, contract: SchemaContractRow) -> Result { + if !ROOT_CONTRACT_STAMPS.contains(&stamp) { return Err(OmniError::manifest(format!( - "__manifest version {version} is {found}: {}", - role.admitted() + "a schema contract read from the graph root is the contract of a v8 or v9 \ + graph, not of a v{stamp} one" ))); } - let stamp = stamp.unwrap_or(ABSENT_STAMP); - require_stored_shape(dataset, stamp)?; - Ok(stamp) + Ok(Self { stamp, contract }) + } +} + +#[async_trait::async_trait] +impl LegacyManifestSource for RootContractSource { + fn admits(&self, dataset: &Dataset, role: SourceRole) -> Result { + admit(dataset, role, self.stamp) } + /// Every pin is linear, so each is returned with `last_linear_version` + /// set to its own table version: every ref's scan then returns the same + /// pin for the same row, inherited or not. async fn scan_head(&self, dataset: &Dataset) -> Result { - let stamp = self.admits(dataset, SourceRole::RetiredHead)?; - let shape = stored_shape(Some(stamp)); - let projection = match shape { - StoredShape::Packed => crate::record::packed_projection(dataset, true), - StoredShape::Flat => flat_projection(), - }; - crate::instrumentation::record_manifest_scan(); - let mut scanner = dataset.scan(); - scanner.project(&projection).map_err(OmniError::storage)?; - let mut batches = scanner - .try_into_stream() - .await - .map_err(OmniError::storage)?; - let mut fold = HeadFold::new(dataset.version().version); - while let Some(batch) = batches.try_next().await.map_err(OmniError::storage)? { - fold.fold(&logical(shape, batch)?)?; + let mut scan = scan_head_as(dataset, self.stamp).await?; + for pin in scan.pins.values_mut() { + pin.metadata = pin + .metadata + .clone() + .with_last_linear_version(Some(pin.table_version)); } - Ok(fold.finish()?.0) + scan.contract = Some(self.contract.clone()); + Ok(scan) } async fn version_schema( @@ -232,102 +250,168 @@ impl LegacyManifestSource for Stamp13Source { dataset: &Dataset, previous: Option<&VersionSchema>, ) -> Result { - let stamp = self.admits(dataset, SourceRole::Version)?; - let shape = stored_shape(Some(stamp)); - let mut projection: Vec = match shape { - StoredShape::Packed => vec!["object_id", "object_type", RECORD_COLUMN], - StoredShape::Flat => vec![ - "object_id", - "object_type", - "location", - "metadata", - "table_key", - "stable_table_id", - "table_incarnation_id", - ], - } - .into_iter() - .map(str::to_string) - .collect(); - if stamp == SCHEMA_CONTENT_STAMP { - projection.push(SCHEMA_CONTENT_COLUMNS[0].to_string()); - } - crate::instrumentation::record_manifest_scan(); - let mut scanner = dataset.scan(); - scanner.project(&projection).map_err(OmniError::storage)?; - scanner.filter_expr(col("object_type").in_list( - vec![lit(OBJECT_TYPE_TABLE), lit(OBJECT_TYPE_SCHEMA_CONTRACT)], - false, - )); - scanner.materialization_style(lance::dataset::scanner::MaterializationStyle::AllEarly); - let mut batches = scanner - .try_into_stream() - .await - .map_err(OmniError::storage)?; - let mut tables = BTreeMap::new(); - let mut found: Option<(SchemaContractHead, String)> = None; - while let Some(batch) = batches.try_next().await.map_err(OmniError::storage)? { - let batch = logical(shape, batch)?; - let table_columns = TableColumns::of(&batch)?; - let object_ids = table_columns.object_ids; - let object_types = string_column(&batch, "object_type")?; - let metadata = metadata_column(&batch)?; - let sources = batch - .column_by_name(SCHEMA_CONTENT_COLUMNS[0]) - .map(|_| large_string_column(&batch, SCHEMA_CONTENT_COLUMNS[0])) - .transpose()?; - for row in 0..batch.num_rows() { - require_contract_type(object_ids, object_types, row)?; - match object_types.value(row) { - OBJECT_TYPE_TABLE => insert_registration( - &mut tables, - table_columns.registration(row, dataset.version().version)?, - )?, - OBJECT_TYPE_SCHEMA_CONTRACT => { - let head = decode_schema_contract_head(object_ids, metadata, row)?; - let source = sources - .filter(|sources| !sources.is_null(row)) - .ok_or_else(|| { - OmniError::manifest_internal(format!( - "__manifest version {} has a schema_contract row without its \ - source text", - dataset.version().version - )) - })? - .value(row) - .to_string(); - if found.replace((head, source)).is_some() { - return Err(OmniError::manifest_internal( - "manifest has two schema_contract rows".to_string(), - )); - } - } - other => { - return Err(OmniError::manifest_internal(format!( - "the table and contract scan returned a '{other}' row" - ))); + version_schema_as(dataset, previous, self.stamp).await + } +} + +/// [`LegacyManifestSource::admits`] for a source whose live heads are stamped `live`. +fn admit(dataset: &Dataset, role: SourceRole, live: u32) -> Result { + let version = dataset.version().version; + let metadata = &dataset.schema().metadata; + let pending = metadata.get(UPGRADE_PENDING_KEY); + if pending.is_some() && role != SourceRole::Version { + return Err(OmniError::manifest(format!( + "__manifest version {version} carries a pending storage conversion \ + ({UPGRADE_PENDING_KEY}); it is not a v{live} source" + ))); + } + let stamp = match metadata.get(INTERNAL_SCHEMA_VERSION_KEY) { + None => None, + Some(value) => Some(value.parse::().map_err(|_| { + OmniError::manifest(format!( + "__manifest version {version} carries the internal-schema stamp '{value}', \ + which is not a version number" + )) + })?), + }; + if let Some(intent) = pending { + return earlier_route_stamp(dataset, stamp, intent, live); + } + if !role.admits(stamp, version, live) { + let found = stamp.map_or_else(|| "unstamped".to_string(), |stamp| format!("v{stamp}")); + return Err(OmniError::manifest(format!( + "__manifest version {version} is {found}: {}", + role.admitted(live) + ))); + } + let stamp = stamp.unwrap_or(ABSENT_STAMP); + require_stored_shape(dataset, stamp)?; + Ok(stamp) +} + +/// [`LegacyManifestSource::scan_head`] for a source whose live heads are stamped `live`. +async fn scan_head_as(dataset: &Dataset, live: u32) -> Result { + let stamp = admit(dataset, SourceRole::RetiredHead, live)?; + let shape = stored_shape(Some(stamp)); + let projection = match shape { + StoredShape::Packed => crate::record::packed_projection(dataset, true), + StoredShape::Flat => flat_projection(), + }; + crate::instrumentation::record_manifest_scan(); + let mut scanner = dataset.scan(); + scanner.project(&projection).map_err(OmniError::storage)?; + let mut batches = scanner + .try_into_stream() + .await + .map_err(OmniError::storage)?; + let mut fold = HeadFold::new(dataset.version().version); + while let Some(batch) = batches.try_next().await.map_err(OmniError::storage)? { + fold.fold(&logical(shape, batch)?)?; + } + Ok(fold.finish()?.0) +} + +/// [`LegacyManifestSource::version_schema`] for a source whose live heads are stamped `live`. +async fn version_schema_as( + dataset: &Dataset, + previous: Option<&VersionSchema>, + live: u32, +) -> Result { + let stamp = admit(dataset, SourceRole::Version, live)?; + let shape = stored_shape(Some(stamp)); + let mut projection: Vec = match shape { + StoredShape::Packed => vec!["object_id", "object_type", RECORD_COLUMN], + StoredShape::Flat => vec![ + "object_id", + "object_type", + "location", + "metadata", + "table_key", + "stable_table_id", + "table_incarnation_id", + ], + } + .into_iter() + .map(str::to_string) + .collect(); + if stamp == SCHEMA_CONTENT_STAMP { + projection.push(SCHEMA_CONTENT_COLUMNS[0].to_string()); + } + crate::instrumentation::record_manifest_scan(); + let mut scanner = dataset.scan(); + scanner.project(&projection).map_err(OmniError::storage)?; + scanner.filter_expr(col("object_type").in_list( + vec![lit(OBJECT_TYPE_TABLE), lit(OBJECT_TYPE_SCHEMA_CONTRACT)], + false, + )); + scanner.materialization_style(lance::dataset::scanner::MaterializationStyle::AllEarly); + let mut batches = scanner + .try_into_stream() + .await + .map_err(OmniError::storage)?; + let mut tables = BTreeMap::new(); + let mut found: Option<(SchemaContractHead, String)> = None; + while let Some(batch) = batches.try_next().await.map_err(OmniError::storage)? { + let batch = logical(shape, batch)?; + let table_columns = TableColumns::of(&batch)?; + let object_ids = table_columns.object_ids; + let object_types = string_column(&batch, "object_type")?; + let metadata = metadata_column(&batch)?; + let sources = batch + .column_by_name(SCHEMA_CONTENT_COLUMNS[0]) + .map(|_| large_string_column(&batch, SCHEMA_CONTENT_COLUMNS[0])) + .transpose()?; + for row in 0..batch.num_rows() { + require_contract_type(object_ids, object_types, row)?; + match object_types.value(row) { + OBJECT_TYPE_TABLE => insert_registration( + &mut tables, + table_columns.registration(row, dataset.version().version)?, + )?, + OBJECT_TYPE_SCHEMA_CONTRACT => { + let head = decode_schema_contract_head(object_ids, metadata, row)?; + let source = sources + .filter(|sources| !sources.is_null(row)) + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "__manifest version {} has a schema_contract row without its \ + source text", + dataset.version().version + )) + })? + .value(row) + .to_string(); + if found.replace((head, source)).is_some() { + return Err(OmniError::manifest_internal( + "manifest has two schema_contract rows".to_string(), + )); } } + other => { + return Err(OmniError::manifest_internal(format!( + "the table and contract scan returned a '{other}' row" + ))); + } } } - let contract = match found { - None => None, - Some((head, source)) => { - let ir = match previous - .and_then(|previous| previous.contract.as_ref()) - .filter(|previous| previous.head == head && previous.source == source) - { - Some(previous) => previous.ir.clone(), - None => read_schema_ir(dataset).await?, - }; - Some(SchemaContractRow { source, ir, head }) - } - }; - Ok(VersionSchema { - tables: tables.into_values().collect(), - contract, - }) } + let contract = match found { + None => None, + Some((head, source)) => { + let ir = match previous + .and_then(|previous| previous.contract.as_ref()) + .filter(|previous| previous.head == head && previous.source == source) + { + Some(previous) => previous.ir.clone(), + None => read_schema_ir(dataset).await?, + }; + Some(SchemaContractRow { source, ir, head }) + } + }; + Ok(VersionSchema { + tables: tables.into_values().collect(), + contract, + }) } /// The current head of a lineage: the commit with the greatest @@ -620,7 +704,7 @@ async fn manifest_refs( Ok(refs) } -/// The upgrade routes a stamp-13 root may have completed, as the +/// The upgrade routes a source root may have completed, as the /// `(protocol, source_format, target_format)` their intents carry. const EARLIER_ROUTES: [(u32, u32, u32); 7] = [ (1, 6, 7), @@ -643,7 +727,12 @@ struct EarlierRoute { /// The stamp the rows of a retained main version under an earlier route's /// pending key are stored as: the target's for its conversion, the source's /// for its fence, which changed the metadata over the rows below it. -fn earlier_route_stamp(dataset: &Dataset, stamp: Option, intent: &str) -> Result { +fn earlier_route_stamp( + dataset: &Dataset, + stamp: Option, + intent: &str, + live: u32, +) -> Result { let version = dataset.version().version; let refused = |why: String| { OmniError::manifest(format!( @@ -665,6 +754,12 @@ fn earlier_route_stamp(dataset: &Dataset, stamp: Option, intent: &str) -> R which no released upgrade ran" ))); } + if target_format > live { + return Err(refused(format!( + "its intent names protocol {protocol} from v{source_format} to v{target_format}, \ + which a v{live} root cannot have completed" + ))); + } if stamp != Some(target_format) { return Err(refused(format!( "its intent targets v{target_format} and the version is stamped {stamp:?}" @@ -792,7 +887,7 @@ struct GraphHeadMetadata { parent_commit_id: Option, } -/// The reduction of a head's rows, shared by [`Stamp13Source::scan_head`] and +/// The reduction of a head's rows, shared by [`LegacyManifestSource::scan_head`] and /// the fixture writer so both decode a row one way. struct HeadFold { version: u64, @@ -967,7 +1062,7 @@ impl HeadFold { } other => { return Err(OmniError::manifest_internal(format!( - "manifest row '{}' has object_type '{other}', which no v13 row carries", + "manifest row '{}' has object_type '{other}', which no source row carries", object_ids.value(row) ))); } @@ -1320,27 +1415,47 @@ pub mod write { pub commit: Option, } - /// A `__manifest` history a stamp-13 engine would have written. Every + /// A `__manifest` history a stamp-13 engine would have written, or one a + /// 0.11.x engine would have at stamp 8 or 9 (`create_stamped`). Every /// publish overwrites the ref's whole row set, the rows it keeps first and /// the rows it writes after, replacing only the ids it writes again /// (`graph_head:*`, `schema_contract` and re-emitted `table:*`). A fork is /// a Lance ref of `__manifest` at its source's head; a retirement is the /// production `retire_branch_recoverably`. - pub struct Stamp13History { + pub struct LegacyHistory { root: String, heads: HashMap, Dataset>, + source_stamp: u32, } - impl Stamp13History { + impl LegacyHistory { /// Create the history at `root` with `genesis` as main's version 1. pub async fn create(root: &str, genesis: LegacyPublish) -> Result { + Self::create_stamped(root, genesis, SCHEMA_CONTENT_STAMP).await + } + + /// [`Self::create`] for a history whose publishes are stamped `stamp`: + /// 13, or 8 or 9, which store the flat shape and no `schema_contract` + /// row, so a publish's `contract` is not written. + pub async fn create_stamped( + root: &str, + genesis: LegacyPublish, + stamp: u32, + ) -> Result { + if stamp != SCHEMA_CONTENT_STAMP && !super::ROOT_CONTRACT_STAMPS.contains(&stamp) { + return Err(OmniError::manifest_internal(format!( + "the fixture publishes at stamp 8, 9 or {SCHEMA_CONTENT_STAMP}, not {stamp}" + ))); + } + let genesis = at_stamp(genesis, stamp); let rows = pending_rows(&genesis, &[], &HashMap::new(), None, 1)?; - let batch = compact_to_storage(&rows, &stamp_13_schema(HashMap::new())?)?; + let batch = to_storage(&rows, &publish_schema(HashMap::new(), stamp)?, stamp)?; let dataset = crate::commit::create_for_test(&crate::layout::manifest_uri(root), batch).await?; Ok(Self { root: root.to_string(), heads: HashMap::from([(None, dataset)]), + source_stamp: stamp, }) } @@ -1355,13 +1470,16 @@ pub mod write { }) } - /// Write `publish` on `native` as its next version, stamped 13, and - /// return that version. + /// Write `publish` on `native` as its next version, stamped as the + /// history is (13 unless `create_stamped` chose 8 or 9), and return + /// that version. pub async fn publish( &mut self, native: Option<&str>, publish: LegacyPublish, ) -> Result { + let stamp = self.source_stamp; + let publish = at_stamp(publish, stamp); let dataset = self.head(native)?.clone(); let version = dataset.version().version + 1; let live = logical_rows(&dataset).await?; @@ -1381,12 +1499,7 @@ pub mod write { .flatten() .map(str::to_string) .collect(); - let mut metadata = dataset.schema().metadata.clone(); - metadata.insert( - INTERNAL_SCHEMA_VERSION_KEY.to_string(), - SCHEMA_CONTENT_STAMP.to_string(), - ); - let schema = stamp_13_schema(metadata)?; + let schema = publish_schema(dataset.schema().metadata.clone(), stamp)?; let mut batches = Vec::with_capacity(live.len() + 1); for batch in &live { let keep = BooleanArray::from_iter( @@ -1396,10 +1509,10 @@ pub mod write { ); let kept = filter_record_batch(batch, &keep).map_err(OmniError::arrow_internal)?; if kept.num_rows() > 0 { - batches.push(compact_to_storage(&kept, &schema)?); + batches.push(to_storage(&kept, &schema, stamp)?); } } - batches.push(compact_to_storage(&pending, &schema)?); + batches.push(to_storage(&pending, &schema, stamp)?); self.overwrite_head(native, dataset, batches).await } @@ -1544,12 +1657,42 @@ pub mod write { } } - fn stamp_13_schema(mut metadata: HashMap) -> Result { - metadata.insert( - INTERNAL_SCHEMA_VERSION_KEY.to_string(), - SCHEMA_CONTENT_STAMP.to_string(), - ); - manifest_storage_schema(metadata, true) + /// The stored schema of a publish stamped `stamp`, over `metadata`. + fn publish_schema( + mut metadata: HashMap, + stamp: u32, + ) -> Result { + metadata.insert(INTERNAL_SCHEMA_VERSION_KEY.to_string(), stamp.to_string()); + match stored_shape(Some(stamp)) { + StoredShape::Flat => Ok(Arc::new( + flat_manifest_schema() + .as_ref() + .clone() + .with_metadata(metadata), + )), + StoredShape::Packed => manifest_storage_schema(metadata, true), + } + } + + /// A logical batch in the shape a publish stamped `stamp` stores. + fn to_storage( + batch: &RecordBatch, + schema: &arrow_schema::SchemaRef, + stamp: u32, + ) -> Result { + match stored_shape(Some(stamp)) { + StoredShape::Flat => flat_to_storage(batch, schema), + StoredShape::Packed => compact_to_storage(batch, schema), + } + } + + /// `publish` as a history stamped `stamp` writes it: below 13 no version + /// holds a `schema_contract` row. + fn at_stamp(publish: LegacyPublish, stamp: u32) -> LegacyPublish { + LegacyPublish { + contract: publish.contract.filter(|_| stamp == SCHEMA_CONTENT_STAMP), + ..publish + } } /// Every row of `dataset` in the logical schema, whatever its stored shape. diff --git a/crates/omnigraph-catalog/src/legacy/census.rs b/crates/omnigraph-catalog/src/legacy/census.rs index 73ada34a5..f63eb30b6 100644 --- a/crates/omnigraph-catalog/src/legacy/census.rs +++ b/crates/omnigraph-catalog/src/legacy/census.rs @@ -505,8 +505,8 @@ impl Contracts { finding( FindingCode::UnsupportedSource, format!( - "version {version} of {} holds no schema_contract row, and neither does its \ - head nor the head of main", + "version {version} of {} holds no schema contract, and the source gives none \ + for its head nor for the head of main", named(native) ), ) diff --git a/crates/omnigraph-catalog/src/migrations.rs b/crates/omnigraph-catalog/src/migrations.rs index d0f64e357..0e27e93ae 100644 --- a/crates/omnigraph-catalog/src/migrations.rs +++ b/crates/omnigraph-catalog/src/migrations.rs @@ -21,10 +21,11 @@ //! binary would misread as reclaimed history. Both RFC 0040 system column //! vintages (`id`/`src`/`dst` and `__id`/`__src`/`__dst`) live under that //! stamp; the vintage is read from the schema IR, never from the stamp. -//! Normal open never converts: a v13 standalone root is converted by the -//! offline `omnigraph upgrade`, whose intent, receipts and fence live here; a -//! graph below v13 is rebuilt by export and load, and normal open refuses a -//! graph that carries a pending storage-upgrade intent. +//! Normal open never converts: a v8, v9 or v13 standalone root is converted +//! by the offline `omnigraph upgrade`, whose intent, receipts and fence live +//! here; a graph at any other stamp below v14 is rebuilt by export and load, +//! and normal open refuses a graph that carries a pending storage-upgrade +//! intent. //! Fresh graphs receive their stamp atomically in the manifest Create commit. //! //! ## Forward-version protection @@ -114,12 +115,13 @@ use crate::state::{SchemaContractHead, SchemaContractRow}; /// /// v1–v13 graphs are not served by this binary (see `MIN_SUPPORTED`); the /// history is kept for provenance and to document what each stamp value meant. -/// A v13 standalone root reaches v14 through the offline `omnigraph upgrade`. +/// A v8, v9 or v13 standalone root reaches v14 through the offline +/// `omnigraph upgrade`. pub const INTERNAL_MANIFEST_SCHEMA_VERSION: u32 = 14; -/// Normal open serves only format 14. A v13 standalone root is converted by -/// the offline `omnigraph upgrade`; a graph at a lower stamp, and a -/// cluster-managed v13 graph, is rebuilt by export and load. +/// Normal open serves only format 14. A v8, v9 or v13 standalone root is +/// converted by the offline `omnigraph upgrade`; a graph at any other lower +/// stamp, and a cluster-managed graph, is rebuilt by export and load. pub const MIN_SUPPORTED_INTERNAL_SCHEMA_VERSION: u32 = 14; /// The stamp a fresh graph of the given system column vintage is born with: @@ -280,8 +282,8 @@ pub fn is_served_stamp(stamp: u32) -> bool { /// half-enforce. /// /// - `stamp > CURRENT`: the graph was written by a newer binary — upgrade omnigraph. -/// - `stamp == UPGRADE_SOURCE_FORMAT`: a standalone root is converted by the -/// offline `omnigraph upgrade`; a cluster-managed graph is rebuilt. +/// - `is_upgrade_source(stamp)` (v8, v9, v13): a standalone root is converted +/// by the offline `omnigraph upgrade`; a cluster-managed graph is rebuilt. /// - any other `stamp < MIN_SUPPORTED`: the graph was made by an older /// omnigraph whose storage format this binary does not read — rebuild via /// export/import. @@ -294,7 +296,7 @@ pub fn refuse_if_stamp_unsupported(stamp: u32) -> Result<()> { stamp, MIN_SUPPORTED_INTERNAL_SCHEMA_VERSION, INTERNAL_MANIFEST_SCHEMA_VERSION, ))); } - if stamp == UPGRADE_SOURCE_FORMAT { + if is_upgrade_source(stamp) { return Err(OmniError::manifest(format!( "__manifest is stamped at internal schema v{stamp}, but this omnigraph reads only v{min} to v{current}. \ This graph was created by omnigraph {release}. For a standalone root, stop all readers, writers and \ @@ -347,13 +349,59 @@ pub const MAX_BRANCHES: usize = 1024; #[doc(hidden)] pub const MAX_INTENT_BYTES: usize = 1024 * 1024; -/// The one route this binary runs, as the intent carries it. +/// The one protocol this binary runs, as the intent carries it; every source +/// format converts under it. #[doc(hidden)] pub const UPGRADE_PROTOCOL: u32 = 6; -/// The stamp the upgrade converts from. +/// A storage format the upgrade converts from: the two stamps of release +/// 0.11.x, whose schema contract is the three schema objects at the graph +/// root, and 13, whose contract is the `schema_contract` row of `__manifest`. #[doc(hidden)] -pub const UPGRADE_SOURCE_FORMAT: u32 = 13; +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum UpgradeSource { + Stamp8, + Stamp9, + Stamp13, +} + +impl UpgradeSource { + /// Every source, oldest first. + pub const ALL: [Self; 3] = [Self::Stamp8, Self::Stamp9, Self::Stamp13]; + + /// The stamp the live heads of this source carry. + pub const fn stamp(self) -> u32 { + match self { + Self::Stamp8 => 8, + Self::Stamp9 => 9, + Self::Stamp13 => 13, + } + } + + /// The source whose live heads are stamped `stamp`, if the upgrade converts one. + pub fn from_stamp(stamp: u32) -> Option { + Self::ALL.into_iter().find(|source| source.stamp() == stamp) + } + + /// Whether the contract is a row of `__manifest` (13) or the root objects (8, 9). + pub const fn contract_in_row(self) -> bool { + matches!(self, Self::Stamp13) + } +} + +/// The stamps the upgrade converts from, oldest first. +#[doc(hidden)] +pub const UPGRADE_SOURCE_FORMATS: [u32; 3] = [ + UpgradeSource::Stamp8.stamp(), + UpgradeSource::Stamp9.stamp(), + UpgradeSource::Stamp13.stamp(), +]; + +/// Whether a standalone root stamped `stamp` is converted by the upgrade. +#[doc(hidden)] +pub fn is_upgrade_source(stamp: u32) -> bool { + UpgradeSource::from_stamp(stamp).is_some() +} /// A live ref pinned at the version the upgrade converts; `parent_version` /// is 0 for main and the fork version for a named ref. A receipt an earlier @@ -369,8 +417,20 @@ pub struct SourceBranch { pub parent_version: u64, } +/// A live ref of the source the upgrade retires once main is fenced, as the +/// branch delete that created it would have: the `__schema_apply_lock__` ref +/// a 0.11.x schema apply did not release. +#[doc(hidden)] +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub struct SourceLock { + pub native: String, + pub identity: BranchIdentifier, +} + /// What the fence binds: the live refs at their source versions, main last, -/// the schema contract and the plan of the legacy objects. +/// the refs retired after the fence, the schema contract and the plan of the +/// legacy objects. #[doc(hidden)] #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(deny_unknown_fields)] @@ -381,12 +441,16 @@ pub struct UpgradeIntent { pub target_format: u32, pub graph_identity: String, pub branches: Vec, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub retire: Vec, #[serde(default, skip_serializing_if = "Option::is_none")] pub schema_contract: Option, pub legacy: LegacyPlan, } -/// The identity and the exact texts of main's schema contract at the fence. +/// The identity and the exact texts of main's schema contract at the fence, +/// and the name of its archive under `__history/schemas/`, written before the +/// fence: a fenced rerun reads the contract from there. #[doc(hidden)] #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(deny_unknown_fields)] @@ -394,19 +458,21 @@ pub struct UpgradeSchemaContract { pub identity: SchemaContractHead, pub source_sha256: String, pub ir_sha256: String, + pub content_sha256: String, } impl UpgradeSchemaContract { - pub fn from_row(row: &SchemaContractRow) -> Self { - Self { + pub fn from_row(row: &SchemaContractRow) -> Result { + Ok(Self { identity: row.head.clone(), source_sha256: format!("{:x}", Sha256::digest(row.source.as_bytes())), ir_sha256: format!("{:x}", Sha256::digest(row.ir.as_bytes())), - } + content_sha256: crate::history::schema_content_hash(row)?, + }) } pub fn validate_row(&self, row: &SchemaContractRow) -> Result<()> { - if *self != Self::from_row(row) { + if *self != Self::from_row(row)? { return Err(invalid( "upgrade schema contract identity or exact text changed", )); @@ -448,12 +514,9 @@ impl UpgradeIntent { self.legacy.layout.version ))); } - let route = (self.protocol, self.source_format, self.target_format) - == ( - UPGRADE_PROTOCOL, - UPGRADE_SOURCE_FORMAT, - INTERNAL_MANIFEST_SCHEMA_VERSION, - ); + let route = self.protocol == UPGRADE_PROTOCOL + && is_upgrade_source(self.source_format) + && self.target_format == INTERNAL_MANIFEST_SCHEMA_VERSION; let contract_valid = self.schema_contract.as_ref().is_some_and(|contract| { contract.identity.schema_identity_domain == self.graph_identity && contract.identity.schema_identity_version != 0 @@ -464,6 +527,7 @@ impl UpgradeIntent { .is_some_and(valid_sha256) && valid_sha256(&contract.source_sha256) && valid_sha256(&contract.ir_sha256) + && valid_sha256(&contract.content_sha256) }); let mut names = HashSet::new(); let branches_valid = !self.branches.is_empty() @@ -478,9 +542,14 @@ impl UpgradeIntent { && branch.parent_version <= branch.version && names.insert(branch.native.as_deref()) }); + let retire_valid = self + .retire + .iter() + .all(|lock| !lock.native.is_empty() && names.insert(Some(lock.native.as_str()))); if !route || !contract_valid || !branches_valid + || !retire_valid || !valid_sha256(&self.legacy.directory_sha256) || self.attempt.parse::().is_err() || self.graph_identity.is_empty() @@ -724,20 +793,51 @@ mod tests { legacy.contains("an unreleased v7 development build"), "got: {legacy}" ); + assert_eq!(UPGRADE_PROTOCOL, 6); assert_eq!( - (UPGRADE_SOURCE_FORMAT, UPGRADE_PROTOCOL), - (MIN_SUPPORTED_INTERNAL_SCHEMA_VERSION - 1, 6) + UPGRADE_SOURCE_FORMATS, + [8, 9, MIN_SUPPORTED_INTERNAL_SCHEMA_VERSION - 1] ); - for stamp in 1..UPGRADE_SOURCE_FORMAT { + for stamp in 1..INTERNAL_MANIFEST_SCHEMA_VERSION { let refusal = refuse_if_stamp_unsupported(stamp) - .expect_err("a sub-floor stamp has no in-place route") + .expect_err("a sub-floor stamp is refused") .to_string(); assert!(refusal.contains("omnigraph export"), "v{stamp}: {refusal}"); assert!( - !refusal.contains("omnigraph upgrade") && !refusal.contains("in-place"), + refusal.contains(release_for_internal_schema_version(stamp)), "v{stamp}: {refusal}" ); + if [8, 9, 13].contains(&stamp) { + assert!(is_upgrade_source(stamp), "v{stamp}"); + assert!( + refusal.contains( + "run `omnigraph upgrade --check` and then `omnigraph upgrade \ + `" + ), + "v{stamp}: {refusal}" + ); + } else { + assert!(!is_upgrade_source(stamp), "v{stamp}"); + assert!( + !refusal.contains("omnigraph upgrade") && !refusal.contains("in-place"), + "v{stamp}: {refusal}" + ); + } } + assert!(!is_upgrade_source(0) && !is_upgrade_source(INTERNAL_MANIFEST_SCHEMA_VERSION)); + let released = refuse_if_stamp_unsupported(9) + .expect_err("a v9 stamp must be refused") + .to_string(); + assert!( + released.contains( + "This graph was created by omnigraph 0.11.x. For a standalone root, stop all \ + readers, writers and maintenance" + ) && released.contains( + "A cluster-managed graph has no in-place route yet: with 0.11.x run `omnigraph \ + export > graph.jsonl`" + ), + "got: {released}" + ); let future_stamp = INTERNAL_MANIFEST_SCHEMA_VERSION + 1; let future = refuse_if_stamp_unsupported(future_stamp) .expect_err("the first unsupported future stamp must be refused") diff --git a/crates/omnigraph-catalog/src/tests.rs b/crates/omnigraph-catalog/src/tests.rs index e44af0282..5f81760db 100644 --- a/crates/omnigraph-catalog/src/tests.rs +++ b/crates/omnigraph-catalog/src/tests.rs @@ -10301,7 +10301,7 @@ fn legacy_contract(source: &str) -> SchemaContractRow { async fn legacy_main( root: &str, ) -> ( - super::legacy::write::Stamp13History, + super::legacy::write::LegacyHistory, [TableRegistration; 3], SchemaContractRow, ) { @@ -10310,14 +10310,28 @@ async fn legacy_main( /// [`legacy_main`] with the ids of its two commits given. async fn legacy_main_with_ids( + root: &str, + ids: [&str; 2], +) -> ( + super::legacy::write::LegacyHistory, + [TableRegistration; 3], + SchemaContractRow, +) { + legacy_main_stamped(root, ids, 13).await +} + +/// [`legacy_main_with_ids`] published at `stamp`. Below 13 no version holds +/// the returned contract: it stands for the one at the graph root. +async fn legacy_main_stamped( root: &str, [genesis, second]: [&str; 2], + stamp: u32, ) -> ( - super::legacy::write::Stamp13History, + super::legacy::write::LegacyHistory, [TableRegistration; 3], SchemaContractRow, ) { - use super::legacy::write::{LegacyPublish, Stamp13History}; + use super::legacy::write::{LegacyHistory, LegacyPublish}; let person = legacy_table(1, "node:Person"); let company = legacy_table(2, "node:Company"); let firm = TableRegistration { @@ -10325,7 +10339,7 @@ async fn legacy_main_with_ids( ..company.clone() }; let contract = legacy_contract("node Person {}"); - let mut history = Stamp13History::create( + let mut history = LegacyHistory::create_stamped( root, LegacyPublish { tables: vec![person.clone(), company.clone()], @@ -10334,6 +10348,7 @@ async fn legacy_main_with_ids( commit: Some(legacy_commit(genesis, 1)), ..Default::default() }, + stamp, ) .await .unwrap(); @@ -10632,7 +10647,7 @@ async fn legacy_source_admits_a_version_by_its_stamp_and_its_stored_shape() { assert_eq!(source.admits(&at_1, SourceRole::Version).unwrap(), 13); let dir = tempfile::tempdir().unwrap(); let root = dir.path().to_str().unwrap(); - let mut genesis = super::legacy::write::Stamp13History::create( + let mut genesis = super::legacy::write::LegacyHistory::create( root, super::legacy::write::LegacyPublish { tables: vec![person.clone(), company.clone()], @@ -10687,6 +10702,339 @@ async fn legacy_source_admits_a_version_by_its_stamp_and_its_stored_shape() { } } +/// A flat head stamped 8 or 9 is a live head only to the `RootContractSource` +/// of that stamp, whose head scan carries the root contract and every pin +/// with its own table version as `last_linear_version`. +#[tokio::test] +async fn root_contract_source_reads_a_flat_head_under_the_root_contract() { + use super::legacy::{ + HeadScan, LegacyManifestSource, RootContractSource, SourceRole, Stamp13Source, + VersionSchema, + }; + use std::collections::BTreeMap; + for stamp in [7u32, 10, 13, 14] { + let error = RootContractSource::new(stamp, legacy_contract("node Person {}")).unwrap_err(); + assert!( + error + .to_string() + .contains(&format!("not of a v{stamp} one")), + "{error}" + ); + } + for stamp in [8u32, 9] { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_str().unwrap(); + let (history, [person, company, firm], contract) = + legacy_main_stamped(root, ["G", "C2"], stamp).await; + let main = history.head(None).unwrap().clone(); + let source = RootContractSource::new(stamp, contract.clone()).unwrap(); + assert_eq!(source.admits(&main, SourceRole::LiveHead).unwrap(), stamp); + let error = Stamp13Source + .admits(&main, SourceRole::LiveHead) + .unwrap_err(); + assert!( + error.to_string().contains(&format!( + "version 2 is v{stamp}: the head of a live branch must be stamped v13" + )), + "{error}" + ); + + let unfilled = BTreeMap::from([ + ((person.identity, 1), legacy_read_pin(1, 1)), + ((company.identity, 1), legacy_read_pin(1, 1)), + ((person.identity, 2), legacy_read_pin(2, 2)), + ]); + let stored = HeadScan { + commits: vec![ + legacy_read_commit("G", None, 1, None), + legacy_read_commit("C2", None, 2, Some("G")), + ], + heads: HashMap::from([("main".to_string(), "C2".to_string())]), + pins: unfilled.clone(), + tombstones: BTreeMap::new(), + contract: None, + rows: 8, + }; + assert_eq!(Stamp13Source.scan_head(&main).await.unwrap(), stored); + let scan = source.scan_head(&main).await.unwrap(); + for pin in scan.pins.values() { + assert_eq!( + ( + pin.metadata.last_linear_version(), + pin.metadata.staged_version() + ), + (Some(pin.table_version), None) + ); + } + let filled = unfilled + .into_iter() + .map(|(key, pin)| { + let metadata = pin + .metadata + .clone() + .with_last_linear_version(Some(pin.table_version)); + (key, TablePin { metadata, ..pin }) + }) + .collect(); + assert_eq!( + scan, + HeadScan { + pins: filled, + contract: Some(contract), + ..stored + } + ); + + for (version, tables) in [(1, vec![person.clone(), company]), (2, vec![person, firm])] { + assert_eq!( + source + .version_schema(&main.checkout_version(version).await.unwrap(), None) + .await + .unwrap(), + VersionSchema { + tables, + contract: None, + } + ); + } + } +} + +/// A root whose live heads are stamped 9 holds no version above 9: a version +/// stamped 10 or 13 and the fence of a route to 10 are refused, a retired head +/// at 8 and the routes the released upgrade ran to 7 and 8 are admitted. +#[tokio::test] +async fn root_contract_source_admits_no_version_above_its_live_stamp() { + use super::legacy::{LegacyManifestSource, RootContractSource, SourceRole, Stamp13Source}; + let contract = legacy_contract("node Person {}"); + let source = RootContractSource::new(9, contract.clone()).unwrap(); + let dir = tempfile::tempdir().unwrap(); + let (at_13, _, _) = legacy_main(dir.path().to_str().unwrap()).await; + let dir = tempfile::tempdir().unwrap(); + let (mut history, _, _) = + legacy_main_stamped(dir.path().to_str().unwrap(), ["G", "C2"], 9).await; + let at_9 = history.head(None).unwrap().clone(); + + assert_eq!(history.restamp_for_test(None, Some(10)).await.unwrap(), 3); + for (dataset, found) in [ + (history.head(None).unwrap(), "version 3 is v10"), + (at_13.head(None).unwrap(), "version 2 is v13"), + ] { + let error = source.admits(dataset, SourceRole::Version).unwrap_err(); + assert!( + error.to_string().contains(&format!( + "{found}: a retained version must be stamped v6 to v9, or be the unstamped \ + version 1 or 2 an older init wrote" + )), + "{error}" + ); + } + + assert_eq!(history.restamp_for_test(None, Some(8)).await.unwrap(), 4); + let at_8 = history.head(None).unwrap().clone(); + assert_eq!(source.admits(&at_8, SourceRole::RetiredHead).unwrap(), 8); + let error = source.admits(&at_8, SourceRole::LiveHead).unwrap_err(); + assert!( + error + .to_string() + .contains("version 4 is v8: the head of a live branch must be stamped v9"), + "{error}" + ); + let stamp_8 = RootContractSource::new(8, contract).unwrap(); + assert_eq!(stamp_8.admits(&at_8, SourceRole::LiveHead).unwrap(), 8); + let error = stamp_8.admits(&at_9, SourceRole::RetiredHead).unwrap_err(); + assert!( + error + .to_string() + .contains("version 2 is v9: the head of a retired branch must be stamped v8 to v8"), + "{error}" + ); + + assert_eq!(history.restamp_for_test(None, Some(6)).await.unwrap(), 5); + let at_6 = history.head(None).unwrap(); + assert_eq!(source.admits(at_6, SourceRole::Version).unwrap(), 6); + let error = source.admits(at_6, SourceRole::RetiredHead).unwrap_err(); + assert!( + error + .to_string() + .contains("version 5 is v6: the head of a retired branch must be stamped v8 to v9"), + "{error}" + ); + + for (protocol, from, to) in [(1, 6, 7), (2, 7, 8)] { + let intent = legacy_route_intent(protocol, from, to); + history.fence_for_test(&intent, to).await.unwrap(); + let fenced = history.head(None).unwrap(); + assert_eq!(source.admits(fenced, SourceRole::Version).unwrap(), to); + let error = source.admits(fenced, SourceRole::LiveHead).unwrap_err(); + assert!( + error.to_string().contains("it is not a v9 source"), + "{error}" + ); + } + let version = history + .fence_for_test(&legacy_route_intent(3, 9, 10), 10) + .await + .unwrap(); + let fenced = history.head(None).unwrap(); + assert_eq!( + Stamp13Source.admits(fenced, SourceRole::Version).unwrap(), + 10 + ); + let error = source.admits(fenced, SourceRole::Version).unwrap_err(); + assert!( + error.to_string().contains(&format!( + "version {version} carries a pending storage conversion \ + (omnigraph:storage_upgrade_pending) that is not the fence or the conversion of a \ + completed earlier upgrade: its intent names protocol 3 from v9 to v10, which a v9 \ + root cannot have completed" + )), + "{error}" + ); +} + +/// The census of a stamp-9 root (main, a branch with a commit of its own, a +/// fork without one and a retired ref) records every commit under the root +/// contract, archives that one contract, and names stamp 9 in the directory. +#[tokio::test] +async fn legacy_census_of_a_stamp_9_root_records_every_commit_under_the_root_contract() { + use super::history::LegacyLayout; + use super::legacy::write::LegacyPublish; + use super::legacy::{FindingCode, RootContractSource, Stamp13Source}; + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_str().unwrap(); + let [g, c2, f3, r3] = [1, 2, 3, 4].map(legacy_id); + let (mut history, [person, _, _], contract) = legacy_main_stamped(root, [&g, &c2], 9).await; + let feature = legacy_native("feature"); + let retired = legacy_native("old"); + for (native, id, table_version) in [(&feature, &f3, 3), (&retired, &r3, 4)] { + assert_eq!(history.fork(None, native).await.unwrap(), 2); + let publish = LegacyPublish { + pins: vec![legacy_pin(&person, table_version)], + commit: Some(legacy_commit(id, 3)), + ..Default::default() + }; + assert_eq!(history.publish(Some(native), publish).await.unwrap(), 3); + } + history.retire(&retired).await.unwrap(); + let fresh = legacy_native("fresh"); + assert_eq!(history.fork(Some(&feature), &fresh).await.unwrap(), 3); + + let input = legacy_census_input(&history, &[]).await; + let (code, message) = legacy_finding(super::legacy::census(root, &input, &Stamp13Source).await); + assert_eq!(code, FindingCode::UnsupportedSource); + assert!( + message.contains("is v9: the head of a live branch must be stamped v13"), + "{message}" + ); + + let source = RootContractSource::new(9, contract.clone()).unwrap(); + let census = super::legacy::census(root, &input, &source).await.unwrap(); + assert_eq!(census.source_stamp, 9); + assert_eq!(census.contracts.iter().collect::>(), vec![&contract]); + assert_eq!( + ( + census.counts.live_refs, + census.counts.retired_refs, + census.counts.legacy_commits, + census.counts.schema_contents, + ), + (3, 1, 4, 1) + ); + let hash = super::history::schema_content_hash(&contract).unwrap(); + let records: Vec<_> = census + .chains + .iter() + .flat_map(|chain| &chain.records) + .chain(census.heads.iter().map(|head| &head.record)) + .collect(); + assert_eq!(records.len(), 4 + 3); + for record in records { + assert_eq!(record.commit.schema_content_hash.as_ref(), Some(&hash)); + for table in &record.tables { + let TableState::Pinned(pin) = &table.state else { + panic!("the fixture drops no table: {table:?}"); + }; + assert_eq!( + pin.metadata.last_linear_version(), + Some(pin.table_version), + "{table:?}" + ); + } + } + let head_of = |native: Option<&str>| { + &census + .heads + .iter() + .find(|head| head.native.as_deref() == native) + .unwrap() + .record + .commit + .graph_commit_id + }; + assert_eq!( + [None, Some(feature.as_str()), Some(fresh.as_str())].map(head_of), + [&c2, &f3, &f3] + ); + + let layout = LegacyLayout::CURRENT; + let plan = super::legacy::plan(&census, &layout).unwrap(); + let session = crate::lance_access::control_session(); + super::legacy::materialize(root, &session, &census, &plan) + .await + .unwrap(); + let directory = super::history::legacy_directory(root, &session, None) + .await + .unwrap() + .unwrap(); + assert_eq!(directory.source_stamp, 9); + let mut converted = history.head(None).unwrap().schema().metadata.clone(); + converted.insert( + super::migrations::INTERNAL_SCHEMA_VERSION_KEY.to_string(), + "14".to_string(), + ); + let (_, batch) = + super::legacy::conversion_batch(root, &session, &census.heads[0].record, converted) + .await + .unwrap(); + let rows = super::state::rows_of_batch(&batch).unwrap(); + assert_eq!(rows.schema_contract.as_ref(), Some(&contract)); + assert_eq!(rows.tables, census.heads[0].record.tables); +} + +/// `validate` owns protocol 6 from each of the three source formats to 14 and +/// no other route. +#[test] +fn migrations_intent_validates_every_source_format_and_no_other_route() { + let route = |protocol, source_format, target_format| { + super::migrations::UpgradeIntent { + protocol, + source_format, + target_format, + ..migrations_intent(1) + } + .validate() + }; + for source_format in super::migrations::UPGRADE_SOURCE_FORMATS { + route(6, source_format, 14).unwrap(); + } + for (protocol, source_format, target_format) in [ + (6, 10, 14), + (6, 12, 14), + (5, 9, 14), + (6, 9, 13), + (6, 14, 14), + ] { + let error = route(protocol, source_format, target_format).unwrap_err(); + assert!( + error + .to_string() + .contains("unsupported or ambiguous storage upgrade intent"), + "({protocol}, {source_format}, {target_format}): {error}" + ); + } +} + fn legacy_id(number: u128) -> String { ulid::Ulid::from(number).to_string() } @@ -10705,7 +11053,7 @@ fn legacy_pinned(table: &TableRegistration, table_version: u64, clock: u64) -> T /// The census input of every ref of `history` at its head, less the refs /// named in `gone`, as an inventory that cannot see them would pin it. async fn legacy_census_input( - history: &super::legacy::write::Stamp13History, + history: &super::legacy::write::LegacyHistory, gone: &[&str], ) -> super::legacy::CensusInput { use super::legacy::{CensusInput, CensusRef}; @@ -10756,7 +11104,7 @@ async fn legacy_census_input( } async fn legacy_census( - history: &super::legacy::write::Stamp13History, + history: &super::legacy::write::LegacyHistory, gone: &[&str], ) -> std::result::Result { let input = legacy_census_input(history, gone).await; @@ -11251,13 +11599,13 @@ async fn legacy_census_adopts_orphans_and_refuses_a_first_parent_gap() { #[tokio::test] async fn legacy_census_verifies_versions_without_a_commit() { use super::legacy::FindingCode; - use super::legacy::write::{LegacyPublish, Stamp13History}; + use super::legacy::write::{LegacyHistory, LegacyPublish}; let dir = tempfile::tempdir().unwrap(); let root = dir.path().to_str().unwrap(); let [g, c4, c7] = [1, 2, 3].map(legacy_id); let person = legacy_table(1, "node:Person"); let born = legacy_contract("node Person {}"); - let mut history = Stamp13History::create( + let mut history = LegacyHistory::create( root, LegacyPublish { tables: vec![person.clone()], @@ -11597,9 +11945,9 @@ async fn legacy_upgraded_main( g: &str, person: &TableRegistration, conversion: super::legacy::write::LegacyPublish, -) -> super::legacy::write::Stamp13History { - use super::legacy::write::{LegacyPublish, Stamp13History}; - let mut history = Stamp13History::create( +) -> super::legacy::write::LegacyHistory { + use super::legacy::write::{LegacyHistory, LegacyPublish}; + let mut history = LegacyHistory::create( root, LegacyPublish { tables: vec![person.clone()], @@ -11763,7 +12111,7 @@ async fn legacy_census_reads_through_the_fence_and_conversion_of_an_earlier_rout /// the source stored after it, which the fork holds, not under a later one. #[tokio::test] async fn legacy_census_records_a_contractless_head_under_the_contract_its_forks_hold() { - use super::legacy::write::{LegacyPublish, Stamp13History}; + use super::legacy::write::{LegacyHistory, LegacyPublish}; let dir = tempfile::tempdir().unwrap(); let root = dir.path().to_str().unwrap(); let [g, c3, c5] = [1, 2, 3].map(legacy_id); @@ -11771,7 +12119,7 @@ async fn legacy_census_records_a_contractless_head_under_the_contract_its_forks_ let born = legacy_contract("node Person {}"); let converted = legacy_contract("node Person { name: String }"); let changed = legacy_contract("node Person { name: String, age: I32 }"); - let mut history = Stamp13History::create( + let mut history = LegacyHistory::create( root, LegacyPublish { tables: vec![person.clone()], @@ -11879,7 +12227,7 @@ where } async fn legacy_census_rewriting( - history: &super::legacy::write::Stamp13History, + history: &super::legacy::write::LegacyHistory, rewrite: impl Fn(&mut super::legacy::LegacyCommit) + Send + Sync, ) -> (super::legacy::FindingCode, String) { let input = legacy_census_input(history, &[]).await; @@ -12114,7 +12462,7 @@ async fn legacy_census_refuses_an_own_head_its_graph_head_row_does_not_name() { #[tokio::test] async fn legacy_census_counts_an_empty_version_below_the_genesis_commit() { use super::legacy::FindingCode; - use super::legacy::write::{LegacyPublish, Stamp13History}; + use super::legacy::write::{LegacyHistory, LegacyPublish}; let dir = tempfile::tempdir().unwrap(); let root = dir.path().to_str().unwrap(); let [g, c2] = [1, 2].map(legacy_id); @@ -12123,7 +12471,7 @@ async fn legacy_census_counts_an_empty_version_below_the_genesis_commit() { contract: Some(legacy_contract("node Person {}")), ..Default::default() }; - let mut history = Stamp13History::create(root, contract_only).await.unwrap(); + let mut history = LegacyHistory::create(root, contract_only).await.unwrap(); let genesis = LegacyPublish { tables: vec![person.clone()], pins: vec![legacy_pin(&person, 1)], @@ -12167,7 +12515,7 @@ async fn legacy_census_counts_an_empty_version_below_the_genesis_commit() { /// A stamp-13 root with its legacy history written, as the upgrade leaves it /// before it fences main. struct LegacyServed { - history: super::legacy::write::Stamp13History, + history: super::legacy::write::LegacyHistory, census: super::legacy::LegacyCensus, contract: SchemaContractRow, feature: String, @@ -12257,7 +12605,7 @@ async fn legacy_served(root: &str) -> LegacyServed { #[tokio::test] async fn legacy_record_at_resolves_a_version_by_writer_parent_walk_and_fence() { use super::history::ExtentCache; - use super::legacy::write::{LegacyPublish, Stamp13History}; + use super::legacy::write::{LegacyHistory, LegacyPublish}; use super::legacy::{LegacyAt, record_at}; let dir = tempfile::tempdir().unwrap(); let root = dir.path().to_str().unwrap(); @@ -12350,7 +12698,7 @@ async fn legacy_record_at_resolves_a_version_by_writer_parent_walk_and_fence() { contract: Some(legacy_contract("node Person {}")), ..Default::default() }; - let mut history = Stamp13History::create(root, contract_only).await.unwrap(); + let mut history = LegacyHistory::create(root, contract_only).await.unwrap(); let genesis = LegacyPublish { tables: vec![person.clone()], pins: vec![legacy_pin(&person, 1)], @@ -12703,7 +13051,8 @@ fn migrations_intent(version: u64) -> super::migrations::UpgradeIntent { parent_version: 0, }, ], - schema_contract: Some(UpgradeSchemaContract::from_row(&contract)), + retire: Vec::new(), + schema_contract: Some(UpgradeSchemaContract::from_row(&contract).unwrap()), legacy: super::legacy::LegacyPlan { layout: super::history::LegacyLayout::CURRENT, commits: 3, diff --git a/crates/omnigraph-cli/src/cli.rs b/crates/omnigraph-cli/src/cli.rs index 01b49d12b..e9ca5e6b2 100644 --- a/crates/omnigraph-cli/src/cli.rs +++ b/crates/omnigraph-cli/src/cli.rs @@ -317,8 +317,8 @@ pub(crate) enum Command { #[arg(long)] force: bool, }, - /// Convert a standalone graph's storage offline from format v13 to the - /// format this binary serves (v14); older formats are rebuilt by export and load + /// Convert a standalone graph's storage offline from format v8, v9 or v13 to + /// the format this binary serves (v14); other formats are rebuilt by export and load Upgrade { /// Standalone graph storage URI; alternatively use --store uri: Option, diff --git a/crates/omnigraph-cli/tests/crossversion_upgrade.rs b/crates/omnigraph-cli/tests/crossversion_upgrade.rs index 5f634224e..06a7c84dd 100644 --- a/crates/omnigraph-cli/tests/crossversion_upgrade.rs +++ b/crates/omnigraph-cli/tests/crossversion_upgrade.rs @@ -20,12 +20,17 @@ //! rebuilt from a 0.9 export. //! `OMNIGRAPH_V6_BIN` (the released 0.10.x CLI) proves both directions of the //! v6/v10 fence (RFC 0062 registration clock, RFC 0042 native-ref retirement -//! metadata and RFC 0040 system columns). +//! metadata and RFC 0040 system columns); that case skips when the variable +//! is unset, as the older fences do. //! `OMNIGRAPH_V13_BIN` (a main build that writes storage format 13, CI builds -//! `c0a4519f`) drives the one in-place route: the offline `omnigraph upgrade` -//! of a genuine stamp-13 root to format 14 with its branches and commit -//! history kept. With `OMNIGRAPH_REQUIRE_STORAGE_UPGRADE_TESTS=1` an unset -//! variable fails instead of skipping. +//! `c0a4519f`) and `OMNIGRAPH_V011_BIN` (the released 0.11.0 CLI, which writes +//! storage format 9 and converts a 0.10.x graph to format 8 or 9) drive the +//! in-place route: the offline `omnigraph upgrade` of a genuine stamp-13, +//! stamp-9 or stamp-8 root to format 14 with its branches and commit history +//! kept. Those two, and `OMNIGRAPH_V6_BIN` where a journey starts from a 0.10.x +//! graph, fall back to a binary under `target/storage-upgrade-binaries/`; with +//! `OMNIGRAPH_REQUIRE_STORAGE_UPGRADE_TESTS=1` an unresolved predecessor of a +//! journey fails instead of skipping. mod support; @@ -80,16 +85,39 @@ fn v5_bin() -> Option { Some(path) } -/// Resolve the final internal-v6 binary: the last release that wrote internal -/// schema v6. +/// Resolve the final internal-v6 binary for the v6/v10 fence: the last release +/// that wrote internal schema v6, from `OMNIGRAPH_V6_BIN` alone. Unset skips, +/// as for the older fences; the fence is never required. fn v6_bin() -> Option { - let path = PathBuf::from(std::env::var_os("OMNIGRAPH_V6_BIN")?); + let Some(path) = std::env::var_os("OMNIGRAPH_V6_BIN").map(PathBuf::from) else { + eprintln!( + "skipping immediate-predecessor v6 upgrade test: OMNIGRAPH_V6_BIN is not set to a 0.10.x binary" + ); + return None; + }; assert!( - path.exists() && path.is_file(), + path.is_file(), "OMNIGRAPH_V6_BIN is set but is not a binary file: {} \ (unset it to skip, or point it at the released 0.10.x omnigraph binary (the last internal-v6 writer))", path.display(), ); + Some(released_v6(path)) +} + +/// Resolve the 0.10.x CLI a storage upgrade journey starts its graph with: +/// `OMNIGRAPH_V6_BIN`, else `target/storage-upgrade-binaries/v0.10.0/`, and +/// required under `OMNIGRAPH_REQUIRE_STORAGE_UPGRADE_TESTS`. +fn v6_bin_for_storage_upgrade() -> Option { + storage_upgrade_predecessor( + "OMNIGRAPH_V6_BIN", + "v0.10.0", + "the released 0.10.x omnigraph binary (the last internal-v6 writer)", + ) + .map(released_v6) +} + +/// `path`, once it has reported itself as a released 0.10.x binary. +fn released_v6(path: PathBuf) -> PathBuf { let version = run_old(&path, &["version"]); assert_ok("v6 version", &version); let reported = String::from_utf8_lossy(&version.stdout); @@ -97,32 +125,57 @@ fn v6_bin() -> Option { reported.contains("omnigraph 0.10."), "OMNIGRAPH_V6_BIN must be a released 0.10.x binary (the last internal-v6 writer), got: {reported}", ); - Some(path) + path } -/// Resolve the stamp-13 predecessor: `OMNIGRAPH_V13_BIN`, else a binary placed -/// under `target/storage-upgrade-binaries/stamp-13/`. The journey proves the -/// stamp by behaviour, so any main build that writes format 13 serves locally. +/// Resolve the stamp-13 predecessor: `OMNIGRAPH_V13_BIN`, else a binary under +/// `target/storage-upgrade-binaries/stamp-13/`. The journey proves the stamp +/// by behaviour, so any main build that writes format 13 serves locally. fn v13_bin() -> Option { - const VARIABLE: &str = "OMNIGRAPH_V13_BIN"; + storage_upgrade_predecessor( + "OMNIGRAPH_V13_BIN", + "stamp-13", + "an omnigraph binary built from a stamp-13 main commit", + ) +} + +/// Resolve the released 0.11.0 CLI: `OMNIGRAPH_V011_BIN`, else a binary under +/// `target/storage-upgrade-binaries/v0.11.0/`. Main builds 10 to 13 print the +/// same version, so the journeys prove the stamp it writes by behaviour. +fn v011_bin() -> Option { + storage_upgrade_predecessor( + "OMNIGRAPH_V011_BIN", + "v0.11.0", + "the released 0.11.0 omnigraph binary", + ) +} + +/// A predecessor of the in-place route: `variable`, else the binary under +/// `target/storage-upgrade-binaries//`. Neither one skips, or +/// fails under `OMNIGRAPH_REQUIRE_STORAGE_UPGRADE_TESTS`. +fn storage_upgrade_predecessor( + variable: &str, + directory: &str, + described: &str, +) -> Option { let local = Path::new(env!("CARGO_MANIFEST_DIR")) - .join("../../target/storage-upgrade-binaries/stamp-13") + .join("../../target/storage-upgrade-binaries") + .join(directory) .join(format!("omnigraph{}", std::env::consts::EXE_SUFFIX)); - let selected = std::env::var_os(VARIABLE) + let selected = std::env::var_os(variable) .map(PathBuf::from) .or_else(|| local.is_file().then_some(local)); let Some(path) = selected else { assert!( std::env::var_os("OMNIGRAPH_REQUIRE_STORAGE_UPGRADE_TESTS").is_none(), - "required storage upgrade predecessor {VARIABLE} is unset" + "required storage upgrade predecessor {variable} is unset" ); - eprintln!("skipping genuine v13 storage upgrade: {VARIABLE} is unset"); + eprintln!("skipping genuine {directory} storage upgrade: {variable} is unset"); return None; }; assert!( path.is_file(), - "{VARIABLE} is not a binary file: {} \ - (unset it to skip, or point it at an omnigraph binary built from a stamp-13 main commit)", + "{variable} is not a binary file: {} (unset it to skip, or point it at {described})", path.display(), ); Some(path) @@ -748,9 +801,6 @@ fn current_v10_refuses_and_rebuilds_genuine_v5_and_v5_refuses_v10() { #[test] fn current_v10_refuses_and_rebuilds_genuine_v6_and_v6_refuses_v10() { let Some(v6) = v6_bin() else { - eprintln!( - "skipping immediate-predecessor v6 upgrade test: OMNIGRAPH_V6_BIN is not set to a released 0.10.x binary" - ); return; }; @@ -1528,13 +1578,57 @@ fn storage_upgrade_refuses_cluster_path_aliases() { } const V13_HANDLER: &str = "history-lance-files-v13-to-v14"; +const V9_HANDLER: &str = "history-lance-files-v9-to-v14"; +const V8_HANDLER: &str = "history-lance-files-v8-to-v14"; + +/// The schema contract a root below stamp 13 keeps as three root objects. +const ROOT_SCHEMA_OBJECTS: [&str; 3] = ["_schema.pg", "_schema.ir.json", "__schema_state.json"]; + +/// The live branches of the journey graph once the predecessor is done with +/// it: `review` is the second branch of that name (the first was merged into +/// main and deleted) and `child` is a fork of the deleted `temp`. +const JOURNEY_LIVE_BRANCHES: [&str; 3] = ["main", "review", "child"]; + +/// Writes the schema, the rows and the queries every in-place journey starts +/// from under `directory`, and returns their paths in that order. +fn journey_fixture(directory: &Path) -> [PathBuf; 3] { + let schema = directory.join("journey.pg"); + let data = directory.join("journey.jsonl"); + let queries = directory.join("journey.gq"); + std::fs::write( + &schema, + format!( + "{}\nedge Cites: Doc -> Doc {{ note: String }}\n", + std::fs::read_to_string(fixture("search.pg")).unwrap() + ), + ) + .unwrap(); + std::fs::write( + &data, + format!( + "{}\n{}\n", + std::fs::read_to_string(fixture("search.jsonl")) + .unwrap() + .trim_end(), + r#"{"edge":"Cites","id":"citation-1","from":"ml-intro","to":"dl-basics","data":{"note":"preserved edge"}}"#, + ), + ) + .unwrap(); + std::fs::write(&queries, journey_query_source()).unwrap(); + [schema, data, queries] +} -/// The live branches of the stamp-13 journey graph once the predecessor is -/// done with it: `review` is the second branch of that name (the first was -/// merged into main and deleted) and `child` is a fork of the deleted `temp`. -const V13_LIVE_BRANCHES: [&str; 3] = ["main", "review", "child"]; +fn assert_writes_format(old: &Path, uri: &str, format: u32) { + let snapshot = run_old(old, &["snapshot", uri, "--json"]); + assert_ok("predecessor snapshot", &snapshot); + assert_eq!( + support::parse_stdout_json(&snapshot)["internal_schema_version"], + format, + "the predecessor must serve this root at storage format {format}" + ); +} -fn v13_query_source() -> &'static str { +fn journey_query_source() -> &'static str { r#" query docs() { match { $d: Doc } @@ -1579,7 +1673,7 @@ fn journey_rows( let output = match binary { Some(old) => { let output = run_old(old, &args); - assert_ok("stamp-13 query", &output); + assert_ok("predecessor query", &output); output } None => output_success(cli().args(args)), @@ -1589,6 +1683,74 @@ fn journey_rows( rows } +/// The queries over the property the stamp-9 journey's schema apply adds to +/// `Doc`, kept apart from [`journey_query_source`] so the binaries that never +/// see that property compile the shared file. +const RATED_QUERY_SOURCE: &str = r#" +query rated() { + match { $d: Doc } + return { $d.slug, $d.rating } + order { $d.slug } +} +query rate() { update Doc set { rating: 5 } where slug = "ml-intro" } +"#; + +/// What a `query --snapshot ` answered: rows, the refusal of a commit whose +/// pinned table version a `cleanup` reclaimed, or ([`property_read`] only) the +/// refusal of a property added later. Any other exit fails the journey. +#[derive(Debug, PartialEq)] +enum SnapshotRead { + Rows(serde_json::Value), + Reclaimed(u64), + NoSuchProperty, +} + +/// [`snapshot_read`] over `property`, which a schema apply added after some of +/// the commits: a refusal naming it is an answer for a commit written before. +fn property_read(label: &str, property: &str, output: std::process::Output) -> SnapshotRead { + if !output.status.success() { + let stderr = String::from_utf8_lossy(&output.stderr); + if stderr.contains(property) && !stderr.contains(" was reclaimed") { + return SnapshotRead::NoSuchProperty; + } + } + snapshot_read(label, output) +} + +fn snapshot_read(label: &str, output: std::process::Output) -> SnapshotRead { + if output.status.success() { + let mut rows = support::parse_stdout_json(&output)["rows"].clone(); + normalize_f32_and_nulls(&mut rows); + return SnapshotRead::Rows(rows); + } + let stderr = String::from_utf8_lossy(&output.stderr); + let reclaimed = stderr + .split_once("historical published dataset version ") + .and_then(|(_, rest)| rest.split_once(" was reclaimed")) + .and_then(|(version, _)| version.parse().ok()) + .unwrap_or_else(|| { + panic!( + "{label} must be served or refused as reclaimed, got status {:?}\nstdout:\n{}\nstderr:\n{stderr}", + output.status, + String::from_utf8_lossy(&output.stdout), + ) + }); + SnapshotRead::Reclaimed(reclaimed) +} + +/// `served` answers as `expected` did: the same rows, or a reclaimed refusal +/// on both sides (each binary names the first pinned table version it misses). +fn assert_same_answer(label: &str, served: &SnapshotRead, expected: &SnapshotRead) { + match (served, expected) { + (SnapshotRead::Rows(served), SnapshotRead::Rows(expected)) => { + assert_eq!(served, expected, "{label}"); + } + (SnapshotRead::Reclaimed(_), SnapshotRead::Reclaimed(_)) + | (SnapshotRead::NoSuchProperty, SnapshotRead::NoSuchProperty) => {} + (served, expected) => panic!("{label}: served {served:?}, the predecessor {expected:?}"), + } +} + /// A commit object the predecessor listed: every field it printed must hold /// the same value in the current listing, which may add fields. fn assert_lists_commit(listing: &serde_json::Value, expected: &serde_json::Value) { @@ -1609,50 +1771,93 @@ fn genuine_v13_storage_upgrade_preserves_history() { let Some(old) = v13_bin() else { return; }; - let old = old.as_path(); + storage_upgrade_preserves_history(&old, 13, V13_HANDLER, None); +} + +/// Stamp 9 as the released 0.11.0 writes it, with a schema apply and an +/// `optimize` in its history and no predecessor `cleanup`. +#[test] +fn genuine_v0_11_0_storage_upgrade_preserves_history() { + let Some(old) = v011_bin() else { + return; + }; + storage_upgrade_preserves_history(&old, 9, V9_HANDLER, None); +} + +/// Stamp 8: a branched 0.10.0 graph the released 0.11.0 converted with +/// `upgrade --to-format 8`, so its history holds stamp-6 versions and the +/// receipts of the two earlier routes. +#[test] +fn genuine_v0_10_0_to_stamp_8_storage_upgrade_preserves_history() { + let (Some(old), Some(born_with)) = (v011_bin(), v6_bin_for_storage_upgrade()) else { + return; + }; + let born_with = BornWith { + binary: &born_with, + to_format: Some(8), + }; + storage_upgrade_preserves_history(&old, 8, V8_HANDLER, Some(born_with)); +} + +/// Stamp 9 as the released 0.11.0's default `upgrade` (6 to 7 to 8 to 9, main +/// alone) leaves a branched 0.10.0 graph: the commits before the system-column +/// respelling pin `id`/`src`/`dst` tables under the one `__id` root contract. +#[test] +fn genuine_v0_10_0_to_stamp_9_by_default_storage_upgrade_preserves_history() { + let (Some(old), Some(born_with)) = (v011_bin(), v6_bin_for_storage_upgrade()) else { + return; + }; + let born_with = BornWith { + binary: &born_with, + to_format: None, + }; + storage_upgrade_preserves_history(&old, 9, V9_HANDLER, Some(born_with)); +} + +/// The 0.10.x CLI that wrote a journey graph before the released 0.11.0 took +/// it over, and the `--to-format` that 0.11.0 `upgrade` was given (`None`: its +/// default target). +struct BornWith<'a> { + binary: &'a Path, + to_format: Option, +} + +/// The handlers the released 0.11.0 reports for its routes from stamp 6, in +/// order: `--to-format 8` runs the first two, its default target all three. +const V0_11_0_HANDLERS_FROM_6: [&str; 3] = [ + "registration-clocks-v6-to-v7", + "native-retirement-v7-to-v8", + "system-columns-v8-to-v9", +]; + +/// The journey of one genuine source: `old` serves the root at +/// `source_format`, the current binary converts it through `handler`. +/// `born_with` is the 0.10.x CLI that wrote the graph before `old` took it over. +fn storage_upgrade_preserves_history( + old: &Path, + source_format: u32, + handler: &str, + born_with: Option>, +) { let temp = tempdir().unwrap(); let graph = temp.path().join("standalone.omni"); let uri = graph.to_str().unwrap(); - let schema = temp.path().join("stamp13.pg"); - let data = temp.path().join("stamp13.jsonl"); - let queries = temp.path().join("stamp13.gq"); - std::fs::write( - &schema, - format!( - "{}\nedge Cites: Doc -> Doc {{ note: String }}\n", - std::fs::read_to_string(fixture("search.pg")).unwrap() - ), - ) - .unwrap(); - std::fs::write( - &data, - format!( - "{}\n{}\n", - std::fs::read_to_string(fixture("search.jsonl")) - .unwrap() - .trim_end(), - r#"{"edge":"Cites","id":"citation-1","from":"ml-intro","to":"dl-basics","data":{"note":"preserved edge"}}"#, - ), - ) - .unwrap(); - std::fs::write(&queries, v13_query_source()).unwrap(); + let [schema, data, queries] = journey_fixture(temp.path()); let query_path = queries.to_str().unwrap(); + let first = born_with.as_ref().map_or(old, |born| born.binary); + let released_writer = source_format < 13 && born_with.is_none(); assert_ok( - "stamp-13 init", - &run_old(old, &["init", "--schema", schema.to_str().unwrap(), uri]), - ); - let born = run_old(old, &["snapshot", uri, "--json"]); - assert_ok("stamp-13 snapshot", &born); - assert_eq!( - support::parse_stdout_json(&born)["internal_schema_version"], - 13, - "OMNIGRAPH_V13_BIN must be a build that writes storage format 13" + "predecessor init", + &run_old(first, &["init", "--schema", schema.to_str().unwrap(), uri]), ); + if born_with.is_none() { + assert_writes_format(old, uri, source_format); + } assert_ok( - "stamp-13 load", + "predecessor load", &run_old( - old, + first, &[ "load", "--mode", @@ -1663,11 +1868,11 @@ fn genuine_v13_storage_upgrade_preserves_history() { ], ), ); - let old_mutate = |name: &str, branch: &str, params: &str| { + let mutate = |binary: &Path, name: &str, branch: &str, params: &str| { assert_ok( - "stamp-13 mutate", + "predecessor mutate", &run_old( - old, + binary, &[ "mutate", name, "--query", query_path, "--store", uri, "--branch", branch, "--params", params, @@ -1675,29 +1880,170 @@ fn genuine_v13_storage_upgrade_preserves_history() { ), ); }; - let old_branch = |args: &[&str]| { + let head_commit = |branch: &str| { + let output = run_old(old, &["commit", "list", uri, "--branch", branch, "--json"]); + assert_ok("predecessor commit list", &output); + let listing = support::parse_stdout_json(&output); + listing["commits"] + .as_array() + .unwrap() + .iter() + .max_by_key(|commit| commit["graph_manifest_version"].as_u64().unwrap()) + .unwrap()["graph_commit_id"] + .as_str() + .unwrap() + .to_owned() + }; + let property_evolution = released_writer.then(|| { + let before_apply = head_commit("main"); + let widened = temp.path().join("journey-widened.pg"); + std::fs::write( + &widened, + format!( + "{}\nnode Tag {{ name: String @key }}\n", + std::fs::read_to_string(&schema).unwrap().replace( + " embedding: Vector(4)\n", + " embedding: Vector(4)\n rating: I32?\n" + ) + ), + ) + .unwrap(); + assert_ok( + "predecessor schema apply", + &run_old( + old, + &[ + "schema", + "apply", + uri, + "--schema", + widened.to_str().unwrap(), + ], + ), + ); + let rated = temp.path().join("journey-rated.gq"); + std::fs::write(&rated, RATED_QUERY_SOURCE).unwrap(); + let rated_path = rated.to_str().unwrap().to_owned(); + assert_ok( + "predecessor mutate of the added property", + &run_old( + old, + &[ + "mutate", + "rate", + "--query", + &rated_path, + "--store", + uri, + "--branch", + "main", + "--params", + "{}", + ], + ), + ); + let after_apply = head_commit("main"); + assert_ne!(before_apply, after_apply); + let answers = [before_apply, after_apply].map(|id| { + let args = [ + "query", + "rated", + "--query", + rated_path.as_str(), + "--store", + uri, + "--snapshot", + id.as_str(), + "--params", + "{}", + "--json", + ]; + let answer = property_read( + &format!("0.11.0 rated at {id}"), + "rating", + run_old(old, &args), + ); + (id, answer) + }); + assert!( + matches!(answers[1].1, SnapshotRead::Rows(_)), + "0.11.0 must serve the added property at the commit that set it: {:?}", + answers[1] + ); + (rated_path, answers) + }); + let branch = |binary: &Path, args: &[&str]| { let mut full = vec!["branch"]; full.extend_from_slice(args); full.extend_from_slice(&["--uri", uri]); - assert_ok("stamp-13 branch", &run_old(old, &full)); + assert_ok("predecessor branch", &run_old(binary, &full)); }; - old_branch(&["create", "review"]); - old_mutate("retitle", "main", r#"{"title":"organism main"}"#); - old_mutate("remove", "review", "{}"); - old_branch(&["merge", "review", "--into", "main"]); - old_branch(&["delete", "review"]); - old_branch(&["create", "review"]); - old_mutate("retitle", "review", r#"{"title":"organism recreated"}"#); - old_branch(&["create", "temp"]); - old_mutate("retitle", "temp", r#"{"title":"organism temp"}"#); - old_branch(&["create", "child", "--from", "temp"]); - old_branch(&["delete", "temp"]); - - let histories: Vec = V13_LIVE_BRANCHES + let fork_temp = |binary: &Path| { + branch(binary, &["create", "temp"]); + mutate(binary, "retitle", "temp", r#"{"title":"organism temp"}"#); + }; + branch(first, &["create", "review"]); + mutate(first, "retitle", "main", r#"{"title":"organism main"}"#); + mutate(first, "remove", "review", "{}"); + let written_before_conversion = born_with.as_ref().map(|born| { + fork_temp(born.binary); + let explicit = born.to_format.map(|format| format.to_string()); + let mut args = vec!["upgrade", uri, "--json"]; + if let Some(explicit) = &explicit { + args.extend(["--to-format", explicit]); + } + let converted = run_old(old, &args); + assert_ok("predecessor upgrade of the 0.10.x graph", &converted); + let report = support::parse_stdout_json(&converted); + assert_eq!(report["outcome"], "completed", "{report}"); + assert_eq!(report["observed_format"], 6, "{report}"); + assert_eq!(report["target_format"], source_format, "{report}"); + assert_eq!( + report["target_defaulted"], + born.to_format.is_none(), + "{report}" + ); + let handlers = &V0_11_0_HANDLERS_FROM_6[..usize::try_from(source_format - 6).unwrap()]; + assert_eq!( + report["completed_handlers"], + serde_json::json!(handlers), + "{report}" + ); + assert_writes_format(old, uri, source_format); + let output = run_old(old, &["commit", "list", uri, "--branch", "main", "--json"]); + assert_ok("predecessor commit list", &output); + let ids: Vec = support::parse_stdout_json(&output)["commits"] + .as_array() + .unwrap() + .iter() + .map(|commit| commit["graph_commit_id"].as_str().unwrap().to_owned()) + .collect(); + assert!(ids.len() >= 2, "{ids:?}"); + ids + }); + branch(old, &["merge", "review", "--into", "main"]); + branch(old, &["delete", "review"]); + branch(old, &["create", "review"]); + mutate( + old, + "retitle", + "review", + r#"{"title":"organism recreated"}"#, + ); + if born_with.is_none() { + fork_temp(old); + } + branch(old, &["create", "child", "--from", "temp"]); + branch(old, &["delete", "temp"]); + if released_writer { + assert_ok("predecessor optimize", &run_old(old, &["optimize", uri])); + } + + let histories: Vec = JOURNEY_LIVE_BRANCHES .into_iter() .map(|branch| { let output = run_old(old, &["commit", "list", uri, "--branch", branch, "--json"]); - assert_ok("stamp-13 commit list", &output); + assert_ok("predecessor commit list", &output); support::parse_stdout_json(&output)["commits"].clone() }) .collect(); @@ -1734,33 +2080,56 @@ fn genuine_v13_storage_upgrade_preserves_history() { "the predecessor wrote fewer commits than the journey needs: {:?}", historical_rows.keys().collect::>() ); - let exports: Vec<_> = V13_LIVE_BRANCHES + if let Some(ids) = &written_before_conversion { + for id in ids { + assert!( + historical_rows.contains_key(id), + "commit {id}, written before the predecessor's conversion, must be read before \ + and after the upgrade: {:?}", + historical_rows.keys().collect::>() + ); + } + } + let exports: Vec<_> = JOURNEY_LIVE_BRANCHES .into_iter() .map(|branch| { let output = run_old(old, &["export", uri, "--branch", branch]); - assert_ok("stamp-13 export", &output); + assert_ok("predecessor export", &output); canonical_export_rows(&output.stdout) }) .collect(); + let predecessor_schema = (source_format < 13).then(|| { + let output = run_old(old, &["schema", "show", uri, "--json"]); + assert_ok("predecessor schema show", &output); + support::parse_stdout_json(&output)["schema_source"].clone() + }); let before = graph_files(&graph); + if source_format < 13 { + for object in ROOT_SCHEMA_OBJECTS { + assert!( + before.contains_key(Path::new(object)), + "a stamp-{source_format} root keeps its schema contract in {object}" + ); + } + } let refused = output_failure(cli().args(["snapshot", uri])); let stderr = String::from_utf8_lossy(&refused.stderr); assert!( stderr.contains("`omnigraph upgrade --check`"), - "the stamp-13 refusal must name the route: {stderr}" + "the stamp-{source_format} refusal must name the route: {stderr}" ); let check = support::parse_stdout_json(&output_success( cli().args(["upgrade", uri, "--check", "--json"]), )); assert_eq!(check["outcome"], "check_passed", "{check}"); - assert_eq!(check["observed_format"], 13); + assert_eq!(check["observed_format"], source_format); assert_eq!(check["target_format"], 14); assert_eq!(check["target_defaulted"], true); - assert_eq!(check["route"], serde_json::json!([V13_HANDLER])); + assert_eq!(check["route"], serde_json::json!([handler])); assert_eq!(check["completed_handlers"], serde_json::json!([])); assert_eq!(check["findings"], serde_json::json!([])); - assert_eq!(check["work"]["live_refs"], V13_LIVE_BRANCHES.len()); + assert_eq!(check["work"]["live_refs"], JOURNEY_LIVE_BRANCHES.len()); assert!( check["work"]["legacy_commits"].as_u64().unwrap() >= historical_rows.len() as u64, "{check}" @@ -1773,8 +2142,8 @@ fn genuine_v13_storage_upgrade_preserves_history() { "upgrade {}: check_passed (check)", check["location"].as_str().unwrap() ), - "format: 13 -> 14 (default target)", - "route: history-lance-files-v13-to-v14", + &format!("format: {source_format} -> 14 (default target)"), + &format!("route: {handler}"), ); assert_eq!( expected[4..6], @@ -1822,12 +2191,9 @@ fn genuine_v13_storage_upgrade_preserves_history() { let upgraded = support::parse_stdout_json(&output_success(cli().args(["upgrade", uri, "--json"]))); assert_eq!(upgraded["outcome"], "completed", "{upgraded}"); - assert_eq!(upgraded["observed_format"], 13); + assert_eq!(upgraded["observed_format"], source_format); assert_eq!(upgraded["target_format"], 14); - assert_eq!( - upgraded["completed_handlers"], - serde_json::json!([V13_HANDLER]) - ); + assert_eq!(upgraded["completed_handlers"], serde_json::json!([handler])); assert_eq!(upgraded["last_durable_completed_boundary"], "activated"); assert_eq!(upgraded["findings"], serde_json::json!([])); assert!(upgraded["recovery"].is_null(), "{upgraded}"); @@ -1841,6 +2207,23 @@ fn genuine_v13_storage_upgrade_preserves_history() { "{:?}", after.keys().collect::>() ); + if let Some(predecessor_schema) = predecessor_schema { + for object in ROOT_SCHEMA_OBJECTS.map(Path::new) { + assert_eq!( + after.get(object), + before.get(object), + "the upgrade must leave {} as the predecessor wrote it", + object.display() + ); + } + let served = support::parse_stdout_json(&output_success( + cli().args(["schema", "show", uri, "--json"]), + )); + assert_eq!( + served["schema_source"], predecessor_schema, + "the schema the predecessor accepted must be the one served after the upgrade" + ); + } for check_mode in [true, false] { let mut command = cli(); command.args(["upgrade", uri, "--json"]); @@ -1857,7 +2240,7 @@ fn genuine_v13_storage_upgrade_preserves_history() { "the predecessor must refuse the upgraded root" ); - for (index, branch) in V13_LIVE_BRANCHES.into_iter().enumerate() { + for (index, branch) in JOURNEY_LIVE_BRANCHES.into_iter().enumerate() { let exported = output_success(cli().args(["export", uri, "--branch", branch])); assert_eq!( canonical_export_rows(&exported.stdout), @@ -1883,7 +2266,7 @@ fn genuine_v13_storage_upgrade_preserves_history() { let check_history = || { tokio::runtime::Runtime::new().unwrap().block_on(async { let db = Omnigraph::open(uri).await.unwrap(); - for (index, branch) in V13_LIVE_BRANCHES.into_iter().enumerate() { + for (index, branch) in JOURNEY_LIVE_BRANCHES.into_iter().enumerate() { db.sync_branch(branch).await.unwrap(); for commit in histories[index].as_array().unwrap() { let written_on = commit["graph_branch"].as_str().unwrap_or("main"); @@ -1927,6 +2310,33 @@ fn genuine_v13_storage_upgrade_preserves_history() { } }; check_history(); + if let Some((rated_path, answers)) = &property_evolution { + for (id, expected) in answers { + let args = [ + "query", + "rated", + "--query", + rated_path.as_str(), + "--store", + uri, + "--snapshot", + id.as_str(), + "--params", + "{}", + "--json", + ]; + let served = property_read( + &format!("rated at {id}"), + "rating", + cli().args(args).output().unwrap(), + ); + assert_same_answer( + &format!("the property added by the predecessor's schema apply, read at {id}"), + &served, + expected, + ); + } + } output_success(cli().args([ "mutate", "revise", "--query", query_path, "--store", uri, "--branch", "review", @@ -2037,32 +2447,26 @@ fn genuine_v13_storage_upgrade_preserves_history() { ]) .output() .unwrap(); - if output.status.success() { - let mut rows = support::parse_stdout_json(&output)["rows"].clone(); - normalize_f32_and_nulls(&mut rows); - assert_eq!(rows, expected[0], "retained docs at {id} after cleanup"); - assert_eq!( - journey_rows(None, uri, query_path, "--snapshot", id, "edges"), - expected[1], - "retained edges at {id} after cleanup" - ); - retained.push(id.as_str()); - continue; + match snapshot_read(&format!("docs at {id} after cleanup"), output) { + SnapshotRead::Rows(rows) => { + assert_eq!(rows, expected[0], "retained docs at {id} after cleanup"); + assert_eq!( + journey_rows(None, uri, query_path, "--snapshot", id, "edges"), + expected[1], + "retained edges at {id} after cleanup" + ); + retained.push(id.as_str()); + } + SnapshotRead::Reclaimed(reclaimed) => assert!( + pinned_before_cleanup[id].contains(&reclaimed), + "commit {id} is refused for table version {reclaimed}, which its snapshot never \ + pinned: {:?}", + pinned_before_cleanup[id] + ), + SnapshotRead::NoSuchProperty => { + unreachable!("`snapshot_read` classifies no refusal as a missing property") + } } - let stderr = String::from_utf8_lossy(&output.stderr); - let reclaimed: u64 = stderr - .split_once("historical published dataset version ") - .and_then(|(_, rest)| rest.split_once(" was reclaimed")) - .and_then(|(version, _)| version.parse().ok()) - .unwrap_or_else(|| { - panic!("commit {id} must be served or refused as reclaimed: {stderr}") - }); - assert!( - pinned_before_cleanup[id].contains(&reclaimed), - "commit {id} is refused for table version {reclaimed}, which its snapshot never \ - pinned: {:?}", - pinned_before_cleanup[id] - ); } for (index, branch) in [(0, "main"), (1, "review")] { let head = histories[index] @@ -2087,9 +2491,209 @@ fn genuine_v13_storage_upgrade_preserves_history() { std::fs::create_dir_all(destination.parent().unwrap()).unwrap(); std::fs::write(destination, bytes).unwrap(); } - for (index, branch) in V13_LIVE_BRANCHES.into_iter().enumerate() { + for (index, branch) in JOURNEY_LIVE_BRANCHES.into_iter().enumerate() { let output = run_old(old, &["export", uri, "--branch", branch]); assert_ok("whole-root backup restore", &output); assert_eq!(canonical_export_rows(&output.stdout), exports[index]); } } + +/// Stamp 9 after the released 0.11.0 ran `optimize` and `cleanup --keep 1`: +/// retained `__manifest` versions then pin table versions that are gone, so +/// the first `cleanup` after the upgrade has to be `--keep 1`. +#[test] +fn genuine_v0_11_0_storage_upgrade_after_predecessor_cleanup() { + const LIVE_BRANCHES: [&str; 2] = ["main", "review"]; + let Some(old) = v011_bin() else { + return; + }; + let old = old.as_path(); + let temp = tempdir().unwrap(); + let graph = temp.path().join("cleaned.omni"); + let uri = graph.to_str().unwrap(); + let [schema, data, queries] = journey_fixture(temp.path()); + let query_path = queries.to_str().unwrap(); + + assert_ok( + "v0.11.0 init", + &run_old(old, &["init", "--schema", schema.to_str().unwrap(), uri]), + ); + assert_writes_format(old, uri, 9); + for _ in 0..4 { + assert_ok( + "v0.11.0 load", + &run_old( + old, + &[ + "load", + "--mode", + "overwrite", + "--data", + data.to_str().unwrap(), + uri, + ], + ), + ); + } + assert_ok( + "v0.11.0 branch create", + &run_old(old, &["branch", "create", "review", "--uri", uri]), + ); + assert_ok( + "v0.11.0 mutate", + &run_old( + old, + &[ + "mutate", + "retitle", + "--query", + query_path, + "--store", + uri, + "--branch", + "review", + "--params", + r#"{"title":"organism review"}"#, + ], + ), + ); + let captured: Vec<(Vec, serde_json::Value)> = LIVE_BRANCHES + .into_iter() + .map(|branch| { + let export = run_old(old, &["export", uri, "--branch", branch]); + assert_ok("v0.11.0 export", &export); + let listing = run_old(old, &["commit", "list", uri, "--branch", branch, "--json"]); + assert_ok("v0.11.0 commit list", &listing); + ( + canonical_export_rows(&export.stdout), + support::parse_stdout_json(&listing)["commits"].clone(), + ) + }) + .collect(); + + assert_ok("v0.11.0 optimize", &run_old(old, &["optimize", uri])); + assert_ok( + "v0.11.0 cleanup", + &run_old(old, &["cleanup", uri, "--keep", "1", "--confirm"]), + ); + let docs_at = |id: &str| { + [ + "query", + "docs", + "--query", + query_path, + "--store", + uri, + "--snapshot", + id, + "--json", + ] + .map(str::to_owned) + }; + let mut predecessor_reads = std::collections::BTreeMap::new(); + for commit in captured + .iter() + .flat_map(|(_, history)| history.as_array().unwrap()) + { + let id = commit["graph_commit_id"].as_str().unwrap(); + let args = docs_at(id); + let args: Vec<&str> = args.iter().map(String::as_str).collect(); + predecessor_reads.insert( + id.to_owned(), + snapshot_read(&format!("0.11.0 docs at {id}"), run_old(old, &args)), + ); + } + assert!( + predecessor_reads + .values() + .any(|read| matches!(read, SnapshotRead::Rows(_))), + "0.11.0 must still serve at least one commit after its cleanup: {predecessor_reads:?}" + ); + + let check = support::parse_stdout_json(&output_success( + cli().args(["upgrade", uri, "--check", "--json"]), + )); + assert_eq!(check["outcome"], "check_passed", "{check}"); + assert_eq!(check["route"], serde_json::json!([V9_HANDLER])); + let upgraded = + support::parse_stdout_json(&output_success(cli().args(["upgrade", uri, "--json"]))); + assert_eq!(upgraded["outcome"], "completed", "{upgraded}"); + assert_eq!(upgraded["observed_format"], 9); + assert_eq!( + upgraded["completed_handlers"], + serde_json::json!([V9_HANDLER]) + ); + + let head_exports = || { + LIVE_BRANCHES.map(|branch| { + let exported = output_success(cli().args(["export", uri, "--branch", branch])); + canonical_export_rows(&exported.stdout) + }) + }; + for ((branch, (export, history)), served) in + LIVE_BRANCHES.into_iter().zip(&captured).zip(head_exports()) + { + assert_eq!(&served, export, "{branch}"); + let listing = support::parse_stdout_json(&output_success( + cli().args(["commit", "list", uri, "--branch", branch, "--json"]), + )); + for commit in history.as_array().unwrap() { + assert_lists_commit(&listing, commit); + } + } + for (id, expected) in &predecessor_reads { + let served = snapshot_read( + &format!("docs at {id}"), + cli().args(docs_at(id)).output().unwrap(), + ); + assert_same_answer( + &format!("commit {id} after the upgrade, as 0.11.0 read it after its cleanup"), + &served, + expected, + ); + } + + output_success(cli().args([ + "mutate", "revise", "--query", query_path, "--store", uri, "--branch", "review", + ])); + output_success(cli().args([ + "branch", "merge", "review", "--into", "main", "--uri", uri, "--json", + ])); + let heads = head_exports(); + let objects = graph_files(&graph); + let refused = cli() + .args(["cleanup", uri, "--older-than", "7d", "--confirm", "--json"]) + .output() + .unwrap(); + let reported = format!( + "{}{}", + String::from_utf8_lossy(&refused.stdout), + String::from_utf8_lossy(&refused.stderr) + ); + assert!( + reported.contains("is absent from the listing"), + "`cleanup --older-than` must refuse while pre-upgrade `__manifest` versions are \ + retained: {reported}" + ); + let left = graph_files(&graph); + for object in objects.keys() { + assert!( + left.contains_key(object), + "a refused cleanup must delete nothing, {} is gone", + object.display() + ); + } + assert_eq!(head_exports(), heads); + for policy in [["--keep", "1"], ["--older-than", "7d"]] { + let report = support::parse_stdout_json(&output_success( + cli() + .args(["cleanup", uri]) + .args(policy) + .args(["--confirm", "--json"]), + )); + for dataset in report["datasets"].as_array().unwrap() { + assert!(dataset["error"].is_null(), "{policy:?}: {report}"); + } + assert_eq!(head_exports(), heads, "{policy:?}"); + } +} diff --git a/crates/omnigraph/src/db/catalog_tests.rs b/crates/omnigraph/src/db/catalog_tests.rs index a47b5cca2..5de059d2f 100644 --- a/crates/omnigraph/src/db/catalog_tests.rs +++ b/crates/omnigraph/src/db/catalog_tests.rs @@ -291,7 +291,12 @@ async fn open_refuses_a_stamp_below_the_served_floor_before_any_effect() { "{error}" ); assert!(error.to_string().contains("omnigraph export"), "{error}"); - assert!(!error.to_string().contains("omnigraph upgrade"), "{error}"); + assert!( + error + .to_string() + .contains("omnigraph upgrade --check"), + "{error}" + ); assert_eq!( reached_effects.load(std::sync::atomic::Ordering::SeqCst), 0, diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index abc8c8e9c..ecd54dd54 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -3279,7 +3279,7 @@ fn blob_properties_for_table_key<'a>( /// fields. Blob reads admit a missing marker only at the exact current physical /// table entry and refuse every older snapshot rather than inferring identity /// from Lance field IDs or positions, even when no rename occurred. -fn fixup_physical_schemas(catalog: &mut Catalog) -> Result<()> { +pub(crate) fn fixup_physical_schemas(catalog: &mut Catalog) -> Result<()> { let system_columns = catalog.system_columns; // Canonically ordered walk (DST determinism: hash order must not leak // into schema fixups). @@ -3627,7 +3627,7 @@ async fn finish_init_coordinator( Ok(coordinator) } -fn schema_table_key(type_kind: SchemaTypeKind, name: &str) -> String { +pub(crate) fn schema_table_key(type_kind: SchemaTypeKind, name: &str) -> String { match type_kind { SchemaTypeKind::Node => format!("node:{}", name), SchemaTypeKind::Edge => format!("edge:{}", name), @@ -3635,7 +3635,7 @@ fn schema_table_key(type_kind: SchemaTypeKind, name: &str) -> String { } } -fn schema_for_table_key(catalog: &Catalog, table_key: &str) -> Result> { +pub(crate) fn schema_for_table_key(catalog: &Catalog, table_key: &str) -> Result> { if let Some(type_name) = table_key.strip_prefix("node:") { let node_type: &NodeType = catalog .node_types diff --git a/crates/omnigraph/src/db/upgrade.rs b/crates/omnigraph/src/db/upgrade.rs index 17356bf15..89e26d60b 100644 --- a/crates/omnigraph/src/db/upgrade.rs +++ b/crates/omnigraph/src/db/upgrade.rs @@ -1,9 +1,12 @@ -//! `omnigraph upgrade`: the offline conversion of a standalone root from -//! storage format 13 to 14. The upgrade reads every ref of `__manifest` at a -//! pinned version, fences main with its intent, writes the graph commits of -//! the source as immutable objects under `__history/legacy/`, converts every -//! live ref to its head alone, validates what it wrote and activates main. A -//! run that stops after the fence is resumed by running the command again. +//! `omnigraph upgrade`: the offline conversion of a standalone root to storage +//! format 14 from each [`UpgradeSource`], by two routes: from 13, whose schema +//! contract is a row of `__manifest`, and from 8 or 9, whose contract is read +//! from the graph root (`root_schema`) until the fence binds it. The upgrade +//! reads every ref of `__manifest` at a pinned version, fences main with its +//! intent, writes the graph commits of the source as immutable objects under +//! `__history/legacy/`, converts every live ref to its head alone, validates +//! what it wrote and activates main. A run that stops after the fence is +//! resumed by running the command again. use std::collections::{BTreeSet, HashMap, HashSet}; use std::sync::Arc; @@ -19,27 +22,56 @@ use lance_file::version::LanceFileVersion; use serde::Serialize; use sha2::{Digest, Sha256}; +use omnigraph_compiler::{SchemaIR, SchemaTypeKind, build_catalog_from_ir}; + +use crate::branch_control::retire_branch_recoverably; +use crate::branch_names::{ensure_logical_branch_name, logical_branch_name}; use crate::db::legacy_sidecars::pending_legacy_sidecars; -use crate::db::manifest::TableIdentity; use crate::db::manifest::history::{self, HistoryRecord, LegacyDirectory, LegacyLayout}; use crate::db::manifest::layout::open_manifest_dataset_native_with_session; use crate::db::manifest::legacy::{ - self, CensusError, CensusInput, CensusRef, HeadScan, LegacyCensus, LegacyManifestSource, - LegacyPlan, MAX_CENSUS_CELLS, MAX_CENSUS_SNAPSHOT_BYTES, SourceRole, Stamp13Source, + self, CensusError, CensusHead, CensusInput, CensusRef, HeadScan, LegacyCensus, + LegacyManifestSource, LegacyPlan, MAX_CENSUS_CELLS, MAX_CENSUS_SNAPSHOT_BYTES, + RootContractSource, SourceRole, Stamp13Source, }; use crate::db::manifest::migrations::{ INTERNAL_MANIFEST_SCHEMA_VERSION, INTERNAL_SCHEMA_VERSION_KEY, MAX_BRANCHES, MAX_INTENT_BYTES, - SourceBranch, UPGRADE_PENDING_KEY, UPGRADE_PROTOCOL, UPGRADE_RECEIPT_KEY, - UPGRADE_SOURCE_FORMAT, UpgradeIntent, UpgradeSchemaContract, branch_completed, fence_operation, - guard_stamp, intent_from, read_stamp, receipt, recovery_guidance, + SourceBranch, SourceLock, UPGRADE_PENDING_KEY, UPGRADE_PROTOCOL, UPGRADE_RECEIPT_KEY, + UPGRADE_SOURCE_FORMATS, UpgradeIntent, UpgradeSchemaContract, UpgradeSource, branch_completed, + fence_operation, guard_stamp, intent_from, read_stamp, receipt, recovery_guidance, }; use crate::db::manifest::retention::retired_manifest_branches; -use crate::db::manifest::state::{TablePin, TableState, read_converted_state}; +use crate::db::manifest::state::{SchemaContractRow, TablePin, TableState, read_converted_state}; +use crate::db::manifest::{DatasetEntry, TableIdentity, TableRegistration, open_dataset_entry}; +use crate::db::omnigraph::{fixup_physical_schemas, schema_for_table_key, schema_table_key}; use crate::error::{OmniError, Result}; use crate::seams::{decide_seam, fail}; -use crate::storage::{normalize_root_uri, storage_for_uri}; +use crate::storage::{StorageAdapter, normalize_root_uri, storage_for_uri}; + +mod root_schema; + +/// What a report names when the run stopped before the stamp of the source +/// told the route: the pending intent is unreadable, or main carries a stamp +/// no route converts. +const UNROUTED_HANDLER: &str = "history-lance-files-to-v14"; +/// The logical name of the ref a 0.11.x schema apply holds while it runs. +const SCHEMA_APPLY_LOCK_BRANCH: &str = "__schema_apply_lock__"; +/// The release whose roots are stamped 8 or 9, as a refusal names it. +const ROOT_CONTRACT_RELEASE: &str = "omnigraph 0.11.x"; + +/// The name of the route from `source`, as a report carries it. +fn handler(source: UpgradeSource) -> String { + format!( + "history-lance-files-v{}-to-v{INTERNAL_MANIFEST_SCHEMA_VERSION}", + source.stamp() + ) +} + +/// [`handler`] of a run that may have stopped before any route was chosen. +fn handler_or_unrouted(routed: Option) -> String { + routed.map_or_else(|| UNROUTED_HANDLER.to_string(), handler) +} -const HANDLER: &str = "history-lance-files-v13-to-v14"; /// The rows one head of `__manifest` may hold. const MAX_ROWS: usize = 1_000_000; /// The decoded bytes one head of `__manifest` may hold. @@ -60,6 +92,9 @@ struct Bounds { head_rows: usize, head_bytes: usize, retired_refs: usize, + /// The bytes one root schema object may hold: the three become one row of + /// a head, so each is bounded as a head is. + root_object_bytes: u64, } impl Bounds { @@ -69,6 +104,7 @@ impl Bounds { head_rows: MAX_ROWS, head_bytes: MAX_METADATA_BYTES, retired_refs: MAX_BRANCHES, + root_object_bytes: MAX_METADATA_BYTES as u64, }; } @@ -129,6 +165,9 @@ pub struct UpgradeWork { pub census_reads: u64, pub census_cells: u64, pub legacy_bytes: u64, + /// The tables of main's head opened to compare their columns with the + /// contract at the graph root; zero on the route whose contract is a row. + pub table_opens: u64, } impl UpgradeWork { @@ -148,6 +187,7 @@ impl UpgradeWork { census_reads: counts.census_reads, census_cells: counts.census_cells, legacy_bytes: counts.legacy_bytes, + table_opens: 0, } } @@ -275,16 +315,45 @@ impl UpgradeReport { }); } + /// Ask for `remedy` under the route this run chose, or under + /// [`UNROUTED_HANDLER`] when it stopped before choosing one. fn recover(&mut self, code: &str, message: impl Into, remedy: Remedy) { + let handler = self + .route + .last() + .map_or(UNROUTED_HANDLER, String::as_str) + .to_string(); + self.recover_as(&handler, code, message, remedy); + } + + fn recover_as( + &mut self, + handler: &str, + code: &str, + message: impl Into, + remedy: Remedy, + ) { self.outcome = UpgradeOutcome::RecoveryRequired; self.finding(code, message); self.recovery = Some(UpgradeRecovery { - failed_handler: HANDLER.into(), + failed_handler: handler.into(), executable_compatibility: remedy.executable().into(), action: remedy.action(self.target_format), }); } + /// The source graph holds unfinished work of the build that wrote it: + /// `build` resolves it by `action` before any conversion is attempted. + fn source_recovery(&mut self, handler: String, message: String, build: &str, action: &str) { + self.outcome = UpgradeOutcome::RecoveryRequired; + self.finding("source_recovery_required", message); + self.recovery = Some(UpgradeRecovery { + failed_handler: handler, + executable_compatibility: build.into(), + action: action.into(), + }); + } + fn census_finding(&mut self, error: CensusError, fenced: bool) -> Result<()> { match error { CensusError::Finding(finding) if fenced => { @@ -401,38 +470,44 @@ async fn run( return Ok(()); } }; + let routed = pending + .as_ref() + .map(|intent| intent.source_format) + .or(report.observed_format) + .and_then(UpgradeSource::from_stamp); if pending.is_some() && options.check { - report.recover("pending_upgrade", recovery_guidance(&main), Remedy::Rerun); + report.recover_as( + &handler_or_unrouted(routed), + "pending_upgrade", + recovery_guidance(&main), + Remedy::Rerun, + ); return Ok(()); } let storage = crate::storage::decorate(storage_for_uri(root)?); let sidecars = pending_legacy_sidecars(root, storage.as_ref()).await?; if !sidecars.is_empty() { - report.outcome = UpgradeOutcome::RecoveryRequired; - report.finding( - "source_recovery_required", + report.source_recovery( + handler_or_unrouted(routed), format!( "{} recovery sidecar(s) under `__recovery/` ({}) must be resolved before the \ storage conversion", sidecars.len(), sidecars.join(", ") ), + "the omnigraph build that wrote the recovery sidecars", + "stop all writers, retain the backup, open the graph read-write with the build that \ + wrote the sidecars so it finishes its recovery, then rerun `omnigraph upgrade \ + --check`", ); - report.recovery = Some(UpgradeRecovery { - failed_handler: HANDLER.into(), - executable_compatibility: "the omnigraph build that wrote the recovery sidecars".into(), - action: "stop all writers, retain the backup, open the graph read-write with the \ - build that wrote the sidecars so it finishes its recovery, then rerun \ - `omnigraph upgrade --check`" - .into(), - }); return Ok(()); } if report.target_format != INTERNAL_MANIFEST_SCHEMA_VERSION { + let [oldest, older, newest] = UPGRADE_SOURCE_FORMATS; report.finding( "unsupported_target", format!( - "this binary converts storage format v{UPGRADE_SOURCE_FORMAT} to \ + "this binary converts storage formats v{oldest}, v{older} and v{newest} to \ v{INTERNAL_MANIFEST_SCHEMA_VERSION} and serves \ v{INTERNAL_MANIFEST_SCHEMA_VERSION} alone; `--to-format` accepts \ {INTERNAL_MANIFEST_SCHEMA_VERSION} only" @@ -440,30 +515,22 @@ async fn run( ); return Ok(()); } - if pending.is_none() { + let Some(source) = routed else { match (guard_stamp(&main), report.observed_format) { - (Ok(_), _) => { - report.outcome = UpgradeOutcome::AlreadyCurrent; - return Ok(()); - } + (Ok(_), _) => report.outcome = UpgradeOutcome::AlreadyCurrent, (Err(error), Some(stamp)) if stamp > INTERNAL_MANIFEST_SCHEMA_VERSION => { report.finding("newer_than_binary", error.to_string()); - return Ok(()); - } - (Err(_), Some(UPGRADE_SOURCE_FORMAT)) => {} - (Err(error), _) => { - report.finding("unsupported_source", error.to_string()); - return Ok(()); } + (Err(error), _) => report.finding("unsupported_source", error.to_string()), } - } - report.route.push(HANDLER.into()); + return Ok(()); + }; + report.route.push(handler(source)); let fenced = pending.is_some(); if fenced { report.last_durable_completed_boundary = Some("source_fenced".into()); } - let source = Stamp13Source; let pinned_main = match &pending { Some(intent) => { let version = intent.branches.last().map_or(0, |main| main.version); @@ -473,42 +540,60 @@ async fn run( } None => main.clone(), }; - let contract = match source.admits(&pinned_main, SourceRole::LiveHead) { - Ok(_) => source.version_schema(&pinned_main, None).await?.contract, - Err(error) if fenced => return Err(error), - Err(error) => { - report.finding("unsupported_source", error.to_string()); - return Ok(()); + let (branches, retire, attempt) = match &pending { + Some(intent) => ( + intent.branches.clone(), + intent.retire.clone(), + intent.attempt.clone(), + ), + None => { + let (branches, retire) = partition_locks(inventory(&main).await?, source); + (branches, retire, ulid::Ulid::new().to_string()) } }; - let Some(contract) = contract else { - let missing = format!( - "version {} of main holds no schema contract row", - pinned_main.version().version - ); - if fenced { - return Err(invalid(missing)); - } - report.finding("unsupported_source", missing); + authorize(&branches, actor, policy)?; + let chosen = route( + root, + storage.as_ref(), + source, + &pinned_main, + pending.as_ref(), + bounds, + report, + ) + .await?; + let Some(Route { + source: reader, + contract, + root_ir, + }) = chosen + else { return Ok(()); }; + let reader = reader.as_ref(); report.graph_identity = Some(contract.head.schema_identity_domain.clone()); - let (branches, attempt) = match &pending { - Some(intent) => { - if intent.graph_identity != contract.head.schema_identity_domain { - return Err(invalid("upgrade schema identity changed")); - } - intent - .schema_contract - .as_ref() - .ok_or_else(|| invalid("storage upgrade intent names no schema contract"))? - .validate_row(&contract)?; - (intent.branches.clone(), intent.attempt.clone()) + if let Some(intent) = &pending { + if intent.graph_identity != contract.head.schema_identity_domain { + return Err(invalid("upgrade schema identity changed")); } - None => (inventory(&main).await?, ulid::Ulid::new().to_string()), - }; - authorize(&branches, actor, policy)?; - verify_inventory(root, &branches, pending.as_ref()).await?; + intent + .schema_contract + .as_ref() + .ok_or_else(|| invalid("storage upgrade intent names no schema contract"))? + .validate_row(&contract)?; + } + for lock in &retire { + report.finding( + "schema_apply_lock_retired", + format!( + "live branch `{}` is the schema-apply lock of {ROOT_CONTRACT_RELEASE}, which a \ + schema apply did not release; the upgrade retires it once main is fenced, as the \ + completed apply would have, and keeps its head as a retired ref", + lock.native + ), + ); + } + verify_inventory(root, &branches, &retire, pending.as_ref(), source).await?; let directory = match &pending { Some(intent) => { @@ -540,9 +625,10 @@ async fn run( report.finding( "history_objects_present", format!( - "`__history/` already holds objects no v{UPGRADE_SOURCE_FORMAT} \ - build writes ({}); restore the whole root, `__history/` included, \ - from the backup taken before the earlier attempt", + "`__history/` already holds objects no v{} build writes ({}); \ + restore the whole root, `__history/` included, from the backup \ + taken before the earlier attempt", + source.stamp(), foreign.join(", ") ), ); @@ -554,30 +640,31 @@ async fn run( return Ok(()); } } - let input = match bounded_input(&main, &branches, &attempt, bounds, fenced).await? { - Ok(input) => input, - Err(over) if fenced => { - report.recover( - "unsupported_source", - over.join("; "), - Remedy::FencingExecutable, - ); - return Ok(()); - } - Err(over) => { - for message in over { - report.finding("unsupported_source", message); + let input = + match bounded_input(&main, &branches, &retire, &attempt, bounds, fenced).await? { + Ok(input) => input, + Err(over) if fenced => { + report.recover( + "unsupported_source", + over.join("; "), + Remedy::FencingExecutable, + ); + return Ok(()); } - return Ok(()); - } - }; + Err(over) => { + for message in over { + report.finding("unsupported_source", message); + } + return Ok(()); + } + }; let layout = pending .as_ref() .map_or(LegacyLayout::CURRENT, |intent| intent.legacy.layout); let census = match legacy::census_within( root, &input, - &source, + reader, bounds.census_cells, bounds.census_snapshot_bytes, ) @@ -586,11 +673,28 @@ async fn run( Ok(census) => census, Err(error) => return report.census_finding(error, fenced), }; + let mut table_opens = 0; + if !fenced && let Some(ir) = &root_ir { + if let Some(differs) = contract_differs_from_tables(ir, &census.heads)? { + report.finding("unsupported_source", differs); + return Ok(()); + } + let (opened, differs) = + columns_differ_from_contract(root, &session, ir, &census.heads).await?; + if let Some(differs) = differs { + report.finding("unsupported_source", differs); + return Ok(()); + } + table_opens = opened; + } let plan = match legacy::plan(&census, &layout) { Ok(plan) => plan, Err(error) => return report.census_finding(error, fenced), }; - report.work = UpgradeWork::of_census(&census, &plan); + report.work = UpgradeWork { + table_opens, + ..UpgradeWork::of_census(&census, &plan) + }; Some((census, plan)) } }; @@ -625,11 +729,12 @@ async fn run( let intent = UpgradeIntent { protocol: UPGRADE_PROTOCOL, attempt, - source_format: UPGRADE_SOURCE_FORMAT, + source_format: source.stamp(), target_format: INTERNAL_MANIFEST_SCHEMA_VERSION, graph_identity: contract.head.schema_identity_domain.clone(), branches, - schema_contract: Some(UpgradeSchemaContract::from_row(&contract)), + retire, + schema_contract: Some(UpgradeSchemaContract::from_row(&contract)?), legacy: plan.clone(), }; let json = intent_json(&intent)?; @@ -637,16 +742,18 @@ async fn run( report.outcome = UpgradeOutcome::CheckPassed; return Ok(()); } - verify_inventory(root, &intent.branches, None).await?; + verify_inventory(root, &intent.branches, &intent.retire, None, source).await?; history::archive_schema(root, &session, &contract).await?; let pinned = intent.branches.last(); - if !fence_main(root, pinned, json, intent.target_format, report).await? { + let target = intent.target_format; + if !fence_main(root, pinned, json, source, target, report).await? { return Ok(()); } intent } }; fail(&UPGRADE_AFTER_FENCE)?; + retire_locks(root, &intent.retire).await?; if let Some((census, _)) = &census { match legacy::materialize(root, &session, census, &intent.legacy).await { @@ -677,9 +784,9 @@ async fn run( if branch_completed(¤t, branch, &intent)? { continue; } - verify_source_head(¤t, branch, native.is_none())?; + verify_source_head(¤t, branch, native.is_none(), source)?; let head = converted_head(root, &session, branch, census.as_ref()).await?; - publish_conversion(root, &session, current, branch, &intent, &head).await?; + publish_conversion(root, &session, reader, current, branch, &intent, &head).await?; report.last_durable_completed_boundary = Some(format!("converted:{}", native.unwrap_or("main"))); fail(&UPGRADE_AFTER_BRANCH)?; @@ -698,24 +805,349 @@ async fn run( named(native) ))); } - let source = current + let pinned = current .checkout_version(branch.version) .await .map_err(OmniError::storage)?; let head = converted_head(root, &session, branch, None).await?; - equivalent_13_14(&source, ¤t, native, &head).await?; + equivalent_to_source(reader, &pinned, ¤t, native, &head).await?; validate_converted_contract(¤t, native, &head, &intent).await?; } - verify_inventory(root, &intent.branches, Some(&intent)).await?; + verify_inventory( + root, + &intent.branches, + &intent.retire, + Some(&intent), + source, + ) + .await?; fail(&UPGRADE_BEFORE_ACTIVATION)?; publish_activation(open(root, None).await?).await?; report.last_durable_completed_boundary = Some("activated".into()); fail(&UPGRADE_AFTER_ACTIVATION)?; report.outcome = UpgradeOutcome::Completed; - report.completed_handlers.push(HANDLER.into()); + report.completed_handlers.push(handler(source)); + Ok(()) +} + +/// The live refs of `source` split into the ones the upgrade converts and the +/// schema-apply locks of 0.11.x it retires; a source whose contract is a row +/// has no such lock. +fn partition_locks( + branches: Vec, + source: UpgradeSource, +) -> (Vec, Vec) { + let is_lock = |branch: &SourceBranch| { + !source.contract_in_row() + && branch + .native + .as_deref() + .is_some_and(|native| logical_branch_name(native) == SCHEMA_APPLY_LOCK_BRANCH) + }; + let (locks, branches): (Vec<_>, Vec<_>) = branches.into_iter().partition(is_lock); + let retire = locks + .into_iter() + .filter_map(|lock| { + lock.native.map(|native| SourceLock { + native, + identity: lock.identity, + }) + }) + .collect(); + (branches, retire) +} + +/// Retire every ref of `retire` as a 0.11.x branch delete does; one already +/// retired under its pinned identity is left as it is. +async fn retire_locks(root: &str, retire: &[SourceLock]) -> Result<()> { + if retire.is_empty() { + return Ok(()); + } + let main = open(root, None).await?; + for lock in retire { + retire_branch_recoverably(&main, &lock.native, &lock.identity).await?; + } Ok(()) } +/// The reader of one source format and the schema contract every live ref is +/// converted under. `root_ir` is the IR of a contract read from the root +/// objects, `None` on the route whose contract is a row. +struct Route { + source: Box, + contract: SchemaContractRow, + root_ir: Option, +} + +/// The route from `source` over `main`, the version of main the upgrade pins. +/// `None` when the report holds why an unfenced graph has none; a fenced graph +/// (`pending`) was admitted before its fence, so what refuses it is an error. +async fn route( + root: &str, + storage: &dyn StorageAdapter, + source: UpgradeSource, + main: &Dataset, + pending: Option<&UpgradeIntent>, + bounds: Bounds, + report: &mut UpgradeReport, +) -> Result> { + let fenced = pending.is_some(); + let refuse = |report: &mut UpgradeReport, error: OmniError| { + if fenced { + Err(error) + } else { + report.finding("unsupported_source", error.to_string()); + Ok(None) + } + }; + if source.contract_in_row() { + let reader = Stamp13Source; + if let Err(error) = reader.admits(main, SourceRole::LiveHead) { + return refuse(report, error); + } + let Some(contract) = reader.version_schema(main, None).await?.contract else { + return refuse( + report, + invalid(format!( + "version {} of main holds no schema contract row", + main.version().version + )), + ); + }; + return Ok(Some(Route { + source: Box::new(reader), + contract, + root_ir: None, + })); + } + let (contract, root_ir) = match pending { + Some(intent) => { + let bound = intent + .schema_contract + .as_ref() + .ok_or_else(|| invalid("storage upgrade intent names no schema contract"))?; + let session = crate::lance_access::control_session(); + match history::read_schema(root, &session, &bound.content_sha256, &bound.identity).await + { + Ok(contract) => (contract, None), + Err(error) if root_schema::store_did_not_answer(&error) => return Err(error), + Err(error) => { + report.recover( + "legacy_objects_differ", + format!( + "the schema content `{}` the upgrade intent binds cannot be read \ + from `__history/schemas/`: {error}; restore the whole root from the \ + backup taken before the attempt", + bound.content_sha256 + ), + Remedy::FencingExecutable, + ); + return Ok(None); + } + } + } + None => { + if let Some(staging) = root_schema::staging_object(root, storage).await? { + report.source_recovery( + handler(source), + format!( + "schema object `{staging}` at the graph root is the unfinished part of \ + a schema apply by {ROOT_CONTRACT_RELEASE}; it must be resolved before \ + the storage conversion" + ), + ROOT_CONTRACT_RELEASE, + "stop all writers, retain the backup, open the graph read-write with \ + omnigraph 0.11.x so it finishes or rolls back the schema apply, then rerun \ + `omnigraph upgrade --check`", + ); + return Ok(None); + } + let loaded = match root_schema::load_validated_schema_contract( + root, + storage, + bounds.root_object_bytes, + ) + .await + { + Ok(loaded) => loaded, + Err(error) if root_schema::store_did_not_answer(&error) => return Err(error), + Err(error) => { + let [pg, ir, state] = root_schema::SCHEMA_FILENAMES; + report.finding( + "unsupported_source", + format!( + "the schema contract of this v{} graph cannot be read from `{pg}`, \ + `{ir}` and `{state}` at the graph root: {error}; restore the whole \ + root from one backup ({ROOT_CONTRACT_RELEASE} refuses to open this \ + state as well)", + source.stamp() + ), + ); + return Ok(None); + } + }; + if let Err(error) = root_schema::stamp_covers_vintage(source.stamp(), &loaded.ir) { + return refuse(report, error); + } + (loaded.row, Some(loaded.ir)) + } + }; + let reader = RootContractSource::new(source.stamp(), contract.clone())?; + if let Err(error) = reader.admits(main, SourceRole::LiveHead) { + return refuse(report, error); + } + Ok(Some(Route { + source: Box::new(reader), + contract, + root_ir, + })) +} + +/// The table key and identity of every node and edge type `ir` declares. +fn described_tables(ir: &SchemaIR) -> Result> { + let nodes = ir.nodes.iter().map(|node| { + ( + SchemaTypeKind::Node, + &node.name, + node.type_id, + node.table_incarnation_id, + ) + }); + let edges = ir.edges.iter().map(|edge| { + ( + SchemaTypeKind::Edge, + &edge.name, + edge.type_id, + edge.table_incarnation_id, + ) + }); + let mut described = BTreeSet::new(); + for (kind, name, type_id, incarnation) in nodes.chain(edges) { + let identity = TableIdentity::new(type_id.get(), incarnation.get()) + .map_err(|error| invalid(error.to_string()))?; + described.insert((schema_table_key(kind, name), identity)); + } + Ok(described) +} + +/// How the first table of main's head differs in its columns from the ones +/// the root contract `ir` declares, with the count of tables opened; `None` +/// when every pinned table stores exactly the declared columns. +async fn columns_differ_from_contract( + root: &str, + session: &Arc, + ir: &SchemaIR, + heads: &[CensusHead], +) -> Result<(u64, Option)> { + let main = heads + .iter() + .find(|head| head.native.is_none()) + .ok_or_else(|| OmniError::manifest_internal("the census derived no head for main"))?; + let mut catalog = build_catalog_from_ir(ir).map_err(|error| invalid(error.to_string()))?; + fixup_physical_schemas(&mut catalog)?; + let mut opened = 0; + for table in &main.record.tables { + let TableState::Pinned(pin) = &table.state else { + continue; + }; + let registration = &table.registration; + let declared = schema_for_table_key(&catalog, ®istration.table_key)?; + let entry = DatasetEntry { + identity: registration.identity, + type_key: registration.table_key.clone(), + dataset_path: registration.table_path.clone(), + published_dataset_version: pin.table_version, + native_dataset_branch: pin.table_branch.clone(), + entity_count: pin.row_count, + version_metadata: pin.metadata.clone(), + manifest_version: pin.manifest_version, + }; + let dataset = open_dataset_entry(&entry, root, Some(session)).await?; + opened += 1; + let stored = arrow_schema::Schema::from(dataset.schema()); + if let Some(differs) = column_difference(®istration.table_key, &declared, &stored) { + return Ok((opened, Some(differs))); + } + } + Ok((opened, None)) +} + +/// The first column of main's `table_key` whose name or type differs between +/// the root contract (`declared`) and the stored table (`stored`). +fn column_difference( + table_key: &str, + declared: &arrow_schema::Schema, + stored: &arrow_schema::Schema, +) -> Option { + const REMEDY: &str = "restore the three root schema objects from the backup taken with the \ + tables, or rebuild the graph by export and load with omnigraph 0.11.x"; + for field in declared.fields() { + match stored.field_with_name(field.name()) { + Err(_) => { + return Some(format!( + "main's table {table_key} has no column `{}`, which the schema contract at \ + the graph root declares as {}; {REMEDY}", + field.name(), + field.data_type() + )); + } + Ok(found) if found.data_type() != field.data_type() => { + return Some(format!( + "main's table {table_key} stores column `{}` as {}, the schema contract at \ + the graph root declares {}; {REMEDY}", + field.name(), + found.data_type(), + field.data_type() + )); + } + Ok(_) => {} + } + } + stored + .fields() + .iter() + .find(|field| declared.field_with_name(field.name()).is_err()) + .map(|field| { + format!( + "main's table {table_key} has column `{}`, which the schema contract at the \ + graph root does not declare; {REMEDY}", + field.name() + ) + }) +} + +/// How the first live head that differs registers other tables than the root +/// contract `ir` describes; `None` when every head registers exactly those, +/// as 0.11.x requires of the refs it serves under the one contract. +fn contract_differs_from_tables(ir: &SchemaIR, heads: &[CensusHead]) -> Result> { + let described = described_tables(ir)?; + let registered = + |registration: &TableRegistration| (registration.table_key.clone(), registration.identity); + for head in heads { + let native = named(head.native.as_deref()); + let registers: BTreeSet<(String, TableIdentity)> = head + .record + .tables + .iter() + .filter(|table| !matches!(table.state, TableState::Dropped { .. })) + .map(|table| registered(&table.registration)) + .collect(); + if let Some((key, _)) = registers.difference(&described).next() { + return Ok(Some(format!( + "{native} registers tables the schema contract at the graph root does not \ + describe: {key}" + ))); + } + if let Some((key, _)) = described.difference(®isters).next() { + return Ok(Some(format!( + "{native} does not register table {key}, which the schema contract at the \ + graph root describes" + ))); + } + } + Ok(None) +} + fn require_bound_directory(directory: &LegacyDirectory, intent: &UpgradeIntent) -> Result<()> { let found = format!("{:x}", Sha256::digest(directory.encode()?)); if found != intent.legacy.directory_sha256 { @@ -804,10 +1236,7 @@ fn authorize( OmniError::Policy("storage upgrade requires an actor when policy is installed".into()) })?; for branch in branches { - let name = branch - .native - .as_deref() - .map_or("main", crate::branch_names::logical_branch_name); + let name = branch.native.as_deref().map_or("main", logical_branch_name); checker .check( omnigraph_policy::PolicyAction::SchemaApply, @@ -858,14 +1287,28 @@ async fn inventory(main: &Dataset) -> Result> { } /// The refs the census reads: the pinned live refs with the ref each forked -/// from, and every retired ref at its head. +/// from, and every retired ref at its head, the locks of `retire` among them +/// whether the fence retired them yet or not. async fn census_input( main: &Dataset, branches: &[SourceBranch], + retire: &[SourceLock], attempt: &str, - retired_refs: HashMap, + mut retired_refs: HashMap, ) -> Result { let contents = crate::branch_control::list_live_manifest_branch_contents(main).await?; + for lock in retire { + if retired_refs.contains_key(&lock.native) { + continue; + } + let live = contents.get(&lock.native).ok_or_else(|| { + invalid(format!( + "ref '{}' is neither live nor retired as the upgrade pinned it", + lock.native + )) + })?; + retired_refs.insert(lock.native.clone(), live.clone()); + } let mut live = Vec::with_capacity(branches.len()); for branch in branches { let parent = match &branch.native { @@ -913,9 +1356,16 @@ async fn census_input( }) } -fn verify_source_head(dataset: &Dataset, source: &SourceBranch, fenced: bool) -> Result<()> { - let native = source.native.as_deref(); - let expected = source +/// `dataset` is the head the upgrade pinned for `branch`: stamped as `source` +/// is, or one version above it and stamped 14 when it is the fenced main. +fn verify_source_head( + dataset: &Dataset, + branch: &SourceBranch, + fenced: bool, + source: UpgradeSource, +) -> Result<()> { + let native = branch.native.as_deref(); + let expected = branch .version .checked_add(u64::from(fenced)) .ok_or_else(|| invalid("upgrade version overflow"))?; @@ -930,7 +1380,7 @@ fn verify_source_head(dataset: &Dataset, source: &SourceBranch, fenced: bool) -> } let stamp = match fenced { true => INTERNAL_MANIFEST_SCHEMA_VERSION, - false => UPGRADE_SOURCE_FORMAT, + false => source.stamp(), }; if read_stamp(dataset) != Some(stamp) { return Err(invalid(format!( @@ -943,17 +1393,21 @@ fn verify_source_head(dataset: &Dataset, source: &SourceBranch, fenced: bool) -> } /// The live refs are the pinned ones, each at its pinned or converted head, -/// and a fenced main still carries `pending`. +/// beside the locks of `retire` until the fence retires them, and a fenced +/// main still carries `pending`. async fn verify_inventory( root: &str, branches: &[SourceBranch], + retire: &[SourceLock], pending: Option<&UpgradeIntent>, + source: UpgradeSource, ) -> Result<()> { let main = open(root, None).await?; let observed: BTreeSet = crate::branch_control::list_live_manifest_branch_contents(&main) .await? .into_keys() + .filter(|native| !retire.iter().any(|lock| lock.native == *native)) .collect(); let expected: BTreeSet = branches .iter() @@ -968,13 +1422,13 @@ async fn verify_inventory( if pending.is_some() && intent_from(&main)?.as_ref() != pending { return Err(invalid("main no longer carries this upgrade's intent")); } - for source in branches { - let native = source.native.as_deref(); + for branch in branches { + let native = branch.native.as_deref(); let dataset = open(root, native).await?; if crate::branch_control::dataset_branch_identifier(&dataset) .await .map_err(OmniError::storage)? - != source.identity + != branch.identity { return Err(invalid(format!( "the native lifetime of {} changed during the offline upgrade", @@ -982,11 +1436,12 @@ async fn verify_inventory( ))); } let converted = match pending { - Some(intent) => branch_completed(&dataset, source, intent)?, + Some(intent) => branch_completed(&dataset, branch, intent)?, None => false, }; if !converted { - verify_source_head(&dataset, source, pending.is_some() && native.is_none())?; + let fenced = pending.is_some() && native.is_none(); + verify_source_head(&dataset, branch, fenced, source)?; } } Ok(()) @@ -1004,8 +1459,8 @@ async fn preflight_refs(root: &str, branches: &[SourceBranch]) -> Result Result Option<&TablePin> { /// The converted version of a ref holds the visible state of its source /// version: the same pinned tables, its own head commit alone under its /// logical name, and none when the ref wrote no commit of its own. -async fn equivalent_13_14( +async fn equivalent_to_source( + scanner: &dyn LegacyManifestSource, source: &Dataset, target: &Dataset, native: Option<&str>, @@ -1179,7 +1636,7 @@ async fn equivalent_13_14( source.version().version )) }; - let scan = Stamp13Source.scan_head(source).await?; + let scan = scanner.scan_head(source).await?; let (mut state, _) = read_converted_state(target).await?; state.entries.sort_by_key(|entry| entry.identity); let mut pinned: Vec<_> = head @@ -1224,7 +1681,7 @@ async fn equivalent_13_14( ))); } } - let key = native.map_or("main", crate::branch_names::logical_branch_name); + let key = native.map_or("main", logical_branch_name); let id = &head.commit.graph_commit_id; let own = head.commit.native_branch.as_deref() == native; let expected = match own { @@ -1283,12 +1740,13 @@ async fn fence_main( root: &str, pinned: Option<&SourceBranch>, intent: String, + source: UpgradeSource, target: u32, report: &mut UpgradeReport, ) -> Result { let unfenced = open(root, None).await?; if let Some(pinned) = pinned { - verify_source_head(&unfenced, pinned, false)?; + verify_source_head(&unfenced, pinned, false, source)?; } match publish_fence(unfenced, intent, target).await { Ok(_) => { @@ -1347,6 +1805,7 @@ async fn publish_activation(dataset: Dataset) -> Result { async fn publish_conversion( root: &str, session: &Arc, + scanner: &dyn LegacyManifestSource, current: Dataset, branch: &SourceBranch, intent: &UpgradeIntent, @@ -1404,7 +1863,7 @@ async fn publish_conversion( named(native) ))); } - equivalent_13_14(&source, &target, native, head).await?; + equivalent_to_source(scanner, &source, &target, native, head).await?; validate_converted_contract(&target, native, head, intent).await } diff --git a/crates/omnigraph/src/db/upgrade/root_schema.rs b/crates/omnigraph/src/db/upgrade/root_schema.rs new file mode 100644 index 000000000..6bde748c7 --- /dev/null +++ b/crates/omnigraph/src/db/upgrade/root_schema.rs @@ -0,0 +1,198 @@ +//! The three schema objects at the root of a graph stamped 8 or 9, which +//! omnigraph 0.11.x wrote in place of a `schema_contract` row: `_schema.pg`, +//! `_schema.ir.json` and `__schema_state.json`. The upgrade reads the contract +//! of such a root from them and leaves them in place; nothing served reads them. + +use omnigraph_compiler::{ + SYSTEM_COLUMNS_V3, SchemaIR, schema_ir_hash, schema_shape_hash_from_ir, validate_schema_ir, +}; +use serde::Deserialize; + +use crate::db::manifest::{SchemaContractHead, SchemaContractRow}; +use crate::db::schema_state::{ + compile_schema_source, schema_lock_conflict, validate_current_source_matches, + validate_identity_envelope, +}; +use crate::error::{OmniError, Result, StorageFailureKind}; +use crate::storage::{StorageAdapter, join_uri}; + +pub(super) const SCHEMA_SOURCE_FILENAME: &str = "_schema.pg"; +pub(super) const SCHEMA_IR_FILENAME: &str = "_schema.ir.json"; +pub(super) const SCHEMA_STATE_FILENAME: &str = "__schema_state.json"; +/// The three root objects, in the order a 0.11.x schema apply staged them. +pub(super) const SCHEMA_FILENAMES: [&str; 3] = [ + SCHEMA_SOURCE_FILENAME, + SCHEMA_IR_FILENAME, + SCHEMA_STATE_FILENAME, +]; +/// The stamp from which a 0.11.x root may spell its system columns +/// `__id`/`__src`/`__dst`. +const SYSTEM_COLUMNS_STAMP: u32 = 9; + +const SCHEMA_STATE_FORMAT_VERSION: u32 = 2; + +const MISSING_SCHEMA_CONTRACT_MESSAGE: &str = "graph is missing the mandatory identity-bearing schema contract (_schema.ir.json and __schema_state.json); automatic bootstrap is not supported"; +const INCOMPLETE_SCHEMA_CONTRACT_MESSAGE: &str = "graph schema contract is incomplete: _schema.ir.json and __schema_state.json must both be present"; + +/// `__schema_state.json` as 0.11.0 writes it: these five fields and no other. +/// A field beside them is not read, as 0.11.0 itself does not read one. +#[derive(Debug, Deserialize)] +struct RootSchemaState { + format_version: u32, + schema_shape_hash: String, + schema_ir_hash: String, + schema_identity_version: u32, + schema_identity_domain: String, +} + +/// The contract of a stamp-8 or stamp-9 root as its three objects carry it, +/// validated: the texts the conversion writes into the `schema_contract` row +/// of every live ref, and the IR those texts parse to. +#[derive(Debug)] +pub(super) struct RootSchemaContract { + pub(super) row: SchemaContractRow, + pub(super) ir: SchemaIR, +} + +/// The `.staging` object a 0.11.x schema apply left at the root, if any. +pub(super) async fn staging_object( + root: &str, + storage: &dyn StorageAdapter, +) -> Result> { + for name in SCHEMA_FILENAMES { + let staging = format!("{name}.staging"); + if storage.exists(&join_uri(root, &staging)).await? { + return Ok(Some(staging)); + } + } + Ok(None) +} + +/// Why a root stamped `stamp` cannot carry `ir`, as 0.11.x itself refused it: +/// only a stamp-8 root whose contract spells the system columns +/// `__id`/`__src`/`__dst`; the `id`/`src`/`dst` spellings convert at either stamp. +pub(super) fn stamp_covers_vintage(stamp: u32, ir: &SchemaIR) -> Result<()> { + if stamp < SYSTEM_COLUMNS_STAMP && ir.system_columns() == SYSTEM_COLUMNS_V3 { + let columns = SYSTEM_COLUMNS_V3; + return Err(OmniError::manifest(format!( + "the graph is stamped v{stamp} but its schema contract declares the \ + `{}`/`{}`/`{}` system columns, which need v{SYSTEM_COLUMNS_STAMP}; restore the \ + three root schema objects from the backup taken with the tables", + columns.id, columns.src, columns.dst + ))); + } + Ok(()) +} + +/// Load the contract from the three root objects, each in one bounded read of +/// at most `max_object_bytes`. A missing or incomplete contract reports an +/// unparseable source first, and a storage error before a parse error. +pub(super) async fn load_validated_schema_contract( + root: &str, + storage: &dyn StorageAdapter, + max_object_bytes: u64, +) -> Result { + let read = |name: &'static str| async move { + storage + .read_text_if_exists_bounded(&join_uri(root, name), max_object_bytes) + .await + .map_err(|error| over_bound(name, max_object_bytes, error)) + }; + let Some(source) = read(SCHEMA_SOURCE_FILENAME).await? else { + return Err(schema_lock_conflict(format!( + "`{SCHEMA_SOURCE_FILENAME}` is absent at the graph root" + ))); + }; + let (ir_json, state_json) = match ( + read(SCHEMA_IR_FILENAME).await?, + read(SCHEMA_STATE_FILENAME).await?, + ) { + (Some(ir_json), Some(state_json)) => (ir_json, state_json), + (None, None) => { + compile_schema_source(&source)?; + return Err(schema_lock_conflict(MISSING_SCHEMA_CONTRACT_MESSAGE)); + } + _ => { + compile_schema_source(&source)?; + return Err(schema_lock_conflict(INCOMPLETE_SCHEMA_CONTRACT_MESSAGE)); + } + }; + let current_source_shape = compile_schema_source(&source)?; + let ir = serde_json::from_str::(&ir_json).map_err(|err| { + invalid_contract_file("accepted compiled schema contract", SCHEMA_IR_FILENAME, err) + })?; + let state = serde_json::from_str::(&state_json) + .map_err(|err| invalid_contract_file("graph schema state", SCHEMA_STATE_FILENAME, err))?; + if state.format_version != SCHEMA_STATE_FORMAT_VERSION { + return Err(schema_lock_conflict(format!( + "graph schema state format {} is unsupported", + state.format_version + ))); + } + let head = SchemaContractHead { + schema_ir_hash: state.schema_ir_hash, + schema_identity_version: state.schema_identity_version, + schema_identity_domain: state.schema_identity_domain, + }; + validate_identity_envelope(&head)?; + validate_schema_ir(&ir).map_err(|error| { + schema_lock_conflict(format!( + "accepted compiled schema is not a valid identity-bearing IR: {error}" + )) + })?; + let actual_hash = schema_ir_hash(&ir).map_err(|err| schema_lock_conflict(err.to_string()))?; + if actual_hash != head.schema_ir_hash { + return Err(schema_lock_conflict( + "accepted compiled schema does not match the recorded schema state", + )); + } + let projected_shape_hash = + schema_shape_hash_from_ir(&ir).map_err(|err| schema_lock_conflict(err.to_string()))?; + if projected_shape_hash != state.schema_shape_hash { + return Err(schema_lock_conflict( + "accepted compiled schema's semantic projection does not match the recorded schema shape", + )); + } + if ir.schema_identity_domain.as_str() != head.schema_identity_domain { + return Err(schema_lock_conflict( + "accepted compiled schema identity domain does not match the recorded schema state", + )); + } + validate_current_source_matches(&ir, ¤t_source_shape)?; + Ok(RootSchemaContract { + row: SchemaContractRow { + source, + ir: ir_json, + head, + }, + ir, + }) +} + +/// Whether loading the contract stopped on the object store, not on the +/// contract: any storage failure but an absent object (`NotFound`) or a body +/// that is not text (`Permanent`). A rerun can succeed without a repair. +pub(super) fn store_did_not_answer(error: &OmniError) -> bool { + error.storage_failure().is_some_and(|failure| { + !matches!( + failure.kind, + StorageFailureKind::NotFound | StorageFailureKind::Permanent + ) + }) +} + +fn invalid_contract_file(subject: &str, file: &str, err: impl std::fmt::Display) -> OmniError { + schema_lock_conflict(format!("{subject} in {file} is invalid: {err}")) +} + +/// A read refused for the size of `name` is a fact about the contract, not +/// about the store: the object is larger than any the upgrade converts. +fn over_bound(name: &str, max_object_bytes: u64, error: OmniError) -> OmniError { + match error { + OmniError::ResourceLimitExceeded { actual, .. } => OmniError::manifest(format!( + "`{name}` is {actual} bytes, above the {max_object_bytes} bytes one root schema \ + object may hold" + )), + other => other, + } +} diff --git a/crates/omnigraph/src/db/upgrade/tests.rs b/crates/omnigraph/src/db/upgrade/tests.rs index 5e2b7f06e..43f5764af 100644 --- a/crates/omnigraph/src/db/upgrade/tests.rs +++ b/crates/omnigraph/src/db/upgrade/tests.rs @@ -1,10 +1,14 @@ use std::collections::BTreeMap; +use std::sync::atomic::{AtomicBool, Ordering}; + +use arrow_array::{RecordBatch, RecordBatchIterator}; +use async_trait::async_trait; use super::*; use crate::db::Omnigraph; use crate::db::manifest::layout::open_manifest_dataset; use crate::db::manifest::legacy::write::{ - LegacyCommitIntent, LegacyPin, LegacyPublish, Stamp13History, + LegacyCommitIntent, LegacyHistory, LegacyPin, LegacyPublish, }; use crate::db::manifest::migrations::set_stamp_for_test; use crate::db::manifest::state::{ @@ -14,6 +18,8 @@ use crate::db::manifest::{ GraphLineageRow, HISTORY_RELEASE_BYTES, HistoryReleaseBytes, LineageIntent, ManifestCoordinator, TableRegistration, TableVersionMetadata, table_path_for_identity, }; +use crate::error::{StorageFailure, StorageFailureKind}; +use crate::storage::{ListDirBounds, ObjectStorageAdapter}; async fn fresh_graph() -> (tempfile::TempDir, String) { let dir = tempfile::tempdir().unwrap(); @@ -50,11 +56,27 @@ const FENCING_EXECUTABLE: (&str, &str) = ( this executable", ); -/// The report names `remedy` (executable, action) as its one recovery, on an -/// offline graph. -fn assert_remedy(report: &UpgradeReport, remedy: (&str, &str)) { +/// The route names a report carries, spelled out: a change of `handler` is a +/// change of what operators and their tooling read. +const UNROUTED: &str = "history-lance-files-to-v14"; +const FROM_8: &str = "history-lance-files-v8-to-v14"; +const FROM_9: &str = "history-lance-files-v9-to-v14"; +const FROM_13: &str = "history-lance-files-v13-to-v14"; + +fn route_name(stamp: u32) -> &'static str { + match stamp { + 8 => FROM_8, + 9 => FROM_9, + 13 => FROM_13, + other => panic!("no route converts v{other}"), + } +} + +/// The report names `remedy` (executable, action) as its one recovery under +/// the route `handler`, on an offline graph. +fn assert_remedy(report: &UpgradeReport, handler: &str, remedy: (&str, &str)) { let recovery = report.recovery.as_ref().expect("a recovery"); - assert_eq!(recovery.failed_handler, HANDLER); + assert_eq!(recovery.failed_handler, handler, "{report:?}"); assert_eq!(recovery.executable_compatibility, remedy.0, "{report:?}"); assert!( recovery @@ -168,32 +190,181 @@ fn native(logical: &str) -> String { crate::branch_names::native_branch_name(logical, &crate::branch_names::mint_incarnation()) } -/// A stamp-13 root with main, the published `feature`, its commit-less fork +/// The schema the three root objects of a stamp-8 or stamp-9 fixture carry. +const ROOT_SCHEMA: &str = "node Person { name: String @key }\nnode Firm { name: String @key }\n"; + +/// The contract of [`ROOT_SCHEMA`] as a 0.11.x root spells its system +/// columns: `id`/`src`/`dst` at `vintage` 8, `__id`/`__src`/`__dst` at 9. +fn root_contract(vintage: u32) -> (SchemaContractRow, SchemaIR) { + let shape = crate::db::schema_state::compile_schema_source(ROOT_SCHEMA).unwrap(); + let domain = omnigraph_compiler::SchemaIdentityDomain::from_ulid(ulid::Ulid::from(77u128)); + let ir = omnigraph_compiler::initialize_schema_ir(domain, &shape) + .unwrap() + .schema_ir; + let ir = match vintage { + 8 => omnigraph_compiler::into_legacy_vintage(ir), + _ => ir, + }; + let row = crate::db::schema_state::render_schema_contract(&ir, ROOT_SCHEMA).unwrap(); + (row, ir) +} + +/// `SchemaState` of v0.11.0 (`db/schema_state.rs`), field for field and in +/// its order. +#[derive(serde::Serialize)] +struct SchemaStateV0_11 { + format_version: u32, + schema_shape_hash: String, + schema_ir_hash: String, + schema_identity_version: u32, + schema_identity_domain: String, +} + +/// The text of `__schema_state.json` v0.11.0 writes for `contract`: its +/// `render_schema_contract` prints `SchemaState` with `to_string_pretty`. +fn root_schema_state(contract: &SchemaContractRow) -> String { + let ir: SchemaIR = serde_json::from_str(&contract.ir).unwrap(); + serde_json::to_string_pretty(&SchemaStateV0_11 { + format_version: 2, + schema_shape_hash: omnigraph_compiler::schema_shape_hash_from_ir(&ir).unwrap(), + schema_ir_hash: contract.head.schema_ir_hash.clone(), + schema_identity_version: contract.head.schema_identity_version, + schema_identity_domain: contract.head.schema_identity_domain.clone(), + }) + .unwrap() +} + +/// Write `contract` as the three schema objects at `root`, as 0.11.x keeps it. +fn write_root_schema(root: &std::path::Path, contract: &SchemaContractRow) { + let [source, ir, state] = root_schema::SCHEMA_FILENAMES; + std::fs::write(root.join(source), &contract.source).unwrap(); + std::fs::write(root.join(ir), &contract.ir).unwrap(); + std::fs::write(root.join(state), root_schema_state(contract)).unwrap(); +} + +/// The bytes of each root schema object under `dir`, `None` for an absent one. +fn root_objects(dir: &std::path::Path) -> Vec<(&'static str, Option>)> { + root_schema::SCHEMA_FILENAMES + .into_iter() + .map(|name| (name, std::fs::read(dir.join(name)).ok())) + .collect() +} + +/// The registration of the node `name` of `ir` under the alias `table_key`. +fn described(ir: &SchemaIR, name: &str, table_key: &str) -> TableRegistration { + let node = ir.nodes.iter().find(|node| node.name == name).unwrap(); + let identity = TableIdentity::new(node.type_id.get(), node.table_incarnation_id.get()).unwrap(); + TableRegistration { + identity, + table_path: table_path_for_identity(table_key, identity).unwrap(), + table_key: table_key.to_string(), + } +} + +/// The columns 0.11.x stored for the type `table_key` of `ir`. +fn declared_columns(ir: &SchemaIR, table_key: &str) -> Arc { + let mut catalog = build_catalog_from_ir(ir).unwrap(); + fixup_physical_schemas(&mut catalog).unwrap(); + schema_for_table_key(&catalog, table_key).unwrap() +} + +/// Create the Lance table of `registration` at `root` with `columns`, empty, +/// and advance it to `versions` versions: the upgrade opens main's pinned +/// tables and compares their columns with the root contract. +async fn write_table( + root: &str, + registration: &TableRegistration, + columns: Arc, + versions: u64, +) { + let uri = format!("{root}/{}", registration.table_path); + let batch = RecordBatch::new_empty(Arc::clone(&columns)); + let params = WriteParams { + mode: WriteMode::Create, + data_storage_version: Some(LanceFileVersion::V2_2), + ..Default::default() + }; + let mut dataset = Dataset::write( + RecordBatchIterator::new(vec![Ok(batch)], columns), + &uri, + Some(params), + ) + .await + .unwrap(); + for version in 2..=versions { + dataset + .update_schema_metadata([("fixture_version".to_string(), version.to_string())]) + .await + .unwrap(); + } + assert_eq!(dataset.version().version, versions); +} + +/// A source root with main, the published `feature`, its commit-less fork /// `fresh`, `child` forked from a branch since retired, and two retired refs. +/// `Temp` is dropped on main alone at stamp 13, before any fork below it. struct Source { dir: tempfile::TempDir, root: String, - history: Stamp13History, + stamp: u32, + history: LegacyHistory, feature: String, fresh: String, child: String, gone: String, + person: TableIdentity, + firm: TableIdentity, + temp: TableIdentity, + /// The contract at the graph root, which only stamps 8 and 9 hold. + root_contract: Option, + /// The live `__schema_apply_lock__` ref of an unfinished 0.11.x schema + /// apply, forked at main's head by [`locked`]. + lock: Option, +} + +impl Source { + /// The table keys a live branch serves once converted. + fn branch_keys(&self) -> Vec { + let mut keys = vec!["node:Firm".to_string(), "node:Person".to_string()]; + if self.stamp == 13 { + keys.push("node:Temp".to_string()); + } + keys + } +} + +async fn source(stamp: u32) -> Source { + source_under(stamp, stamp).await } -async fn stamp_13_source() -> Source { +/// [`source`] stamped `stamp`; below 13 its root objects hold the contract of +/// a root at `vintage`. +async fn source_under(stamp: u32, vintage: u32) -> Source { let dir = tempfile::tempdir().unwrap(); let root = dir.path().to_str().unwrap().to_string(); - let [person, company, temp] = [ - table(1, "node:Person"), - table(2, "node:Company"), - table(3, "node:Temp"), - ]; + let at_root = (stamp != 13).then(|| root_contract(vintage)); + let [person, company, temp] = match &at_root { + Some((_, ir)) => [ + described(ir, "Person", "node:Person"), + described(ir, "Firm", "node:Company"), + table(900, "node:Temp"), + ], + None => [ + table(1, "node:Person"), + table(2, "node:Company"), + table(3, "node:Temp"), + ], + }; let firm = TableRegistration { table_key: "node:Firm".to_string(), ..company.clone() }; + let drop_temp = |here: bool| match here { + true => vec![(temp.identity, 1)], + false => Vec::new(), + }; let original = contract("node Person {}"); - let mut history = Stamp13History::create( + let mut history = LegacyHistory::create_stamped( &root, LegacyPublish { tables: vec![person.clone(), company.clone(), temp.clone()], @@ -202,12 +373,14 @@ async fn stamp_13_source() -> Source { commit: commit(1, None), ..Default::default() }, + stamp, ) .await .unwrap(); let second = LegacyPublish { tables: vec![firm], pins: vec![pin(&person, 2)], + drops: drop_temp(at_root.is_some()), commit: commit(2, None), ..Default::default() }; @@ -255,24 +428,45 @@ async fn stamp_13_source() -> Source { let merging = LegacyPublish { contract: Some(contract("node Person {}\n")), pins: vec![pin(&person, 7)], - drops: vec![(temp.identity, 1)], + drops: drop_temp(at_root.is_none()), commit: commit(4, Some(3)), ..Default::default() }; assert_eq!(history.publish(None, merging).await.unwrap(), 4); assert_eq!(history.fork(None, &idle).await.unwrap(), 4); history.retire(&idle).await.unwrap(); + if let Some((contract, ir)) = &at_root { + write_root_schema(dir.path(), contract); + write_table(&root, &person, declared_columns(ir, "node:Person"), 8).await; + write_table(&root, &company, declared_columns(ir, "node:Firm"), 1).await; + } Source { dir, root, + stamp, history, feature, fresh, child, gone, + person: person.identity, + firm: company.identity, + temp: temp.identity, + root_contract: at_root.map(|(contract, _)| contract), + lock: None, } } +/// `source` with the live `__schema_apply_lock__` ref a 0.11.x schema apply +/// that was killed before releasing it leaves: forked at main's head, nothing +/// published on it. +async fn locked(mut source: Source) -> Source { + let lock = native(SCHEMA_APPLY_LOCK_BRANCH); + assert_eq!(source.history.fork(None, &lock).await.unwrap(), 4); + source.lock = Some(lock); + source +} + /// The table keys a converted ref serves, sorted, and its heads. async fn served(root: &str, native: Option<&str>) -> (Vec, Vec<(String, String)>) { let dataset = open(root, native).await.unwrap(); @@ -304,11 +498,7 @@ async fn served(root: &str, native: Option<&str>) -> (Vec, Vec<(String, async fn assert_converted(source: &Source) { let root = source.root.as_str(); let both = vec!["node:Firm".to_string(), "node:Person".to_string()]; - let with_temp = vec![ - "node:Firm".to_string(), - "node:Person".to_string(), - "node:Temp".to_string(), - ]; + let with_temp = source.branch_keys(); assert_eq!( served(root, None).await, (both, vec![("main".to_string(), id(4))]) @@ -330,6 +520,46 @@ async fn assert_converted(source: &Source) { let dataset = open(root, Some(branch)).await.unwrap(); assert_eq!(dataset.version().version, version, "{branch}"); } + assert_eq!( + root_objects(source.dir.path()), + root_objects_of(source), + "the upgrade leaves the root schema objects as it read them" + ); + if let Some(lock) = &source.lock { + assert_lock_retired(root, lock).await; + } +} + +/// The lock `lock` is no live branch of the converted graph at `root` and is +/// retired as a completed 0.11.x apply leaves it. +async fn assert_lock_retired(root: &str, lock: &str) { + let main = open(root, None).await.unwrap(); + assert!( + !crate::branch_control::list_live_manifest_branch_contents(&main) + .await + .unwrap() + .contains_key(lock), + "the lock `{lock}` is no live branch of the converted graph" + ); + assert!( + retired_manifest_branches(&main) + .await + .unwrap() + .contains_key(lock), + "the lock `{lock}` is retired as a completed 0.11.x apply leaves it" + ); +} + +/// The root schema objects the fixture wrote for `source`: none at stamp 13. +fn root_objects_of(source: &Source) -> Vec<(&'static str, Option>)> { + let contract = source.root_contract.as_ref(); + let [pg, ir, state] = root_schema::SCHEMA_FILENAMES; + let text = |of: fn(&SchemaContractRow) -> String| contract.map(|row| of(row).into_bytes()); + vec![ + (pg, text(|row| row.source.clone())), + (ir, text(|row| row.ir.clone())), + (state, text(root_schema_state)), + ] } #[tokio::test] @@ -386,6 +616,11 @@ async fn a_target_other_than_the_served_format_is_unsupported() { .unwrap(); assert_eq!(report.outcome, UpgradeOutcome::CheckFailed); assert_eq!(codes(&report), ["unsupported_target"]); + assert_eq!( + report.findings[0].message, + "this binary converts storage formats v8, v9 and v13 to v14 and serves v14 alone; \ + `--to-format` accepts 14 only" + ); assert_eq!(report.target_format, target); assert!(!report.target_defaulted); } @@ -393,8 +628,8 @@ async fn a_target_other_than_the_served_format_is_unsupported() { /// Set `intent` as main's pending key on a fresh graph and report the upgrade /// in `mode`: recovery is required under `code`, nothing is written, and the -/// finding carries the guidance of the keyed version. Returns that guidance. -async fn pending_report(intent: &str, check: bool, code: &str) -> String { +/// finding carries that version's guidance under `handler`. Returns the guidance. +async fn pending_report(intent: &str, check: bool, code: &str, handler: &str) -> String { let (_dir, uri) = fresh_graph().await; let mut manifest = open_manifest_dataset(&uri, None).await.unwrap(); manifest @@ -413,6 +648,7 @@ async fn pending_report(intent: &str, check: bool, code: &str) -> String { assert_eq!(report.findings[0].message, recovery_guidance(&manifest)); assert_remedy( &report, + handler, match code { "pending_upgrade" => RERUN, _ => PRESERVE_AND_CHECK, @@ -425,44 +661,50 @@ async fn pending_report(intent: &str, check: bool, code: &str) -> String { #[tokio::test] async fn a_pending_conversion_with_a_valid_intent_is_reported_by_check_as_pending() { let attempt = id(9); - let hash = "a".repeat(64); - let intent = UpgradeIntent { - protocol: UPGRADE_PROTOCOL, - attempt: attempt.clone(), - source_format: UPGRADE_SOURCE_FORMAT, - target_format: INTERNAL_MANIFEST_SCHEMA_VERSION, - graph_identity: "domain".to_string(), - branches: vec![SourceBranch { - native: None, - identity: lance::dataset::refs::BranchIdentifier::main(), - version: 1, - parent_version: 0, - }], - schema_contract: Some(UpgradeSchemaContract::from_row(&contract("node Person {}"))), - legacy: LegacyPlan { - layout: LegacyLayout::CURRENT, - commits: 1, - data_files: 1, - id_shards: 1, - writer_shards: 1, - directory_sha256: hash, - }, - }; - let json = serde_json::to_string(&intent).unwrap(); - let guidance = pending_report(&json, true, "pending_upgrade").await; - assert!( - guidance.contains(&format!( - "the pending storage conversion of attempt {attempt} to format v14" - )) && guidance - .contains("rerun `omnigraph upgrade ` without `--check` with this executable"), - "{guidance}" - ); + for source_format in [8, 9, 13] { + let intent = UpgradeIntent { + protocol: UPGRADE_PROTOCOL, + attempt: attempt.clone(), + source_format, + target_format: INTERNAL_MANIFEST_SCHEMA_VERSION, + graph_identity: "domain".to_string(), + branches: vec![SourceBranch { + native: None, + identity: lance::dataset::refs::BranchIdentifier::main(), + version: 1, + parent_version: 0, + }], + retire: Vec::new(), + schema_contract: Some( + UpgradeSchemaContract::from_row(&contract("node Person {}")).unwrap(), + ), + legacy: LegacyPlan { + layout: LegacyLayout::CURRENT, + commits: 1, + data_files: 1, + id_shards: 1, + writer_shards: 1, + directory_sha256: "a".repeat(64), + }, + }; + let json = serde_json::to_string(&intent).unwrap(); + let handler = route_name(source_format); + let guidance = pending_report(&json, true, "pending_upgrade", handler).await; + assert!( + guidance.contains(&format!( + "the pending storage conversion of attempt {attempt} to format v14" + )) && guidance.contains( + "rerun `omnigraph upgrade ` without `--check` with this executable" + ), + "{guidance}" + ); + } } #[tokio::test] async fn a_pending_conversion_with_an_unreadable_intent_is_unknown_ownership() { for check in [true, false] { - let guidance = pending_report("{}", check, "unknown_upgrade_ownership").await; + let guidance = pending_report("{}", check, "unknown_upgrade_ownership", UNROUTED).await; assert!( guidance.contains("whose ownership this executable cannot establish") && guidance.contains("unrecognized upgrade ownership: missing field") @@ -475,84 +717,107 @@ async fn a_pending_conversion_with_an_unreadable_intent_is_unknown_ownership() { #[tokio::test] async fn check_has_no_local_store_effects() { - let source = stamp_13_source().await; - let before = stored_files(source.dir.path()); - let report = check(&source.root).await; - assert_eq!(report.outcome, UpgradeOutcome::CheckPassed, "{report:?}"); - assert!(report.findings.is_empty() && report.recovery.is_none()); - assert_eq!(report.observed_format, Some(13)); - assert_eq!(report.graph_identity.as_deref(), Some("domain")); - assert_eq!(report.route, [HANDLER]); - assert!(report.completed_handlers.is_empty()); - let work = report.work; - assert_eq!( - ( - work.live_refs, - work.retired_refs, - work.orphan_writers, - work.legacy_commits, - work.absent_parents, - work.data_files, - work.id_shards, - work.writer_shards, - work.schema_contents, - ), - (4, 2, 0, 6, 0, 3, 1, 1, 2), - "{work:?}" - ); - assert!( - work.bookkeeping_versions >= 2 - && work.census_reads > work.live_refs + work.retired_refs - && work.census_cells > 0 - && work.legacy_bytes > 0, - "{work:?}" - ); - assert_eq!(stored_files(source.dir.path()), before); + for (stamp, identity, schema_contents, table_opens) in [ + (13, "domain".to_string(), 2, 0), + (9, id(77), 1, 2), + (8, id(77), 1, 2), + ] { + let source = source(stamp).await; + let before = stored_files(source.dir.path()); + let report = check(&source.root).await; + assert_eq!(report.outcome, UpgradeOutcome::CheckPassed, "{report:?}"); + assert!(report.findings.is_empty() && report.recovery.is_none()); + assert_eq!(report.observed_format, Some(stamp)); + assert_eq!(report.graph_identity, Some(identity)); + assert_eq!(report.route, [route_name(stamp)]); + assert!(report.completed_handlers.is_empty()); + let work = report.work; + assert_eq!( + ( + work.live_refs, + work.retired_refs, + work.orphan_writers, + work.legacy_commits, + work.absent_parents, + work.data_files, + work.id_shards, + work.writer_shards, + work.schema_contents, + work.table_opens, + ), + (4, 2, 0, 6, 0, 3, 1, 1, schema_contents, table_opens), + "v{stamp}: {work:?}" + ); + assert!( + work.bookkeeping_versions >= 2 + && work.census_reads > work.live_refs + work.retired_refs + && work.census_cells > 0 + && work.legacy_bytes > 0, + "v{stamp}: {work:?}" + ); + assert_eq!(stored_files(source.dir.path()), before, "v{stamp}"); + assert_eq!( + root_objects(source.dir.path()), + root_objects_of(&source), + "v{stamp}" + ); + } } #[tokio::test] -async fn a_stamp_13_root_is_converted_once_and_is_then_current() { +async fn a_source_root_is_converted_once_and_is_then_current() { #[cfg(feature = "failpoints")] let _scenario = crate::seams::FailScenario::setup(); - let source = stamp_13_source().await; - let checked = check(&source.root).await.work; - let report = execute(&source.root).await; - assert_eq!(report.outcome, UpgradeOutcome::Completed, "{report:?}"); - assert!(report.findings.is_empty() && report.recovery.is_none()); - assert_eq!(report.completed_handlers, [HANDLER]); - assert_eq!( - report.last_durable_completed_boundary.as_deref(), - Some("activated") - ); - assert_eq!(report.work, checked); - assert_converted(&source).await; + for stamp in [13, 9, 8] { + let source = source(stamp).await; + let checked = check(&source.root).await.work; + let report = execute(&source.root).await; + assert_eq!(report.outcome, UpgradeOutcome::Completed, "{report:?}"); + assert!(report.findings.is_empty() && report.recovery.is_none()); + assert_eq!(report.observed_format, Some(stamp)); + assert_eq!(report.route, [route_name(stamp)]); + assert_eq!(report.completed_handlers, [route_name(stamp)]); + assert_eq!( + report.last_durable_completed_boundary.as_deref(), + Some("activated") + ); + assert_eq!(report.work, checked); + assert_converted(&source).await; - let files = stored_files(source.dir.path()); - let legacy: Vec<&str> = files - .keys() - .filter_map(|name| name.strip_prefix("__history/legacy/")) - .collect(); - assert_eq!( - legacy, - [ - "data/00000000.lance", - "data/00000001.lance", - "data/00000002.lance", - "locator/directory.oglx", - "locator/ids/00000000.oglx", - "locator/writers/00000000.oglx", - ] - ); - for again in [check(&source.root).await, execute(&source.root).await] { - assert_eq!(again.outcome, UpgradeOutcome::AlreadyCurrent, "{again:?}"); - assert_eq!(again.observed_format, Some(14)); + let files = stored_files(source.dir.path()); + let legacy: Vec<&str> = files + .keys() + .filter_map(|name| name.strip_prefix("__history/legacy/")) + .collect(); + assert_eq!( + legacy, + [ + "data/00000000.lance", + "data/00000001.lance", + "data/00000002.lance", + "locator/directory.oglx", + "locator/ids/00000000.oglx", + "locator/writers/00000000.oglx", + ], + "v{stamp}" + ); + for again in [check(&source.root).await, execute(&source.root).await] { + assert_eq!(again.outcome, UpgradeOutcome::AlreadyCurrent, "{again:?}"); + assert_eq!(again.observed_format, Some(14)); + } + assert_eq!(stored_files(source.dir.path()), files, "v{stamp}"); } - assert_eq!(stored_files(source.dir.path()), files); } #[tokio::test] async fn history_leftovers_refuse_before_fence() { - let source = stamp_13_source().await; + for stamp in [9, 13] { + history_leftovers_refuse(source(stamp).await).await; + } +} + +async fn history_leftovers_refuse(source: Source) { + let stamp = source.stamp; let planted = source.dir.path().join("__history/legacy/data"); std::fs::create_dir_all(&planted).unwrap(); std::fs::write( @@ -564,13 +829,12 @@ async fn history_leftovers_refuse_before_fence() { for report in [check(&source.root).await, execute(&source.root).await] { assert_eq!(report.outcome, UpgradeOutcome::CheckFailed, "{report:?}"); assert_eq!(codes(&report), ["history_objects_present"]); + let message = &report.findings[0].message; assert!( - report.findings[0] - .message - .contains("legacy/data/00000000.lance") - && report.findings[0] - .message - .contains("restore the whole root, `__history/` included"), + message.starts_with(&format!( + "`__history/` already holds objects no v{stamp} build writes (" + )) && message.contains("legacy/data/00000000.lance") + && message.contains("restore the whole root, `__history/` included"), "{report:?}" ); } @@ -579,7 +843,12 @@ async fn history_leftovers_refuse_before_fence() { #[tokio::test] async fn over_bound_record_refuses_before_fence() { - let mut source = stamp_13_source().await; + for stamp in [9, 13] { + over_bound_record_refuses(source(stamp).await).await; + } +} + +async fn over_bound_record_refuses(mut source: Source) { let fat = LegacyPublish { commit: Some(LegacyCommitIntent { actor_id: Some("a".repeat(omnigraph_catalog::HISTORY_RELEASE_BYTES)), @@ -616,7 +885,7 @@ async fn own_head_without_its_graph_head_row_refuses_before_fence() { commit: commit(1, None), ..Default::default() }; - let history = Stamp13History::create(root, genesis).await.unwrap(); + let history = LegacyHistory::create(root, genesis).await.unwrap(); let mut main = history.head(None).unwrap().clone(); main.delete("object_type = 'graph_head'").await.unwrap(); assert_eq!(main.version().version, 2); @@ -642,7 +911,12 @@ async fn own_head_without_its_graph_head_row_refuses_before_fence() { #[tokio::test] async fn census_over_bound_refuses_before_reads() { - let source = stamp_13_source().await; + for stamp in [9, 13] { + census_over_bound_refuses(source(stamp).await).await; + } +} + +async fn census_over_bound_refuses(source: Source) { let cells = check(&source.root).await.work.census_cells; let before = stored_files(source.dir.path()); for check in [true, false] { @@ -687,7 +961,12 @@ async fn census_over_bound_refuses_before_reads() { #[tokio::test] async fn table_snapshots_over_bound_refuse_before_fence() { - let source = stamp_13_source().await; + for stamp in [9, 13] { + table_snapshots_over_bound_refuse(source(stamp).await).await; + } +} + +async fn table_snapshots_over_bound_refuse(source: Source) { let before = stored_files(source.dir.path()); let version = manifest_version(&source.root).await; let bounds = Bounds { @@ -728,6 +1007,8 @@ async fn bounded(root: &str, check: bool, bounds: Bounds) -> UpgradeReport { #[tokio::test] async fn retired_head_over_budget_refuses_before_fence() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); let dir = tempfile::tempdir().unwrap(); let root = dir.path().to_str().unwrap(); let person = table(1, "node:Person"); @@ -738,7 +1019,7 @@ async fn retired_head_over_budget_refuses_before_fence() { commit: commit(1, None), ..Default::default() }; - let mut history = Stamp13History::create(root, genesis).await.unwrap(); + let mut history = LegacyHistory::create(root, genesis).await.unwrap(); let gone = native("gone"); assert_eq!(history.fork(None, &gone).await.unwrap(), 1); let own = LegacyPublish { @@ -827,99 +1108,1034 @@ async fn retired_head_over_budget_refuses_before_fence() { assert_eq!(report.work, UpgradeWork::default()); } } - assert_eq!(stored_files(dir.path()), before); - assert_eq!(manifest_version(root).await, 1); - let main = open_manifest_dataset(root, None).await.unwrap(); + assert_eq!(stored_files(dir.path()), before); + assert_eq!(manifest_version(root).await, 1); + let main = open_manifest_dataset(root, None).await.unwrap(); + assert!(!main.schema().metadata.contains_key(UPGRADE_PENDING_KEY)); + + let admitted = Bounds { + head_rows: gone_rows, + retired_refs: 1, + ..Bounds::SERVED + }; + let report = bounded(root, false, admitted).await; + assert_eq!(report.outcome, UpgradeOutcome::Completed, "{report:?}"); + assert_eq!(report.work.retired_refs, 1); +} + +#[tokio::test] +async fn policy_denial_precedes_effects() { + struct DenySchemaApply; + impl omnigraph_policy::PolicyChecker for DenySchemaApply { + fn check( + &self, + action: omnigraph_policy::PolicyAction, + scope: &omnigraph_policy::ResourceScope, + actor: &str, + ) -> std::result::Result<(), omnigraph_policy::PolicyError> { + assert_eq!(action, omnigraph_policy::PolicyAction::SchemaApply); + assert!(matches!( + scope, + omnigraph_policy::ResourceScope::TargetBranch(_) + )); + assert_eq!(actor, "blocked-actor"); + Err(omnigraph_policy::PolicyError::Denied("test denial".into())) + } + } + /// What a stamp-9 root may hold beside its contract: the denial precedes + /// every report the upgrade would make of it. + #[derive(Debug, Clone, Copy)] + enum RootState { + Clean, + StagedObject, + UnparsableSource, + LiveLock, + } + for (stamp, state) in [ + (13, RootState::Clean), + (9, RootState::Clean), + (9, RootState::StagedObject), + (9, RootState::UnparsableSource), + (9, RootState::LiveLock), + ] { + let source = match state { + RootState::LiveLock => locked(source(stamp).await).await, + _ => source(stamp).await, + }; + let [pg, _, _] = root_schema::SCHEMA_FILENAMES; + match state { + RootState::StagedObject => { + std::fs::write(source.dir.path().join(format!("{pg}.staging")), "staged").unwrap(); + } + RootState::UnparsableSource => { + std::fs::write(source.dir.path().join(pg), "node {").unwrap(); + } + RootState::Clean | RootState::LiveLock => {} + } + let before = stored_files(source.dir.path()); + let report = upgrade_storage_as( + &source.root, + UpgradeOptions::default(), + Some("blocked-actor"), + Some(&DenySchemaApply), + ) + .await + .unwrap(); + assert_eq!( + report.outcome, + UpgradeOutcome::CheckFailed, + "v{stamp} {state:?}: {report:?}" + ); + assert_eq!(codes(&report), ["preflight_failed"], "v{stamp} {state:?}"); + assert!( + report.findings[0].message.contains("test denial"), + "v{stamp} {state:?}: {report:?}" + ); + assert_eq!(report.graph_identity, None, "v{stamp} {state:?}"); + assert_eq!(stored_files(source.dir.path()), before); + } +} + +/// Main at another version than the inventory pinned is found before the +/// fence commit: nothing is written, so the report asks for no recovery. +#[tokio::test] +async fn main_moved_after_inventory_fails_preflight_without_recovery() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + for stamp in [9, 13] { + main_moved_after_inventory(source(stamp).await).await; + } +} + +async fn main_moved_after_inventory(source: Source) { + let root = source.root.as_str(); + let before = stored_files(source.dir.path()); + let main = open(root, None).await.unwrap(); + let mut pinned = inventory(&main).await.unwrap().pop().unwrap(); + assert_eq!(pinned.native, None, "main is the last pinned ref"); + pinned.version -= 1; + + let mut report = UpgradeReport::started(root.to_string(), UpgradeOptions::default()); + let error = fence_main( + root, + Some(&pinned), + "{}".into(), + UpgradeSource::from_stamp(source.stamp).unwrap(), + report.target_format, + &mut report, + ) + .await + .unwrap_err(); + report.failed(&error); + + assert_eq!(report.outcome, UpgradeOutcome::CheckFailed, "{report:?}"); + assert_eq!(codes(&report), ["preflight_failed"]); + assert!( + report.findings[0] + .message + .contains("a writer moved it during the offline upgrade"), + "{report:?}" + ); + assert!(report.recovery.is_none(), "{report:?}"); + assert_eq!(report.last_durable_completed_boundary, None); + assert_eq!(stored_files(source.dir.path()), before); + assert_eq!( + execute(root).await.outcome, + UpgradeOutcome::Completed, + "the unfenced root still converts" + ); +} + +/// The report refuses the source for unfinished work of the build that wrote +/// it: `message` under `handler`, resolved by `build` through `action`, with +/// nothing read of the history and nothing written. +fn assert_source_recovery( + report: &UpgradeReport, + handler: &str, + message: &str, + build: &str, + action: &str, +) { + assert_eq!( + report.outcome, + UpgradeOutcome::RecoveryRequired, + "{report:?}" + ); + assert_eq!(codes(report), ["source_recovery_required"], "{report:?}"); + assert_eq!(report.findings[0].message, message); + let recovery = report.recovery.as_ref().expect("a recovery"); + assert_eq!( + ( + recovery.failed_handler.as_str(), + recovery.executable_compatibility.as_str(), + recovery.action.as_str() + ), + (handler, build, action) + ); + assert_eq!(report.last_durable_completed_boundary, None); + assert_eq!(report.work, UpgradeWork::default()); +} + +const RELEASE_0_11: &str = "omnigraph 0.11.x"; + +#[tokio::test] +async fn recovery_sidecars_refuse_under_the_route_of_the_stamp() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + const BUILD: &str = "the omnigraph build that wrote the recovery sidecars"; + const ACTION: &str = "stop all writers, retain the backup, open the graph read-write with \ + the build that wrote the sidecars so it finishes its recovery, then \ + rerun `omnigraph upgrade --check`"; + const MESSAGE: &str = "1 recovery sidecar(s) under `__recovery/` (01A) must be resolved \ + before the storage conversion"; + let plant = |dir: &std::path::Path| { + std::fs::create_dir(dir.join("__recovery")).unwrap(); + std::fs::write(dir.join("__recovery/01A.json"), "{}").unwrap(); + }; + for stamp in [8, 9, 13] { + let source = source(stamp).await; + plant(source.dir.path()); + let before = stored_files(source.dir.path()); + for report in [check(&source.root).await, execute(&source.root).await] { + assert_source_recovery(&report, route_name(stamp), MESSAGE, BUILD, ACTION); + assert!(report.route.is_empty(), "{report:?}"); + } + assert_eq!(stored_files(source.dir.path()), before); + } + + let (dir, uri) = fresh_graph().await; + let mut manifest = open_manifest_dataset(&uri, None).await.unwrap(); + set_stamp_for_test(&mut manifest, 10).await.unwrap(); + plant(dir.path()); + let report = check(&uri).await; + assert_source_recovery(&report, UNROUTED, MESSAGE, BUILD, ACTION); +} + +/// The staging probe runs before anything the stamp decides, so each object +/// is staged once, under one of the two routes. +#[tokio::test] +async fn a_staged_schema_object_at_the_root_refuses_before_the_contract_is_read() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + for (stamp, name) in [8, 9, 9].into_iter().zip(root_schema::SCHEMA_FILENAMES) { + let source = source(stamp).await; + let staging = format!("{name}.staging"); + std::fs::write(source.dir.path().join(&staging), "staged").unwrap(); + let before = stored_files(source.dir.path()); + for report in [check(&source.root).await, execute(&source.root).await] { + assert_source_recovery( + &report, + route_name(stamp), + &format!( + "schema object `{staging}` at the graph root is the unfinished part of \ + a schema apply by omnigraph 0.11.x; it must be resolved before the \ + storage conversion" + ), + RELEASE_0_11, + "stop all writers, retain the backup, open the graph read-write with \ + omnigraph 0.11.x so it finishes or rolls back the schema apply, then rerun \ + `omnigraph upgrade --check`", + ); + assert_eq!(report.graph_identity, None); + } + assert_eq!(stored_files(source.dir.path()), before); + assert_eq!(manifest_version(&source.root).await, 4); + } +} + +/// Every way `load_validated_schema_contract` refuses the three root objects, +/// one case per arm, each reported as the unsupported source with the loader's +/// own words. +#[tokio::test] +async fn an_unreadable_root_contract_refuses_as_unsupported_source() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + let [pg, ir, state] = root_schema::SCHEMA_FILENAMES; + let remove = |names: &'static [&'static str]| { + move |dir: &std::path::Path, _: &SchemaContractRow| { + for name in names { + std::fs::remove_file(dir.join(name)).unwrap(); + } + } + }; + let write = |name: &'static str, text: &'static str| { + move |dir: &std::path::Path, _: &SchemaContractRow| { + std::fs::write(dir.join(name), text).unwrap() + } + }; + let edit_state = |edit: fn(&mut serde_json::Value)| { + move |dir: &std::path::Path, contract: &SchemaContractRow| { + let mut recorded: serde_json::Value = + serde_json::from_str(&root_schema_state(contract)).unwrap(); + edit(&mut recorded); + std::fs::write(dir.join(state), recorded.to_string()).unwrap(); + } + }; + let other_domain = |dir: &std::path::Path, contract: &SchemaContractRow| { + let mut recorded: serde_json::Value = + serde_json::from_str(&root_schema_state(contract)).unwrap(); + recorded["schema_identity_domain"] = + omnigraph_compiler::SchemaIdentityDomain::from_ulid(ulid::Ulid::from(78u128)) + .as_str() + .into(); + std::fs::write(dir.join(state), recorded.to_string()).unwrap(); + }; + let foreign_ir_version = |dir: &std::path::Path, contract: &SchemaContractRow| { + let mut accepted: serde_json::Value = serde_json::from_str(&contract.ir).unwrap(); + accepted["ir_version"] = 999.into(); + std::fs::write(dir.join(ir), accepted.to_string()).unwrap(); + }; + type Damage<'a> = Box; + let cases: [(Damage<'_>, &[&str]); 14] = [ + ( + Box::new(remove(&["_schema.pg"])), + &["`_schema.pg` is absent at the graph root"], + ), + ( + Box::new(remove(&["_schema.ir.json", "__schema_state.json"])), + &[ + "graph is missing the mandatory identity-bearing schema contract (_schema.ir.json \ + and __schema_state.json); automatic bootstrap is not supported", + ], + ), + ( + Box::new(remove(&["_schema.ir.json"])), + &[ + "graph schema contract is incomplete: _schema.ir.json and __schema_state.json \ + must both be present", + ], + ), + ( + Box::new(write(pg, "node {")), + &["schema-contract source is not a valid accepted schema definition: "], + ), + ( + Box::new(write(ir, "{")), + &["accepted compiled schema contract in _schema.ir.json is invalid: "], + ), + ( + Box::new(write(state, "{")), + &["graph schema state in __schema_state.json is invalid: "], + ), + ( + Box::new(edit_state(|recorded| recorded["format_version"] = 1.into())), + &["graph schema state format 1 is unsupported"], + ), + ( + Box::new(edit_state(|recorded| { + recorded["schema_identity_version"] = 1.into() + })), + &["graph schema identity version 1 is unsupported"], + ), + ( + Box::new(edit_state(|recorded| { + recorded["schema_identity_domain"] = "not-a-domain".into() + })), + &["graph schema identity domain is invalid: "], + ), + ( + Box::new(foreign_ir_version), + &[ + "accepted compiled schema is not a valid identity-bearing IR: ", + "unsupported ir_version 999", + ], + ), + ( + Box::new(edit_state(|recorded| { + recorded["schema_ir_hash"] = + serde_json::Value::from(format!("sha256:{}", "0".repeat(64))) + })), + &["accepted compiled schema does not match the recorded schema state"], + ), + ( + Box::new(edit_state(|recorded| { + recorded["schema_shape_hash"] = + serde_json::Value::from(format!("sha256:{}", "0".repeat(64))) + })), + &[ + "accepted compiled schema's semantic projection does not match the recorded \ + schema shape", + ], + ), + ( + Box::new(other_domain), + &[ + "accepted compiled schema identity domain does not match the recorded schema \ + state", + ], + ), + ( + Box::new(write( + pg, + "node Person {\n name: String @key\n age: I64\n}\nnode Firm { name: String @key }\n", + )), + &["schema-contract source no longer matches the accepted compiled schema"], + ), + ]; + for (damage, causes) in cases { + let source = source(9).await; + damage(source.dir.path(), source.root_contract.as_ref().unwrap()); + let before = stored_files(source.dir.path()); + for report in [check(&source.root).await, execute(&source.root).await] { + assert_eq!(report.outcome, UpgradeOutcome::CheckFailed, "{report:?}"); + assert_eq!(codes(&report), ["unsupported_source"]); + let message = &report.findings[0].message; + let loader_error = message + .strip_prefix( + "the schema contract of this v9 graph cannot be read from `_schema.pg`, \ + `_schema.ir.json` and `__schema_state.json` at the graph root: ", + ) + .and_then(|rest| { + rest.strip_suffix( + "; restore the whole root from one backup (omnigraph 0.11.x refuses to \ + open this state as well)", + ) + }) + .unwrap_or_else(|| panic!("{message}")); + assert!( + causes.iter().all(|cause| loader_error.contains(cause)), + "{loader_error}" + ); + assert_eq!(report.route, [FROM_9]); + assert!(report.recovery.is_none() && report.last_durable_completed_boundary.is_none()); + assert_eq!(report.work, UpgradeWork::default()); + } + assert_eq!(stored_files(source.dir.path()), before); + assert_eq!(manifest_version(&source.root).await, 4); + } +} + +/// The local store, whose first bounded read fails as a transient store +/// failure. Loading a root contract probes for staging and reads the three +/// objects bounded; it calls nothing else. +#[derive(Debug)] +struct FirstReadFails { + inner: ObjectStorageAdapter, + failed: AtomicBool, +} + +const ROUTE_ONLY_READS: &str = "the route probes for staging and reads the root contract bounded"; + +#[async_trait] +impl StorageAdapter for FirstReadFails { + async fn read_text(&self, _: &str) -> Result { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn exists(&self, uri: &str) -> Result { + self.inner.exists(uri).await + } + + async fn read_text_if_exists(&self, _: &str) -> Result> { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn read_text_if_exists_bounded(&self, uri: &str, max: u64) -> Result> { + if self.failed.swap(true, Ordering::SeqCst) { + return self.inner.read_text_if_exists_bounded(uri, max).await; + } + Err(OmniError::Storage(StorageFailure::new( + StorageFailureKind::Transient, + format!("storage read failed for '{uri}': connection reset"), + ))) + } + + async fn read_bytes_if_exists_bounded(&self, _: &str, _: u64) -> Result>> { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn write_text(&self, _: &str, _: &str) -> Result<()> { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn write_bytes(&self, _: &str, _: &[u8]) -> Result<()> { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn write_text_if_absent(&self, _: &str, _: &str) -> Result { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn rename_text(&self, _: &str, _: &str) -> Result<()> { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn delete(&self, _: &str) -> Result<()> { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn list_dir(&self, _: &str) -> Result> { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn list_dir_bounded(&self, _: &str, _: &str, _: ListDirBounds) -> Result> { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn read_text_versioned(&self, _: &str) -> Result<(String, String)> { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn write_text_if_match(&self, _: &str, _: &str, _: &str) -> Result> { + unreachable!("{ROUTE_ONLY_READS}") + } + + async fn delete_prefix(&self, _: &str) -> Result<()> { + unreachable!("{ROUTE_ONLY_READS}") + } +} + +/// A store that fails a read of a root schema object says nothing of the +/// contract: the run fails preflight, and the same root routes on a rerun. +#[tokio::test] +async fn a_store_failure_reading_the_root_contract_fails_preflight_and_a_rerun_routes() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + let source = source(9).await; + let root = normalize_root_uri(&source.root).unwrap(); + let before = stored_files(source.dir.path()); + let storage = FirstReadFails { + inner: ObjectStorageAdapter::local(), + failed: AtomicBool::new(false), + }; + let main = open(&root, None).await.unwrap(); + + let mut report = UpgradeReport::started(root.clone(), UpgradeOptions::default()); + let error = route( + &root, + &storage, + UpgradeSource::Stamp9, + &main, + None, + Bounds::SERVED, + &mut report, + ) + .await + .err() + .expect("the failed read stops the route"); + report.failed(&error); + assert_eq!(report.outcome, UpgradeOutcome::CheckFailed, "{report:?}"); + assert_eq!(codes(&report), ["preflight_failed"]); + assert_eq!( + report.findings[0].message, + format!("storage read failed for '{root}/_schema.pg': connection reset") + ); + assert!(report.recovery.is_none() && report.last_durable_completed_boundary.is_none()); + + let mut rerun = UpgradeReport::started(root.clone(), UpgradeOptions::default()); + let routed = route( + &root, + &storage, + UpgradeSource::Stamp9, + &main, + None, + Bounds::SERVED, + &mut rerun, + ) + .await; + assert!(matches!(routed, Ok(Some(_))), "{rerun:?}"); + assert!(rerun.findings.is_empty(), "{rerun:?}"); + assert_eq!(stored_files(source.dir.path()), before); +} + +/// A live ref under another stamp than main's is refused before the fence, +/// whichever of the two 0.11.x stamps main carries. +#[tokio::test] +async fn a_live_ref_stamped_differently_from_main_fails_preflight() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + for (stamp, other) in [(9, 8), (8, 9)] { + let source = source(stamp).await; + let mut feature = open(&source.root, Some(&source.feature)).await.unwrap(); + set_stamp_for_test(&mut feature, other).await.unwrap(); + let version = feature.version().version; + let before = stored_files(source.dir.path()); + for report in [check(&source.root).await, execute(&source.root).await] { + assert_eq!(report.outcome, UpgradeOutcome::CheckFailed, "{report:?}"); + assert_eq!(codes(&report), ["preflight_failed"]); + assert_eq!( + report.findings[0].message, + format!( + "ref '{}' is stamped Some({other}) at version {version}, the upgrade \ + expects v{stamp} there", + source.feature + ) + ); + assert!(report.recovery.is_none() && report.last_durable_completed_boundary.is_none()); + } + assert_eq!(stored_files(source.dir.path()), before); + let main = open_manifest_dataset(&source.root, None).await.unwrap(); + assert!(!main.schema().metadata.contains_key(UPGRADE_PENDING_KEY)); + } +} + +/// The fixture's `__schema_state.json` is the text v0.11.0 writes, and a +/// field beside its five, which v0.11.0 does not read either, converts. +#[tokio::test] +async fn the_schema_state_v0_11_0_writes_converts_with_or_without_an_unread_field() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + let [_, _, state] = root_schema::SCHEMA_FILENAMES; + let plain = source(9).await; + let contract = plain.root_contract.clone().unwrap(); + let ir: SchemaIR = serde_json::from_str(&contract.ir).unwrap(); + let written = std::fs::read_to_string(plain.dir.path().join(state)).unwrap(); + assert_eq!( + written, + format!( + "{{\n \"format_version\": 2,\n \"schema_shape_hash\": \"{}\",\n \ + \"schema_ir_hash\": \"{}\",\n \"schema_identity_version\": {},\n \ + \"schema_identity_domain\": \"{}\"\n}}", + omnigraph_compiler::schema_shape_hash_from_ir(&ir).unwrap(), + contract.head.schema_ir_hash, + contract.head.schema_identity_version, + contract.head.schema_identity_domain, + ) + ); + let report = execute(&plain.root).await; + assert_eq!(report.outcome, UpgradeOutcome::Completed, "{report:?}"); + assert_converted(&plain).await; + + let wider = source(9).await; + let mut recorded: serde_json::Value = serde_json::from_str(&written).unwrap(); + recorded["publication"] = serde_json::json!({"graph_commit_id": id(4)}); + let recorded = serde_json::to_string_pretty(&recorded).unwrap(); + std::fs::write(wider.dir.path().join(state), &recorded).unwrap(); + let report = execute(&wider.root).await; + assert_eq!(report.outcome, UpgradeOutcome::Completed, "{report:?}"); + let main = open(&wider.root, None).await.unwrap(); + let (_, converted) = read_converted_state(&main).await.unwrap(); + assert_eq!(converted, contract); + assert_eq!( + std::fs::read_to_string(wider.dir.path().join(state)).unwrap(), + recorded + ); +} + +/// As 0.11.x does, the upgrade refuses `__id`/`__src`/`__dst` under stamp 8 +/// and admits `id`/`src`/`dst` under stamp 9; the converted contract is the +/// root contract. +#[tokio::test] +async fn stamp_8_refuses_v3_system_columns_and_stamp_9_converts_the_legacy_ones() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + let refused = source_under(8, 9).await; + let before = stored_files(refused.dir.path()); + for report in [check(&refused.root).await, execute(&refused.root).await] { + assert_eq!(report.outcome, UpgradeOutcome::CheckFailed, "{report:?}"); + assert_eq!(codes(&report), ["unsupported_source"]); + assert_eq!( + report.findings[0].message, + "the graph is stamped v8 but its schema contract declares the \ + `__id`/`__src`/`__dst` system columns, which need v9; restore the three root \ + schema objects from the backup taken with the tables" + ); + assert!(report.recovery.is_none()); + } + assert_eq!(stored_files(refused.dir.path()), before); + + for (stamp, vintage, v3_columns) in [(9, 8, false), (9, 9, true)] { + let converted = source_under(stamp, vintage).await; + let report = execute(&converted.root).await; + assert_eq!(report.outcome, UpgradeOutcome::Completed, "{report:?}"); + assert_converted(&converted).await; + let main = open(&converted.root, None).await.unwrap(); + let (_, contract) = read_converted_state(&main).await.unwrap(); + assert_eq!(Some(&contract), converted.root_contract.as_ref()); + let ir: SchemaIR = serde_json::from_str(&contract.ir).unwrap(); + assert_eq!( + ir.system_columns() == omnigraph_compiler::SYSTEM_COLUMNS_V3, + v3_columns, + "v{stamp} under the contract of a v{vintage} root" + ); + } +} + +/// The lock a killed 0.11.x schema apply left on an otherwise clean root is +/// retired by the upgrade as the completed apply would have retired it; a +/// lock the apply did retire converts as any retired ref. +#[tokio::test] +async fn a_live_schema_apply_lock_is_retired_and_a_retired_one_converts() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + let locked = locked(source(9).await).await; + let lock = locked.lock.clone().unwrap(); + let message = format!( + "live branch `{lock}` is the schema-apply lock of omnigraph 0.11.x, which a schema \ + apply did not release; the upgrade retires it once main is fenced, as the completed \ + apply would have, and keeps its head as a retired ref" + ); + let before = stored_files(locked.dir.path()); + let checked = check(&locked.root).await; + assert_eq!(checked.outcome, UpgradeOutcome::CheckPassed, "{checked:?}"); + assert_eq!(codes(&checked), ["schema_apply_lock_retired"]); + assert_eq!(checked.findings[0].message, message); + assert_eq!(checked.work.retired_refs, 3, "{:?}", checked.work); + assert_eq!(stored_files(locked.dir.path()), before); + let main = open(&locked.root, None).await.unwrap(); + assert!( + crate::branch_control::list_live_manifest_branch_contents(&main) + .await + .unwrap() + .contains_key(&lock), + "the check retires nothing" + ); + + let report = execute(&locked.root).await; + assert_eq!(report.outcome, UpgradeOutcome::Completed, "{report:?}"); + assert_eq!(codes(&report), ["schema_apply_lock_retired"]); + assert_eq!(report.findings[0].message, message); + assert_eq!(report.work, checked.work); + assert_converted(&locked).await; + + let mut released = source(9).await; + let lock = native(SCHEMA_APPLY_LOCK_BRANCH); + assert_eq!(released.history.fork(None, &lock).await.unwrap(), 4); + released.history.retire(&lock).await.unwrap(); + let report = execute(&released.root).await; + assert_eq!(report.outcome, UpgradeOutcome::Completed, "{report:?}"); + assert!(report.findings.is_empty(), "{report:?}"); + assert_eq!(report.work.retired_refs, 3); + assert_converted(&released).await; +} + +/// A lock beside the other leftovers of a killed 0.11.x schema apply says +/// nothing about the root: those leftovers keep their refusals. +#[tokio::test] +async fn a_schema_apply_lock_beside_staging_or_a_sidecar_stays_refused() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + let [pg, _, _] = root_schema::SCHEMA_FILENAMES; + let staged = locked(source(9).await).await; + std::fs::write(staged.dir.path().join(format!("{pg}.staging")), "staged").unwrap(); + let before = stored_files(staged.dir.path()); + for report in [check(&staged.root).await, execute(&staged.root).await] { + assert_source_recovery( + &report, + FROM_9, + &format!( + "schema object `{pg}.staging` at the graph root is the unfinished part of a \ + schema apply by omnigraph 0.11.x; it must be resolved before the storage \ + conversion" + ), + RELEASE_0_11, + "stop all writers, retain the backup, open the graph read-write with omnigraph \ + 0.11.x so it finishes or rolls back the schema apply, then rerun `omnigraph upgrade \ + --check`", + ); + } + assert_eq!(stored_files(staged.dir.path()), before); + + let with_sidecar = locked(source(9).await).await; + std::fs::create_dir(with_sidecar.dir.path().join("__recovery")).unwrap(); + std::fs::write(with_sidecar.dir.path().join("__recovery/01A.json"), "{}").unwrap(); + let before = stored_files(with_sidecar.dir.path()); + for report in [ + check(&with_sidecar.root).await, + execute(&with_sidecar.root).await, + ] { + assert_source_recovery( + &report, + FROM_9, + "1 recovery sidecar(s) under `__recovery/` (01A) must be resolved before the \ + storage conversion", + "the omnigraph build that wrote the recovery sidecars", + "stop all writers, retain the backup, open the graph read-write with the build that \ + wrote the sidecars so it finishes its recovery, then rerun `omnigraph upgrade \ + --check`", + ); + } + assert_eq!(stored_files(with_sidecar.dir.path()), before); +} + +/// A root schema object larger than the upgrade reads is a fact about the +/// contract: refused as unsupported, with the bound named. +#[tokio::test] +async fn an_over_bound_root_schema_object_refuses_as_unsupported_source() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + let source = source(9).await; + let [pg, _, _] = root_schema::SCHEMA_FILENAMES; + let written = std::fs::metadata(source.dir.path().join(pg)).unwrap().len(); + let bound = written - 1; + let few = Bounds { + root_object_bytes: bound, + ..Bounds::SERVED + }; + let before = stored_files(source.dir.path()); + for check in [true, false] { + let report = bounded(&source.root, check, few).await; + assert_eq!(report.outcome, UpgradeOutcome::CheckFailed, "{report:?}"); + assert_eq!(codes(&report), ["unsupported_source"]); + assert!( + report.findings[0].message.contains(&format!( + ": `{pg}` is {written} bytes, above the {bound} bytes one root schema object may \ + hold; restore the whole root from one backup (omnigraph 0.11.x refuses to open \ + this state as well)" + )), + "{report:?}" + ); + assert!(report.recovery.is_none() && report.last_durable_completed_boundary.is_none()); + } + assert_eq!(stored_files(source.dir.path()), before); + assert_eq!( + bounded(&source.root, true, Bounds::SERVED).await.outcome, + UpgradeOutcome::CheckPassed + ); +} + +/// Root objects restored from a backup older than a property-only schema +/// apply describe the right tables with the wrong columns: refused before +/// any write, naming the table and the column. +#[tokio::test] +async fn a_root_contract_whose_columns_the_tables_lack_refuses_before_any_write() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + let source = source(9).await; + let accepted: SchemaIR = + serde_json::from_str(&source.root_contract.as_ref().unwrap().ir).unwrap(); + let wider = + "node Person {\n name: String @key\n age: I64\n}\nnode Firm { name: String @key }\n"; + let shape = crate::db::schema_state::compile_schema_source(wider).unwrap(); + let evolved = omnigraph_compiler::resolve_schema_ir(&accepted, &shape) + .unwrap() + .schema_ir; + let contract = crate::db::schema_state::render_schema_contract(&evolved, wider).unwrap(); + write_root_schema(source.dir.path(), &contract); + let before = stored_files(source.dir.path()); + for report in [check(&source.root).await, execute(&source.root).await] { + assert_eq!(report.outcome, UpgradeOutcome::CheckFailed, "{report:?}"); + assert_eq!(codes(&report), ["unsupported_source"]); + assert_eq!( + report.findings[0].message, + "main's table node:Person has no column `age`, which the schema contract at the \ + graph root declares as Int64; restore the three root schema objects from the \ + backup taken with the tables, or rebuild the graph by export and load with \ + omnigraph 0.11.x" + ); + assert!(report.recovery.is_none() && report.last_durable_completed_boundary.is_none()); + assert_eq!(report.work, UpgradeWork::default()); + } + assert_eq!(stored_files(source.dir.path()), before); + let main = open_manifest_dataset(&source.root, None).await.unwrap(); assert!(!main.schema().metadata.contains_key(UPGRADE_PENDING_KEY)); - - let admitted = Bounds { - head_rows: gone_rows, - retired_refs: 1, - ..Bounds::SERVED - }; - let report = bounded(root, false, admitted).await; - assert_eq!(report.outcome, UpgradeOutcome::Completed, "{report:?}"); - assert_eq!(report.work.retired_refs, 1); } #[tokio::test] -async fn policy_denial_precedes_effects() { - struct DenySchemaApply; - impl omnigraph_policy::PolicyChecker for DenySchemaApply { - fn check( - &self, - action: omnigraph_policy::PolicyAction, - scope: &omnigraph_policy::ResourceScope, - actor: &str, - ) -> std::result::Result<(), omnigraph_policy::PolicyError> { - assert_eq!(action, omnigraph_policy::PolicyAction::SchemaApply); - assert!(matches!( - scope, - omnigraph_policy::ResourceScope::TargetBranch(_) - )); - assert_eq!(actor, "blocked-actor"); - Err(omnigraph_policy::PolicyError::Denied("test denial".into())) +async fn a_root_contract_that_does_not_describe_a_live_ref_refuses() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + let mut wider = source(9).await; + let feature = wider.feature.clone(); + let extra = table(901, "node:Extra"); + let registering = LegacyPublish { + tables: vec![extra.clone()], + pins: vec![pin(&extra, 1)], + commit: commit(8, None), + ..Default::default() + }; + assert_eq!( + wider + .history + .publish(Some(&feature), registering) + .await + .unwrap(), + 6 + ); + + let mut narrower = source(9).await; + let dropping = LegacyPublish { + drops: vec![(narrower.firm, 1)], + commit: commit(8, None), + ..Default::default() + }; + assert_eq!(narrower.history.publish(None, dropping).await.unwrap(), 5); + + for (source, message) in [ + ( + &wider, + format!( + "ref '{feature}' registers tables the schema contract at the graph root does \ + not describe: node:Extra" + ), + ), + ( + &narrower, + "main does not register table node:Firm, which the schema contract at the graph \ + root describes" + .to_string(), + ), + ] { + let before = stored_files(source.dir.path()); + for report in [check(&source.root).await, execute(&source.root).await] { + assert_eq!(report.outcome, UpgradeOutcome::CheckFailed, "{report:?}"); + assert_eq!(codes(&report), ["unsupported_source"]); + assert_eq!(report.findings[0].message, message); + assert!(report.recovery.is_none() && report.last_durable_completed_boundary.is_none()); + assert_eq!(report.work, UpgradeWork::default()); } + assert_eq!(stored_files(source.dir.path()), before); + let main = open_manifest_dataset(&source.root, None).await.unwrap(); + assert!(!main.schema().metadata.contains_key(UPGRADE_PENDING_KEY)); } - let source = stamp_13_source().await; - let before = stored_files(source.dir.path()); - let report = upgrade_storage_as( - &source.root, - UpgradeOptions::default(), - Some("blocked-actor"), - Some(&DenySchemaApply), +} + +/// Create the Lance table of `registration` at `root` as the 0.11.0 respelling +/// leaves it: version 1 stores `legacy` (`id`/`src`/`dst`), version 2 +/// overwrites it with `respelled`, and the table reaches `versions` versions. +async fn write_respelled_table( + root: &str, + registration: &TableRegistration, + legacy: Arc, + respelled: Arc, + versions: u64, +) { + write_table(root, registration, legacy, 1).await; + let uri = format!("{root}/{}", registration.table_path); + let params = WriteParams { + mode: WriteMode::Overwrite, + data_storage_version: Some(LanceFileVersion::V2_2), + ..Default::default() + }; + let batch = RecordBatch::new_empty(Arc::clone(&respelled)); + let mut dataset = Dataset::write( + RecordBatchIterator::new(vec![Ok(batch)], respelled), + &uri, + Some(params), ) .await .unwrap(); - assert_eq!(report.outcome, UpgradeOutcome::CheckFailed, "{report:?}"); - assert_eq!(codes(&report), ["preflight_failed"]); - assert!( - report.findings[0].message.contains("test denial"), - "{report:?}" - ); - assert_eq!(stored_files(source.dir.path()), before); + assert_eq!(dataset.version().version, 2); + for version in 3..=versions { + dataset + .update_schema_metadata([("fixture_version".to_string(), version.to_string())]) + .await + .unwrap(); + } + assert_eq!(dataset.version().version, versions); } -/// Main at another version than the inventory pinned is found before the -/// fence commit: nothing is written, so the report asks for no recovery. +/// Main as the released 0.11.0's default `upgrade` leaves a graph born at 6, as +/// far as the writer spells it (version 1 is written at 9 and restamped 6): the +/// stamp-6, 7 and 8 versions pin `id` tables, the respelling commit `__id` ones. #[tokio::test] -async fn main_moved_after_inventory_fails_preflight_without_recovery() { - let source = stamp_13_source().await; - let root = source.root.as_str(); - let before = stored_files(source.dir.path()); - let main = open(root, None).await.unwrap(); - let mut pinned = inventory(&main).await.unwrap().pop().unwrap(); - assert_eq!(pinned.native, None, "main is the last pinned ref"); - pinned.version -= 1; - - let mut report = UpgradeReport::started(root.to_string(), UpgradeOptions::default()); - let error = fence_main( - root, - Some(&pinned), - "{}".into(), - report.target_format, - &mut report, +async fn a_stamp_9_root_reached_from_6_by_the_released_upgrade_converts_its_retained_versions() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().to_str().unwrap().to_string(); + let (_, legacy_ir) = root_contract(8); + let (contract, ir) = root_contract(9); + let person = described(&ir, "Person", "node:Person"); + let firm = described(&ir, "Firm", "node:Firm"); + assert_eq!( + described(&legacy_ir, "Person", "node:Person").identity, + person.identity, + "the respelling keeps the table identity" + ); + let genesis = LegacyPublish { + tables: vec![person.clone(), firm.clone()], + pins: vec![pin(&person, 1), pin(&firm, 1)], + commit: commit(1, None), + ..Default::default() + }; + let mut history = LegacyHistory::create_stamped(&root, genesis, 9) + .await + .unwrap(); + assert_eq!(history.restamp_for_test(None, Some(6)).await.unwrap(), 2); + let earlier_route = |protocol: u32, from: u32, to: u32| { + format!( + r#"{{"protocol":{protocol},"attempt":"{}","source_format":{from},"target_format":{to},"graph_identity":"domain","branches":[{{"native":null,"version":2}}]}}"#, + id(9) + ) + }; + let mut version = 2; + for (protocol, from, to) in [(1, 6, 7), (2, 7, 8)] { + let intent = earlier_route(protocol, from, to); + assert_eq!( + history.fence_for_test(&intent, to).await.unwrap(), + version + 1 + ); + assert_eq!( + history.restamp_for_test(None, Some(to)).await.unwrap(), + version + 2 + ); + assert_eq!(history.activate_for_test().await.unwrap(), version + 3); + version += 3; + } + let pinning = |number: u128, table_version: u64| LegacyPublish { + pins: vec![pin(&person, table_version)], + commit: commit(number, None), + ..Default::default() + }; + let stamp_advance = LegacyPublish::default(); + assert_eq!(history.publish(None, stamp_advance).await.unwrap(), 9); + let respelling = pinning(2, 2); + assert_eq!(history.publish(None, respelling).await.unwrap(), 10); + assert_eq!(history.publish(None, pinning(3, 3)).await.unwrap(), 11); + let feature = native("feature"); + assert_eq!(history.fork(None, &feature).await.unwrap(), 11); + assert_eq!( + history + .publish(Some(&feature), pinning(4, 4)) + .await + .unwrap(), + 12 + ); + write_root_schema(dir.path(), &contract); + write_respelled_table( + &root, + &person, + declared_columns(&legacy_ir, "node:Person"), + declared_columns(&ir, "node:Person"), + 4, ) - .await - .unwrap_err(); - report.failed(&error); + .await; + write_table(&root, &firm, declared_columns(&ir, "node:Firm"), 1).await; - assert_eq!(report.outcome, UpgradeOutcome::CheckFailed, "{report:?}"); - assert_eq!(codes(&report), ["preflight_failed"]); - assert!( - report.findings[0] - .message - .contains("a writer moved it during the offline upgrade"), - "{report:?}" + let checked = check(&root).await; + assert_eq!(checked.outcome, UpgradeOutcome::CheckPassed, "{checked:?}"); + let report = execute(&root).await; + assert_eq!(report.outcome, UpgradeOutcome::Completed, "{report:?}"); + assert!(report.findings.is_empty() && report.recovery.is_none()); + assert_eq!(report.route, [FROM_9]); + assert_eq!(report.work, checked.work); + assert_eq!( + ( + report.work.live_refs, + report.work.retired_refs, + report.work.legacy_commits, + report.work.table_opens + ), + (2, 0, 4, 2), + "{:?}", + report.work ); - assert!(report.recovery.is_none(), "{report:?}"); - assert_eq!(report.last_durable_completed_boundary, None); - assert_eq!(stored_files(source.dir.path()), before); + let both = vec!["node:Firm".to_string(), "node:Person".to_string()]; assert_eq!( - execute(root).await.outcome, - UpgradeOutcome::Completed, - "the unfenced root still converts" + served(&root, None).await, + (both.clone(), vec![("main".to_string(), id(3))]) ); + assert_eq!( + served(&root, Some(&feature)).await, + (both, vec![("feature".to_string(), id(4))]) + ); + for (branch, version, head, person_version) in [ + (None, 2, Some(id(1)), 1), + (None, 5, Some(id(1)), 1), + (None, 8, Some(id(1)), 1), + (None, 9, Some(id(1)), 1), + (None, 10, Some(id(2)), 2), + (None, 11, Some(id(3)), 3), + (Some("feature"), 12, Some(id(4)), 4), + ] { + assert_eq!( + served_at(&root, branch, version).await, + (version, head, person_version), + "{branch:?} at {version}" + ); + } + assert_eq!(execute(&root).await.outcome, UpgradeOutcome::AlreadyCurrent); } const PERSON: &str = "node:Person"; @@ -1003,7 +2219,7 @@ fn replayed(before: Option<&ManifestState>, after: &ManifestState, number: u128) struct EngineSource { _dir: tempfile::TempDir, root: String, - history: Stamp13History, + history: LegacyHistory, feature: String, /// The detached `Person` version commit 2 pins, as the collector roots it. a_root: u64, @@ -1060,7 +2276,7 @@ async fn engine_source() -> EngineSource { contract: Some(contract.clone()), ..replayed(None, &genesis, 1) }; - let mut history = Stamp13History::create(&root, created).await.unwrap(); + let mut history = LegacyHistory::create(&root, created).await.unwrap(); let second = replayed(Some(&genesis), &forked, 2); assert_eq!(history.publish(None, second).await.unwrap(), 2); assert_eq!(history.fork(None, &feature).await.unwrap(), 2); @@ -1507,7 +2723,14 @@ async fn served_at(root: &str, branch: Option<&str>, version: u64) -> (u64, Opti async fn numeric_snapshot_below_upgrade() { #[cfg(feature = "failpoints")] let _scenario = crate::seams::FailScenario::setup(); - let source = stamp_13_source().await; + for (stamp, archived_source) in [(9, ROOT_SCHEMA), (13, "node Person {}")] { + numeric_snapshot_below(source(stamp).await, archived_source).await; + } +} + +/// `archived_source` is the schema source the bookkeeping version 3 of main +/// is served under: its own contract row at stamp 13, the root contract below. +async fn numeric_snapshot_below(source: Source, archived_source: &str) { assert_eq!( execute(&source.root).await.outcome, UpgradeOutcome::Completed @@ -1521,13 +2744,6 @@ async fn numeric_snapshot_below_upgrade() { .unwrap(); let on_main = publish(root, None, 10, HISTORY_RELEASE_BYTES).await; let on_late = publish(root, Some("late"), 11, HISTORY_RELEASE_BYTES).await; - assert_eq!( - ( - on_main.graph_manifest_version, - on_late.graph_manifest_version - ), - (8, 8) - ); let head = |row: &GraphLineageRow| Some(row.graph_commit_id.clone()); for (branch, version, expected_head, person) in [ (None, 1, Some(id(1)), 1), @@ -1574,18 +2790,23 @@ async fn numeric_snapshot_below_upgrade() { .await .unwrap() .source, - "node Person {}" + archived_source ); } #[tokio::test] async fn fork_of_deleted_branch_serves_inherited_version() { + #[cfg(feature = "failpoints")] + let _scenario = crate::seams::FailScenario::setup(); + for stamp in [9, 13] { + fork_of_deleted_branch(source(stamp).await).await; + } +} + +async fn fork_of_deleted_branch(source: Source) { use crate::db::manifest::commit_graph::{ CommitGraph, HistoryCache, graph_commit_from_manifest_row, }; - #[cfg(feature = "failpoints")] - let _scenario = crate::seams::FailScenario::setup(); - let source = stamp_13_source().await; assert_eq!( execute(&source.root).await.outcome, UpgradeOutcome::Completed @@ -1672,10 +2893,15 @@ async fn fork_of_deleted_branch_serves_inherited_version() { #[tokio::test] async fn retired_commit_graphs_serve_legacy_heads() { - use crate::db::manifest::commit_graph::HistoryCache; #[cfg(feature = "failpoints")] let _scenario = crate::seams::FailScenario::setup(); - let source = stamp_13_source().await; + for stamp in [9, 13] { + retired_commit_graphs(source(stamp).await).await; + } +} + +async fn retired_commit_graphs(source: Source) { + use crate::db::manifest::commit_graph::HistoryCache; assert_eq!( execute(&source.root).await.outcome, UpgradeOutcome::Completed @@ -1710,33 +2936,43 @@ async fn retired_commit_graphs_serve_legacy_heads() { async fn named_and_fresh_fork_conversions_pass_equivalence() { #[cfg(feature = "failpoints")] let _scenario = crate::seams::FailScenario::setup(); - let source = stamp_13_source().await; + for stamp in [8, 9, 13] { + fork_conversions_pass_equivalence(source(stamp).await).await; + } +} + +/// Every converted ref serves the pins of its source head. A pin converted +/// from stamp 8 or 9 names its own table version as its last linear one; a +/// stamp-13 pin keeps the metadata it had. +async fn fork_conversions_pass_equivalence(source: Source) { + let stamp = source.stamp; let report = execute(&source.root).await; assert_eq!(report.outcome, UpgradeOutcome::Completed, "{report:?}"); assert_eq!(report.work.live_refs, 4); - let firm = TableIdentity::new(2, 1).unwrap(); - let person = TableIdentity::new(1, 1).unwrap(); - let temp = TableIdentity::new(3, 1).unwrap(); + let (firm, person) = (source.firm, source.person); + let temp: Vec<(TableIdentity, u64)> = match stamp { + 13 => vec![(source.temp, 1)], + _ => Vec::new(), + }; + let on_branch = |person_version: u64| { + let mut pins = vec![(firm, 1), (person, person_version)]; + pins.extend(temp.clone()); + pins + }; for (native, head, pins) in [ (None, Some(("main", 4)), vec![(firm, 1), (person, 7)]), ( Some(source.feature.as_str()), Some(("feature", 6)), - vec![(firm, 1), (person, 8), (temp, 1)], - ), - ( - Some(source.fresh.as_str()), - None, - vec![(firm, 1), (person, 6), (temp, 1)], - ), - ( - Some(source.child.as_str()), - None, - vec![(firm, 1), (person, 5), (temp, 1)], + on_branch(8), ), + (Some(source.fresh.as_str()), None, on_branch(6)), + (Some(source.child.as_str()), None, on_branch(5)), ] { let state = state_of(&source.root, native).await; - let mut served: Vec<(TableIdentity, u64, u64)> = state + let last_linear = |version: u64| (stamp != 13).then_some(version); + type Pin = (TableIdentity, u64, u64, Option, Option); + let mut served: Vec = state .entries .iter() .map(|entry| { @@ -1744,16 +2980,20 @@ async fn named_and_fresh_fork_conversions_pass_equivalence() { entry.identity, entry.published_dataset_version, entry.entity_count, + entry.version_metadata.last_linear_version(), + entry.version_metadata.staged_version(), ) }) .collect(); served.sort(); - let mut expected: Vec<(TableIdentity, u64, u64)> = pins + let mut expected: Vec = pins .into_iter() - .map(|(identity, version)| (identity, version, version * 10)) + .map(|(identity, version)| { + (identity, version, version * 10, last_linear(version), None) + }) .collect(); expected.sort(); - assert_eq!(served, expected, "{native:?}"); + assert_eq!(served, expected, "v{stamp} {native:?}"); assert_eq!( state.graph_heads, head.map(|(branch, number)| (branch.to_string(), id(number))) @@ -1766,10 +3006,15 @@ async fn named_and_fresh_fork_conversions_pass_equivalence() { #[tokio::test] async fn fork_head_and_writer_head_release_equal_copies() { - use crate::db::manifest::history::{read_lineage, read_records_of}; #[cfg(feature = "failpoints")] let _scenario = crate::seams::FailScenario::setup(); - let source = stamp_13_source().await; + for stamp in [9, 13] { + fork_and_writer_heads_release(source(stamp).await).await; + } +} + +async fn fork_and_writer_heads_release(source: Source) { + use crate::db::manifest::history::{read_lineage, read_records_of}; assert_eq!( execute(&source.root).await.outcome, UpgradeOutcome::Completed @@ -1841,10 +3086,22 @@ mod interrupted { use super::*; use crate::seams::{DecideSeam, FailScenario, catalog}; - /// Run the upgrade of a new source with `seam` failing at its crossing - /// `occurrence`; `None` when the run completed without reaching it. + /// Run the upgrade of a new stamp-13 source with `seam` failing at its + /// crossing `occurrence`; `None` when the run completed without reaching it. async fn interrupt(seam: &'static DecideSeam, occurrence: u64) -> Option { - let source = stamp_13_source().await; + interrupt_from(13, false, seam, occurrence).await + } + + /// [`interrupt`] over a new source stamped `stamp`, with the lock of a + /// killed 0.11.x schema apply when `lock`. + async fn interrupt_from( + stamp: u32, + lock: bool, + seam: &'static DecideSeam, + occurrence: u64, + ) -> Option { + let source = source(stamp).await; + let source = if lock { locked(source).await } else { source }; let report = { let _fault = seam.fire_once_at(occurrence); execute(&source.root).await @@ -1859,8 +3116,13 @@ mod interrupted { "{} at {occurrence}: {report:?}", seam.name() ); - assert_eq!(codes(&report), ["upgrade_interrupted"], "{report:?}"); - assert_remedy(&report, RERUN); + let expected_codes: &[&str] = if lock { + &["schema_apply_lock_retired", "upgrade_interrupted"] + } else { + &["upgrade_interrupted"] + }; + assert_eq!(codes(&report), expected_codes, "{report:?}"); + assert_remedy(&report, route_name(stamp), RERUN); assert!(report.last_durable_completed_boundary.is_some()); Some(source) } @@ -1868,17 +3130,56 @@ mod interrupted { #[tokio::test] async fn interruption_boundaries_retry_without_mixed_visibility() { let _scenario = FailScenario::setup(); - for (seam, crossings) in [ - (&catalog::UPGRADE_AFTER_FENCE, 1), - (&catalog::UPGRADE_BETWEEN_LEGACY_FILES, 5), - (&catalog::UPGRADE_AFTER_LEGACY, 1), - (&catalog::UPGRADE_AFTER_STAGE, 4), - (&catalog::UPGRADE_AFTER_BRANCH, 4), - (&catalog::UPGRADE_BEFORE_ACTIVATION, 1), - (&catalog::UPGRADE_AFTER_ACTIVATION, 1), - ] { + interruption_boundaries_retry(13, false, [1, 5, 1, 4, 4, 1, 1], (6, 3, 1, 1)).await; + } + + /// The seams and the work of the stamp-9 source are counted on their own: + /// a rerun reads the contract from the archive the fence bound, not from + /// the root objects, which the interrupted run left as it found them. + #[tokio::test] + async fn interruption_boundaries_retry_from_stamp_9() { + let _scenario = FailScenario::setup(); + interruption_boundaries_retry(9, false, [1, 5, 1, 4, 4, 1, 1], (6, 3, 1, 1)).await; + } + + /// A stamp-9 source with the lock of a killed schema apply crosses the same + /// seams: a rerun stopped before the retirement retires the lock, one + /// stopped after it finds it retired, and the lock adds no legacy object. + #[tokio::test] + async fn interruption_boundaries_retry_from_stamp_9_with_a_schema_apply_lock() { + let _scenario = FailScenario::setup(); + interruption_boundaries_retry(9, true, [1, 5, 1, 4, 4, 1, 1], (6, 3, 1, 1)).await; + } + + /// The stamp-8 source, whose root contract spells the legacy system + /// columns, crosses the same seams with the same work as stamp 9. + #[tokio::test] + async fn interruption_boundaries_retry_from_stamp_8() { + let _scenario = FailScenario::setup(); + interruption_boundaries_retry(8, false, [1, 5, 1, 4, 4, 1, 1], (6, 3, 1, 1)).await; + } + + /// Stop the upgrade of a source stamped `stamp` at each of the `crossings` + /// of every seam: no reader opens the graph, and the rerun completes with + /// `work` (commits, data files, id shards, writer shards). + async fn interruption_boundaries_retry( + stamp: u32, + lock: bool, + crossings: [u64; 7], + work: (u64, u64, u64, u64), + ) { + let seams = [ + &catalog::UPGRADE_AFTER_FENCE, + &catalog::UPGRADE_BETWEEN_LEGACY_FILES, + &catalog::UPGRADE_AFTER_LEGACY, + &catalog::UPGRADE_AFTER_STAGE, + &catalog::UPGRADE_AFTER_BRANCH, + &catalog::UPGRADE_BEFORE_ACTIVATION, + &catalog::UPGRADE_AFTER_ACTIVATION, + ]; + for (seam, crossings) in seams.into_iter().zip(crossings) { let mut occurrence = 1; - while let Some(source) = interrupt(seam, occurrence).await { + while let Some(source) = interrupt_from(stamp, lock, seam, occurrence).await { let context = format!("{} at {occurrence}", seam.name()); let root = source.root.as_str(); let main = open(root, None).await.unwrap(); @@ -1899,7 +3200,7 @@ mod interrupted { "{context}: {refusal}" ); assert_eq!(codes(&checked), ["pending_upgrade"], "{context}"); - assert_remedy(&checked, RERUN); + assert_remedy(&checked, route_name(stamp), RERUN); assert!( refusal.contains(&checked.findings[0].message), "{context}: {checked:?}" @@ -1910,9 +3211,17 @@ mod interrupted { let resumed = execute(root).await; assert!(resumed.success(), "{context}: {resumed:?}"); - assert!(resumed.findings.is_empty() && resumed.recovery.is_none()); + let expected_codes: &[&str] = if lock && !activated { + &["schema_apply_lock_retired"] + } else { + &[] + }; + assert_eq!(codes(&resumed), expected_codes, "{context}: {resumed:?}"); + assert!(resumed.recovery.is_none(), "{context}: {resumed:?}"); if !activated { assert_eq!(resumed.outcome, UpgradeOutcome::Completed, "{context}"); + assert_eq!(resumed.observed_format, Some(14), "{context}"); + assert_eq!(resumed.route, [route_name(stamp)], "{context}"); assert_eq!( resumed.work.census_reads > 0, before_legacy, @@ -1926,7 +3235,7 @@ mod interrupted { resumed.work.id_shards, resumed.work.writer_shards ), - (6, 3, 1, 1), + work, "{context}" ); } @@ -1938,10 +3247,142 @@ mod interrupted { ); occurrence += 1; } - assert_eq!(occurrence - 1, crossings, "{}", seam.name()); + assert_eq!(occurrence - 1, crossings, "v{stamp} {}", seam.name()); + } + } + + /// Once main is fenced, the contract of an 8/9 source is the archive the + /// intent binds: every refusal the root objects can raise before the fence + /// (absent, changed, respelled, staged, staged beside a lock) is unreachable. + #[tokio::test] + async fn a_fenced_stamp_8_or_9_rerun_reads_the_archived_contract_not_the_root_objects() { + let _scenario = FailScenario::setup(); + type Damage = fn(&std::path::Path); + let remove_pg: Damage = + |dir| std::fs::remove_file(dir.join(root_schema::SCHEMA_SOURCE_FILENAME)).unwrap(); + let widen_pg: Damage = |dir| { + std::fs::write( + dir.join(root_schema::SCHEMA_SOURCE_FILENAME), + "node Person {\n name: String @key\n age: I64\n}\nnode Firm { name: String @key }\n", + ) + .unwrap() + }; + let corrupt_ir: Damage = + |dir| std::fs::write(dir.join(root_schema::SCHEMA_IR_FILENAME), "{").unwrap(); + let stage_pg: Damage = |dir| { + std::fs::write( + dir.join(format!("{}.staging", root_schema::SCHEMA_SOURCE_FILENAME)), + "staged", + ) + .unwrap() + }; + let respell_root: Damage = |dir| write_root_schema(dir, &root_contract(9).0); + let cases: [(u32, bool, Damage); 6] = [ + (9, false, remove_pg), + (9, false, widen_pg), + (9, false, corrupt_ir), + (9, false, stage_pg), + (9, true, stage_pg), + (8, false, respell_root), + ]; + for (stamp, lock, damage) in cases { + let source = interrupt_from(stamp, lock, &catalog::UPGRADE_AFTER_FENCE, 1) + .await + .unwrap(); + damage(source.dir.path()); + let damaged = root_objects(source.dir.path()); + let context = format!("v{stamp} lock {lock}"); + let resumed = execute(&source.root).await; + assert_eq!( + resumed.outcome, + UpgradeOutcome::Completed, + "{context}: {resumed:?}" + ); + let expected_codes: &[&str] = if lock { + &["schema_apply_lock_retired"] + } else { + &[] + }; + assert_eq!(codes(&resumed), expected_codes, "{context}: {resumed:?}"); + assert!(resumed.recovery.is_none(), "{context}: {resumed:?}"); + assert!( + resumed.work.census_reads > 0, + "{context}: {:?}", + resumed.work + ); + assert_eq!(resumed.work.table_opens, 0, "{context}: {:?}", resumed.work); + assert_eq!(resumed.route, [route_name(stamp)], "{context}"); + let main = open(&source.root, None).await.unwrap(); + let (_, converted) = read_converted_state(&main).await.unwrap(); + assert_eq!(Some(&converted), source.root_contract.as_ref(), "{context}"); + assert_eq!( + root_objects(source.dir.path()), + damaged, + "{context}: the rerun reads and writes no root object" + ); + if let Some(lock) = &source.lock { + assert_lock_retired(&source.root, lock).await; + } + assert_eq!( + execute(&source.root).await.outcome, + UpgradeOutcome::AlreadyCurrent, + "{context}" + ); } } + /// The archive the intent binds is the one thing a fenced stamp-9 rerun + /// needs beside `__manifest`: without it the run asks for the backup, not + /// for a rerun that cannot succeed. + #[tokio::test] + async fn a_fenced_stamp_9_rerun_without_its_archived_contract_asks_for_the_backup() { + let _scenario = FailScenario::setup(); + let source = interrupt_from(9, false, &catalog::UPGRADE_AFTER_FENCE, 1) + .await + .unwrap(); + let archives: Vec = stored_files(source.dir.path()) + .into_keys() + .filter(|name| name.starts_with("__history/schemas/")) + .collect(); + let [archive] = archives.as_slice() else { + panic!("one archived contract, found {archives:?}"); + }; + let digest = intent_from(&open(&source.root, None).await.unwrap()) + .unwrap() + .unwrap() + .schema_contract + .unwrap() + .content_sha256; + assert!(archive.contains(&digest), "{archive} names {digest}"); + let path = source.dir.path().join(archive); + let written = std::fs::read(&path).unwrap(); + std::fs::remove_file(&path).unwrap(); + let before = stored_files(source.dir.path()); + let report = execute(&source.root).await; + assert_eq!( + report.outcome, + UpgradeOutcome::RecoveryRequired, + "{report:?}" + ); + assert_eq!(codes(&report), ["legacy_objects_differ"]); + assert!( + report.findings[0].message.starts_with(&format!( + "the schema content `{digest}` the upgrade intent binds cannot be read from \ + `__history/schemas/`: " + )) && report.findings[0] + .message + .ends_with("; restore the whole root from the backup taken before the attempt"), + "{report:?}" + ); + assert_remedy(&report, FROM_9, FENCING_EXECUTABLE); + assert_eq!(stored_files(source.dir.path()), before); + + std::fs::write(&path, &written).unwrap(); + let resumed = execute(&source.root).await; + assert_eq!(resumed.outcome, UpgradeOutcome::Completed, "{resumed:?}"); + assert_converted(&source).await; + } + #[tokio::test] async fn partial_legacy_write_is_completed_on_retry() { let _scenario = FailScenario::setup(); @@ -2001,7 +3442,7 @@ mod interrupted { report.findings[0].message, "the graph holds 2 retired refs, one storage upgrade reads at most 1" ); - assert_remedy(&report, FENCING_EXECUTABLE); + assert_remedy(&report, FROM_13, FENCING_EXECUTABLE); let small = Bounds { head_rows: 0, ..Bounds::SERVED @@ -2015,7 +3456,7 @@ mod interrupted { && !message.contains(RETIRED_REMEDY), "{report:?}" ); - assert_remedy(&report, FENCING_EXECUTABLE); + assert_remedy(&report, FROM_13, FENCING_EXECUTABLE); assert_eq!(stored_files(source.dir.path()), before); assert_eq!(execute(root).await.outcome, UpgradeOutcome::Completed); assert_converted(&source).await; @@ -2049,7 +3490,7 @@ mod interrupted { ), "{report:?}" ); - assert_remedy(&report, PRESERVE_AND_CHECK); + assert_remedy(&report, UNROUTED, PRESERVE_AND_CHECK); } assert_eq!(stored_files(source.dir.path()), before); } @@ -2098,7 +3539,7 @@ mod interrupted { )), "{message}" ); - assert_remedy(&report, FENCING_EXECUTABLE); + assert_remedy(&report, FROM_13, FENCING_EXECUTABLE); assert_eq!( report.last_durable_completed_boundary.as_deref(), Some("source_fenced") @@ -2128,7 +3569,7 @@ mod interrupted { .contains("id shard 0 does not hash to the digest the directory lists"), "{report:?}" ); - assert_remedy(&report, FENCING_EXECUTABLE); + assert_remedy(&report, FROM_13, FENCING_EXECUTABLE); assert_eq!(stored_files(source.dir.path()), before); std::fs::write(&shard, &written).unwrap(); @@ -2155,7 +3596,7 @@ mod interrupted { std::fs::set_permissions(&shard, readable).unwrap(); assert_eq!(report.outcome, UpgradeOutcome::RecoveryRequired); assert_eq!(codes(&report), ["upgrade_interrupted"], "{report:?}"); - assert_remedy(&report, RERUN); + assert_remedy(&report, FROM_13, RERUN); assert_eq!(stored_files(source.dir.path()), before); let resumed = execute(&source.root).await; diff --git a/docs/dev/blob.md b/docs/dev/blob.md index ebb4d30bc..4231cc41b 100644 --- a/docs/dev/blob.md +++ b/docs/dev/blob.md @@ -84,7 +84,7 @@ Export emits managed values as base64 and whole-object external values as URI de `optimize` compacts Blob-bearing tables. The pinned Lance release must pass the substrate guard proving null, empty, non-empty, neighboring payloads, stable row IDs, and range reads survive fragment compaction. Lance materializes every managed payload of a compaction scanner batch, so `stage_compaction` executes the plan's tasks itself and derives each task's batch size from a descriptor-only scan of that task's fragments: the largest row's managed Blob bytes, summed over its Blob columns (external descriptors are carried unread and count nothing), so one batch materializes at most `COMPACTION_BLOB_BATCH_BYTES` of managed payload. A row over 16 MiB puts its whole task on one-row batches, and a row over the budget is materialized whole. Every task of a Blob table gets the derived size explicitly (1 to 8192 rows, or a caller's smaller one), so it takes precedence over `LANCE_DEFAULT_BATCH_SIZE`, including when the variable is smaller. All tasks are sized before any executes, so when a fragment being compacted holds a descriptor the decoder refuses, sizing can refuse the compaction with a Blob integrity error before that table is rewritten, and nothing is committed; a table with nothing to compact is not scanned. This bounds payload per batch, not heap: Lance's writer copies inline payloads into its prepared arrays while the batch is held; see the compaction fence in [lance.md](lance.md#current-compatibility-fences). `cleanup` can reclaim old managed bytes with their dataset versions; callers must quiesce long-lived readers before destructive GC. -`omnigraph upgrade` reads no Blob descriptor and contacts no Blob store: its one in-place route (a standalone stamp-13 graph to 14) validates the source's commit history and rewrites no table row (see [versioning.md](versioning.md#storage-upgrade-support-matrix)). A graph at any other storage format is rebuilt by export and load (see [versioning.md](versioning.md#rebuild)), and the export rule above is what governs its external descriptors. +`omnigraph upgrade` reads no Blob descriptor and contacts no Blob store: its in-place conversion (a standalone graph at stamp 8, 9 or 13 to 14) validates the source's commit history and rewrites no table row (see [versioning.md](versioning.md#storage-upgrade-support-matrix)). A graph at any other storage format is rebuilt by export and load (see [versioning.md](versioning.md#rebuild)), and the export rule above is what governs its external descriptors. ## Test owners diff --git a/docs/dev/ci.md b/docs/dev/ci.md index 3073e869c..c5555f761 100644 --- a/docs/dev/ci.md +++ b/docs/dev/ci.md @@ -103,15 +103,33 @@ runs on every change and in the merge queue. It builds the genuine stamp-13 CLI from the immutable `STAMP_13_SOURCE_COMMIT` in a detached worktree into the job's target directory, copies the binary out, cleans the path-package artifacts through both manifests (the predecessor and current packages share -names and versions) and exports `OMNIGRAPH_V13_BIN`. With -`OMNIGRAPH_REQUIRE_STORAGE_UPGRADE_TESTS=1` a missing binary fails -`genuine_v13_storage_upgrade_preserves_history` instead of skipping it. Four +names and versions) and exports `OMNIGRAPH_V13_BIN`. The step +`Install released v0.10.0 and v0.11.0 CLIs` downloads the two releases with +`scripts/install.sh` and exports `OMNIGRAPH_V6_BIN` and `OMNIGRAPH_V011_BIN`: +0.11.0 writes stamp 9, and 0.10.0 followed by the 0.11.0 `upgrade` gives the +stamp-8 source (`--to-format 8`) and the stamp-9 source its default target +leaves. The step trusts the release assets exactly as the `test` job's +v0.9.0 and v0.10.0 installs do: `install.sh` downloads the archive and its +`.sha256` from the same GitHub release and verifies the one against the other +before extracting; no digest is pinned in the workflow. Each download gets +three attempts (`install_release`), the only retry in the job; a failed +attempt leaves no binary behind. The job's added runtime (two downloads and +the five journeys) is UNVERIFIED against its `timeout-minutes: 90` until the +first hosted run; the timeout was not changed. With +`OMNIGRAPH_REQUIRE_STORAGE_UPGRADE_TESTS=1` a missing binary fails the five +genuine journeys (`genuine_v13_storage_upgrade_preserves_history`, +`genuine_v0_11_0_storage_upgrade_preserves_history`, +`genuine_v0_11_0_storage_upgrade_after_predecessor_cleanup`, +`genuine_v0_10_0_to_stamp_8_storage_upgrade_preserves_history`, +`genuine_v0_10_0_to_stamp_9_by_default_storage_upgrade_preserves_history`) +instead of skipping them; the v6 format fence in the `test` job keeps its own +resolver and skips, never panics, without `OMNIGRAPH_V6_BIN`. Four scopes then run, each checked against its log by `scripts/check-storage-upgrade-ci.py --check-log`: the `storage_upgrade` cases of `crossversion_upgrade.rs`, the engine `db::upgrade::tests`, `lance_version_columns` and `forbidden_apis`. The script's `--self-test` pins -the scopes, the predecessor build script (compared exactly) and the required -case names, so removing or altering one fails `Check Workflow Action Pins` and +the scopes, the predecessor build and install scripts (compared exactly) and +the required case names, so removing or altering one fails `Check Workflow Action Pins` and this job. The 90-minute budget covers two cold builds into one target directory: the predecessor CLI alone (one package, one bin, no test features), then the current tree's test targets, whose dev-dependency features (lance-io diff --git a/docs/dev/invariants.md b/docs/dev/invariants.md index 9cbe9e68a..58fa1223c 100644 --- a/docs/dev/invariants.md +++ b/docs/dev/invariants.md @@ -136,7 +136,7 @@ different: - maintained parallel truth or cold full-history reconstruction per request. Cases an RFC has argued: the flat locator objects and legacy data files under -`__history/legacy/`, written once by the stamp-13 storage upgrade, in the +`__history/legacy/`, written once by the storage upgrade (from stamp 8, 9 or 13), in the [RFC 0068 amendment](../rfcs/0068-graph-commit-record.md#why-this-is-not-a-deny-list-shape). ## Current support boundaries diff --git a/docs/dev/recovery.md b/docs/dev/recovery.md index f628363de..247155ad6 100644 --- a/docs/dev/recovery.md +++ b/docs/dev/recovery.md @@ -79,15 +79,45 @@ do not introduce historical schema-language semantics. See ## Pending storage conversion -The offline stamp-13 to stamp-14 upgrade sets `UPGRADE_PENDING_KEY` +The offline upgrade to stamp 14, from stamp 8, 9 or 13, sets `UPGRADE_PENDING_KEY` (`omnigraph:storage_upgrade_pending`) on main's `__manifest` from its fence -commit to its activation commit; the value is the `UpgradeIntent`. Ordinary +commit to its activation commit; the value is the `UpgradeIntent`, whose +`source_format` names the route. Ordinary open, read-only included, refuses a graph that carries it with -`recovery_guidance`. Only main is fenced: a stamp-13 process opened before the +`recovery_guidance`. Only main is fenced: a source-build process opened before the fence can still publish on an unconverted named ref, which is why the upgrade requires every process stopped and refuses a moved ref (`verify_source_head`) instead of adopting it. Never remove the key by hand. +A stamp-8 or stamp-9 source has one stop of its own before the fence, +`source_recovery_required` and writing nothing: a `.staging` copy of +`_schema.pg`, `_schema.ir.json` or `__schema_state.json` at the graph root, +the unfinished schema apply of release 0.11.x, or a recovery sidecar, each +with or without a live `__schema_apply_lock__` ref beside it. A read-write +open by 0.11.x (`omnigraph snapshot` is one) removes the sidecar and finishes +or rolls back an apply whose live and staging schemas imply different table +sets; when both imply the same table set (a property-only apply) 0.11.x +refuses too and leaves the choice between the live and the staging schema to +the operator. 0.11.x releases the lock only in the process that took it, so +the lock outlives that open. A live `__schema_apply_lock__` ref on an +otherwise clean root (no `.staging` copy, no recovery sidecar) is the lock of +an apply that was killed before releasing it, or whose release failed after +the apply completed; every such kill point leaves a consistent root, so the +upgrade reports it as `schema_apply_lock_retired` and, +once main is fenced, retires the ref with the same +`retire_branch_recoverably` a 0.11.x branch delete runs, so the converted +graph does not hold it as a live ref. The retirement is idempotent: a rerun +stopped before it retires the lock, one stopped after it finds it retired, +and the census reads the lock as a retired ref either way. The upgrade +deletes no root object, but a fenced rerun does not read them: the fence +binds the digest of the contract archived under `__history/schemas/` before +it, and the rerun reads that archive, so a root object changed or removed +after the fence does not strand the graph (an unreadable archive is +`legacy_objects_differ`, resolved from the backup). The route also compares +the contract's columns with each registered table's Lance columns before any +write, so root objects restored from a backup older than a property-only +apply are `unsupported_source`, not a contract of every commit. + The upgrade owns its own recovery; no other writer resumes it. Every durable effect is either create-only under `__history/` (`put_if_absent`, equal bytes accepted again) or one zero-retry Lance commit on one ref, and the intent diff --git a/docs/dev/testing.md b/docs/dev/testing.md index 05eed77e4..ba9ff234c 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -167,30 +167,67 @@ The older seams work the same way with released binaries: `OMNIGRAPH_OLD_BIN` (0 The separate `Storage Upgrade Compatibility` CI job requires the `storage_upgrade` cases of `crossversion_upgrade.rs` (the report on a fresh -graph, the cluster-path refusal and -`genuine_v13_storage_upgrade_preserves_history`), the engine +graph, the cluster-path refusal and the five genuine journeys: +`genuine_v13_storage_upgrade_preserves_history`, +`genuine_v0_11_0_storage_upgrade_preserves_history`, +`genuine_v0_11_0_storage_upgrade_after_predecessor_cleanup`, +`genuine_v0_10_0_to_stamp_8_storage_upgrade_preserves_history` and +`genuine_v0_10_0_to_stamp_9_by_default_storage_upgrade_preserves_history`), the engine `db::upgrade::tests`, `lance_version_columns` and `forbidden_apis`. Missing cases, empty runs and skipped required cases fail the job. -The genuine journey needs the stamp-13 CLI: the job builds it from the commit +The stamp-13 journey needs the stamp-13 CLI: the job builds it from the commit `ci.yml` pins as `STAMP_13_SOURCE_COMMIT` and exports `OMNIGRAPH_V13_BIN`. It proves its predecessor by behaviour (that binary's `snapshot --json` reports `internal_schema_version` 13 and the current `upgrade --check` observes 13), builds branches, a merge, a deleted branch, a recreated one and a fork of a deleted branch with the old binary, upgrades, and compares commit history, -rows, `cleanup` and a backup restore after it. Without the binary it prints a -skip line locally and panics when `OMNIGRAPH_REQUIRE_STORAGE_UPGRADE_TESTS=1`, -as CI sets. To run the crossversion scope locally, build the predecessor as -for the v5 fence above and pass it: +rows, `cleanup` and a backup restore after it. + +The 0.11.x journeys need the released CLIs, which the job installs: +`OMNIGRAPH_V011_BIN` (0.11.0, writes stamp 9) and `OMNIGRAPH_V6_BIN` (0.10.0). +`genuine_v0_11_0_storage_upgrade_preserves_history` runs the same script as +the stamp-13 journey plus a schema apply (a new type and a nullable property +on the existing `Doc`, with a commit before and one after it) and an +`optimize` with the old binary, asserts that the three root schema objects +are byte-identical after `completed`, and compares what the two commits +around the apply answer for the added property before and after the upgrade. +`genuine_v0_11_0_storage_upgrade_after_predecessor_cleanup` +runs the old binary's `cleanup --keep 1` before the upgrade and asserts the +cleanup order of the upgrade guide after it; a commit the old binary refused +as reclaimed must be refused after the upgrade, any other failure of a read +fails the journey, and at least one commit must still be served. +`genuine_v0_10_0_to_stamp_8_storage_upgrade_preserves_history` builds the +graph with 0.10.0, takes it to stamp 8 with the 0.11.0 +`upgrade --to-format 8`, and upgrades from there; +`genuine_v0_10_0_to_stamp_9_by_default_storage_upgrade_preserves_history` +lets the 0.11.0 `upgrade` run to its default target (stamp 9, the three +handlers through the system-column respelling) and asserts the reads at the +commits written before that respelling. Main builds 10 to 13 also +print `0.11.0`, so these journeys prove their source by the stamp +`snapshot --json` reports, not by `--version`. + +Each journey resolver reads its variable, else the binary under +`target/storage-upgrade-binaries/` (`stamp-13/`, `v0.11.0/`, `v0.10.0/`). +Without a binary the journey prints a skip line locally and panics when +`OMNIGRAPH_REQUIRE_STORAGE_UPGRADE_TESTS=1`, as CI sets. The v6 format fence +reads `OMNIGRAPH_V6_BIN` alone and skips when it is unset, whatever that +variable says. To run the +crossversion scope locally, build the stamp-13 predecessor as for the v5 fence +above, install the two releases with `scripts/install.sh` (`VERSION=v0.11.0` +and `VERSION=v0.10.0`, each with its `INSTALL_DIR`), place the three binaries +in those directories and run: ```bash -OMNIGRAPH_V13_BIN=/target/debug/omnigraph cargo test --workspace --locked --test crossversion_upgrade --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints storage_upgrade -- --test-threads=1 +cargo test --workspace --locked --test crossversion_upgrade --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints storage_upgrade -- --test-threads=1 ``` `db/upgrade/tests.rs` owns the `upgrade_storage` protocol over -`legacy::write::Stamp13History`, the test-only stamp-13 writer of -`omnigraph-catalog` that replays scripted publishes in the stamp-13 overwrite -order: the reports (`already_current` with no write on a fresh graph, +`legacy::write::LegacyHistory`, the test-only writer of +`omnigraph-catalog` that replays scripted publishes: `create` in the stamp-13 +overwrite order, `create_stamped` in the flat shape of stamps 8 and 9, for +which the test module also writes the three root schema objects. It covers +the reports (`already_current` with no write on a fresh graph, `unsupported_source`, `unsupported_target`, the pending-marker reports, `--check` leaving the store untouched), conversion and equivalence of main, named refs and fresh forks, the pre-fence refusals and bounds, every seam diff --git a/docs/dev/versioning.md b/docs/dev/versioning.md index a4048c3f4..7c812a679 100644 --- a/docs/dev/versioning.md +++ b/docs/dev/versioning.md @@ -10,7 +10,7 @@ version axes. Never derive one axis from another. |---|---|---| | Release | Published workspace artifacts move in lockstep. | Workspace manifests, lockfile, generated metadata, release automation. | | CLI ↔ server wire | One v0.12 contract; coordinated client/server upgrades. Exact request admission and CLI discovery/response validation. | Shared contract header, DTOs, HTTP refusal tests and OpenAPI drift tests. | -| Graph storage | Closed stamp range `[MIN_SUPPORTED, CURRENT]`, independent of system column vintage; a standalone stamp-13 graph is converted by the offline `omnigraph upgrade`, a graph at any other stamp is rebuilt by export and load; no open-time migration. | Main-manifest stamp guard on both bounds. | +| Graph storage | Closed stamp range `[MIN_SUPPORTED, CURRENT]`, independent of system column vintage; a standalone graph at stamp 8, 9 or 13 is converted by the offline `omnigraph upgrade`, a graph at any other stamp is rebuilt by export and load; no open-time migration. | Main-manifest stamp guard on both bounds. | | Lance dependency and file format | One deliberately pinned Lance family and explicit stable file version. | Lockfile, write parameters, and Lance surface guards. | ## Current storage contract @@ -30,7 +30,8 @@ feature set determines the vintage, never the storage stamp. `omnigraph schema upgrade-system-columns` converts the spellings on a supported standalone graph without changing its storage stamp. Normal open refuses a graph at any other stamp. The one in-place storage conversion is -the offline `omnigraph upgrade` from stamp 13. +the offline `omnigraph upgrade`, from stamps 8 and 9 (release 0.11.x) and +from stamp 13. - v4 was the last released pre-identity format, used by OmniGraph 0.8.x. - v5 was an unreleased development format that introduced SchemaIR v2, @@ -123,7 +124,7 @@ the offline `omnigraph upgrade` from stamp 13. would read the buffer as the whole history, so the stamp refuses it before any open. Two unreleased prototype layouts were stamped 14 and 15 in development trees only; neither shipped and neither has a conversion route. - A root converted from stamp 13 also holds `__history/legacy/`, written once + A root converted from stamp 8, 9 or 13 also holds `__history/legacy/`, written once by the upgrade and never again: the pre-upgrade commits as Lance data files under `legacy/data/` and the flat locator objects under `legacy/locator/` that find a commit by id and a writer by native name. A root born at 14 has @@ -139,21 +140,62 @@ the offline `omnigraph upgrade` from stamp 13. rebuild at a fresh root. Normal open refuses lower and higher stamps before recovery or table decoding; -it never migrates a graph. `omnigraph upgrade` is the one conversion: route -`history-lance-files-v13-to-v14` (`UPGRADE_PROTOCOL` 6, -`UPGRADE_SOURCE_FORMAT` 13) on a standalone root. A graph at stamp 14 reports -`already_current`. A stamp above 14 is the finding `newer_than_binary`, any -other stamp than 13 is `unsupported_source` with the open guard's refusal -text, and a `--to-format` other than 14 is `unsupported_target`, checked -before the stamp. A graph carrying `UPGRADE_PENDING_KEY` -(`omnigraph:storage_upgrade_pending`) stays refused by normal open; `--check` -reports it `recovery_required` and a run without `--check` resumes the attempt -its intent names. +it never migrates a graph. `omnigraph upgrade` is the one conversion, on a +standalone root, under one protocol (`UPGRADE_PROTOCOL` 6) from the stamps in +`UPGRADE_SOURCE_FORMATS` (8, 9 and 13). `UpgradeIntent.source_format` chooses +the route, and the report names it per source: +`history-lance-files-v8-to-v14`, `history-lance-files-v9-to-v14` and +`history-lance-files-v13-to-v14`. A stop before any route is chosen (an +unreadable intent, or recovery sidecars on a stamp no route converts) names +`history-lance-files-to-v14` as its failed handler. A graph at stamp 14 +reports `already_current`. A stamp above 14 is the finding +`newer_than_binary`, any other stamp outside `UPGRADE_SOURCE_FORMATS` is +`unsupported_source` with the open guard's refusal text, and a `--to-format` +other than 14 is `unsupported_target`, checked before the stamp. A graph +carrying `UPGRADE_PENDING_KEY` (`omnigraph:storage_upgrade_pending`) stays +refused by normal open; `--check` reports it `recovery_required` and a run +without `--check` resumes the attempt its intent names. + +The two kinds of source differ only in where the schema contract comes from +and in the shape of the `__manifest` rows read (`legacy::LegacyManifestSource`): + +- Stamp 13 (`Stamp13Source`): the packed shape, with the contract in the + `schema_contract` row of each version. +- Stamps 8 and 9 (`RootContractSource`): the flat shape of release 0.11.x, + which holds no contract row. The contract is read from `_schema.pg`, + `_schema.ir.json` and `__schema_state.json` at the graph root + (`db/upgrade/root_schema.rs`), each object in one read bounded by + `MAX_METADATA_BYTES` (64 MiB, the head bound), and is the contract of + every version. A `.staging` copy of one of the three objects, or + a recovery sidecar, is `source_recovery_required`, with or without a live + `__schema_apply_lock__` ref beside it; a live `__schema_apply_lock__` ref on + an otherwise clean root (no `.staging` copy, no sidecar) is reported as + `schema_apply_lock_retired` and handled once main is + fenced, so it is not a live ref of the converted graph, as the 0.11.x apply + that created it would have left it; a missing, inconsistent or over-bound + object, a stamp 8 whose IR declares `__id`/`__src`/`__dst`, a live ref whose + tables are not exactly those of the contract, or a registered table whose + Lance columns are not the ones the contract declares (one table open per + table, counted as `table_opens`; the check runs in `--check` and before any + write, so root objects restored from a backup older than a property-only + apply are refused), is `unsupported_source`. An absent object, or one whose + bytes cannot be read, is `unsupported_source`; every other storage failure + reading the three objects is `preflight_failed`: rerun the check. Once main + is fenced the contract is read from the archive under `__history/schemas/` + the intent binds, never from the root objects again, so a root object + changed or removed after the fence does not stop the rerun. Every decoded + pin gets + `last_linear_version = Some(table_version)`. The three root objects are + left in place: no stamp-14 reader opens them, and they go stale at the next + schema apply. The run, in order (`crates/omnigraph/src/db/upgrade.rs`): -1. Preflight, writing nothing: every live ref is stamp 13 in the packed shape - with a contract row, `__history/` holds nothing but schema archives, and +1. Preflight, writing nothing, after the actor is authorized (a denied actor + gets the policy denial, not a source finding): every live ref carries the + source stamp in the shape of its source and the contract is read (from the + contract row at 13, from the root objects at 8 and 9), `__history/` holds nothing but + schema archives, and the census (`omnigraph-catalog/src/legacy/census.rs`) reads every own version of every live and retired ref at its pinned head and derives the `HistoryRecord` of every commit. `--check` stops here with `check_passed` @@ -179,7 +221,10 @@ it exists the directory is the plan and no census runs. The bounds are `HISTORY_RELEASE_BYTES` of commit fields and `MAX_RECORD_BYTES` per record, `TAIL_BYTES` for the directory, and `MAX_CENSUS_CELLS` (2^33): a stamp-13 publish rewrites every row, so the census reads about 1.5 N² cells for N -commits on one lineage and refuses above about 75,000 commits. +commits on one lineage and refuses above about 75,000 commits. A stamp-8 or +stamp-9 `__manifest` never loses a row or a version: its head holds one +registration row per table and publish, the census reads every version since +`init`, and the commit count at which it refuses has not been derived. `MAX_CENSUS_SNAPSHOT_BYTES` (1 GiB) bounds the `TableRow` values the census retains, one per table for every commit record and head, and the `TableRegistration` values it keeps for the first version of every ref and @@ -200,8 +245,31 @@ with the requested version; a version of main below its genesis commit is `manifest_not_found`. Retired refs are never converted: their heads resolve through the writer shards. +What a converted 0.11.x root (stamp 8 or 9) does not carry over: + +- One schema contract for every pre-upgrade commit. 0.11.x overwrote the root + objects at each schema apply, so every legacy `HistoryRecord` names the + contract live at the upgrade (`UpgradeWork.schema_contents` is 1). The + table set and pins of an old commit are exact. A read at an old commit id + is planned against the live contract, as 0.11.x did. +- Lance table versions above a pin, left by an interrupted 0.11.x write, are + not detected: the upgrade opens each table of main's head at its pinned + version only, for the column comparison. `omnigraph repair` reports them + as `foreign_drift` and `cleanup` lists them as foreign versions. +- 0.11.x `cleanup`, and 0.11.x `schema apply --allow-data-loss` (which ran + the same version removal on the tables it changed), removed table versions + and no `__manifest` version, so a retained pre-upgrade `__manifest` version + can pin a table version that is gone. The collector then reports `is absent + from the listing` and sweeps nothing. `cleanup --keep 1` alone, as the first + cleanup after the upgrade, drops those `__manifest` versions; the upgrade + guide gives the order. +- `_graph_commit_recoveries.lance/` stays as an unreferenced dataset. +- The route writes no `last_linear_version` fill commit: `commit list` after + the upgrade equals `commit list` before it. + Stamps 10 to 13 were written by main development builds only, 13 from -2026-10-01 to 2026-10-04. Stamps that exist only between releases get no +2026-10-01 to 2026-10-04. Stamps 10 to 12 and every stamp below 8 stay +rebuild-only. Stamps that exist only between releases get no conversion unless a route is written for them, as for 13; each layout change still takes a new stamp, so a graph written in between is refused and never misread. [RFC 0064](../rfcs/0064-explicit-storage-upgrades.md) holds the @@ -229,7 +297,8 @@ Current compatibility fences and the required upstream reading set are in ## Rebuild -A graph at any stamp below 13, and a cluster-managed graph at 13, is rebuilt: +A graph at stamps 10 to 12 or below 8, and a cluster-managed graph at 8, 9 or +13, is rebuilt: export with the binary that wrote it, relocate `data.id` into top-level `id` in each record that has no top-level `id` (an existing one is preserved), then `init` and `load --mode overwrite` with this binary. @@ -239,7 +308,8 @@ Data, vectors and blobs are preserved; commit history and branches are not. See ## Storage upgrade support matrix The `storage_upgrade_compatibility` CI job (Storage Upgrade Compatibility) -builds the genuine stamp-13 CLI from main commit `c0a4519f` and requires the +builds the genuine stamp-13 CLI from main commit `c0a4519f`, installs the +released 0.10.0 and 0.11.0 CLIs, and requires the `storage_upgrade` cases of `crossversion_upgrade.rs`, the engine `db::upgrade::tests`, Lance version qualification and protocol guards on every change. Missing test cases, empty runs, a missing predecessor binary @@ -248,8 +318,10 @@ and skipped required cases fail the job. | Source format | Normal open | `omnigraph upgrade` | Required coverage owner | |---|---|---|---| | Current / v14 | Accepted | `already_current` for the default or target 14; any other target is `unsupported_target` | `upgrade/tests.rs` and CLI `crossversion_upgrade.rs::storage_upgrade_current_binary_reports_already_current_on_a_fresh_graph`; stamp tests in `migrations.rs` | -| v13 (main builds 2026-10-01 to 2026-10-04), standalone | Refused, naming `omnigraph upgrade` | Converted in place: `check_passed`, then `completed`; branches and history kept | CLI `crossversion_upgrade.rs::genuine_v13_storage_upgrade_preserves_history` against the genuine `c0a4519f` binary; `upgrade/tests.rs` over `legacy::write::Stamp13History` fixtures; `omnigraph-catalog` `legacy_` tests | -| v13, cluster-managed | Refused | Refused before any read; export and rebuild | `crossversion_upgrade.rs::storage_upgrade_refuses_cluster_path_aliases` | +| v13 (main builds 2026-10-01 to 2026-10-04), standalone | Refused, naming `omnigraph upgrade` | Converted in place: `check_passed`, then `completed`; branches and history kept | CLI `crossversion_upgrade.rs::genuine_v13_storage_upgrade_preserves_history` against the genuine `c0a4519f` binary; `upgrade/tests.rs` over `legacy::write::LegacyHistory` fixtures; `omnigraph-catalog` `legacy_` tests | +| v9 (release 0.11.x), standalone | Refused, naming `omnigraph upgrade` | Converted in place; branches and history kept, one schema contract for every pre-upgrade commit, root schema objects left in place | CLI `crossversion_upgrade.rs::genuine_v0_11_0_storage_upgrade_preserves_history` and `genuine_v0_11_0_storage_upgrade_after_predecessor_cleanup` against the released 0.11.0 binary, `genuine_v0_10_0_to_stamp_9_by_default_storage_upgrade_preserves_history` against the released 0.10.0 and 0.11.0 binaries; `upgrade/tests.rs` over `LegacyHistory::create_stamped` fixtures (`a_stamp_9_root_reached_from_6_by_the_released_upgrade_converts_its_retained_versions` for the retained stamp-6/7/8 versions); `omnigraph-catalog` `RootContractSource` tests | +| v8 (release 0.11.x, legacy system column spellings), standalone | Refused, naming `omnigraph upgrade` | Converted in place, as v9 | CLI `crossversion_upgrade.rs::genuine_v0_10_0_to_stamp_8_storage_upgrade_preserves_history` against the released 0.10.0 and 0.11.0 binaries; the stamp-8 cases of `upgrade/tests.rs` | +| v8, v9 or v13, cluster-managed | Refused | Refused before any read; export and rebuild | `crossversion_upgrade.rs::storage_upgrade_refuses_cluster_path_aliases` | | Any other stamp | Refused | `unsupported_source` (`newer_than_binary` above 14); export and rebuild | `upgrade/tests.rs`, the format fences and the refuse-and-rebuild tests in `crossversion_upgrade.rs` | | Pending conversion marker | Refused | `--check`: `recovery_required`; execute: resumes its own intent, refuses another's | `upgrade/tests.rs` | diff --git a/docs/rfcs/0064-explicit-storage-upgrades.md b/docs/rfcs/0064-explicit-storage-upgrades.md index a95471778..260e2a5d2 100644 --- a/docs/rfcs/0064-explicit-storage-upgrades.md +++ b/docs/rfcs/0064-explicit-storage-upgrades.md @@ -474,3 +474,28 @@ No maintainer decision on the questions above is recorded yet. The v6 to v9 routes described above were removed with their source decoders; those stamps are rebuilt. Cluster entry and object-store qualification stay open. +- 2026-10-04: the same protocol (intent protocol 6) also admits stamps 8 and + 9, the two formats of release v0.11.0, for local standalone roots. This + replaces the last sentence but one of the entry above for those two + stamps: they are converted, not rebuilt. `UpgradeIntent.source_format` + chooses the route, and the report names one handler per source: + `history-lance-files-v8-to-v14`, `history-lance-files-v9-to-v14` and + `history-lance-files-v13-to-v14`, with `history-lance-files-to-v14` as the + failed handler of a stop before any route is chosen. A stamp-8 or stamp-9 + `__manifest` holds no contract row, so the schema contract is read from + `_schema.pg`, `_schema.ir.json` and `__schema_state.json` at the graph + root and is the contract of every pre-upgrade commit. The upgrade leaves + the three objects in place: its write set equals the stamp-13 route's, no + stamp-14 reader opens them, and they go stale at the next schema apply. + Removing them is not part of this protocol. The contract's columns are + compared with each registered table's Lance columns before any write, so + root objects that no longer match the tables are refused. An unfinished + v0.11.0 schema apply (a `.staging` root object, or a recovery sidecar, with + or without a live `__schema_apply_lock__` ref) is refused before the fence; + a live `__schema_apply_lock__` ref alone, on an otherwise clean root, is + handled by the upgrade and is not a live ref of the converted graph. Once + fenced, the rerun reads the contract from the archive under + `__history/schemas/`, not from the root objects. The genuine predecessor is the released v0.11.0 + CLI, with v0.10.0 and the v0.11.0 `upgrade --to-format 8` for the stamp-8 + source. Stamps 10 to 12 and every stamp below 8 stay rebuild-only, and so + does every cluster-managed graph. diff --git a/docs/rfcs/0068-graph-commit-record.md b/docs/rfcs/0068-graph-commit-record.md index 4e0aa64f5..5cb513e0d 100644 --- a/docs/rfcs/0068-graph-commit-record.md +++ b/docs/rfcs/0068-graph-commit-record.md @@ -555,3 +555,16 @@ this amendment reads no `legacy/` and reports pre-upgrade ids as not found. upgrade: Lance data files plus flat locator objects under `__history/legacy/`, in place of a Lance dataset with an id index. The draft's record stream is not changed by it. +- 2026-10-04: the legacy area of the amendment also holds the pre-upgrade + commits of a stamp-8 or stamp-9 source (release v0.11.0), which the same + upgrade now converts ([RFC 0064](0064-explicit-storage-upgrades.md)). The + layout, the bounds and the argument of the amendment are unchanged; the + directory's source stamp field is 8, 9 or 13. Two statements of the + amendment hold for a stamp-13 source only. The schema content row lists + each distinct contract a legacy record names: for a stamp-8 or stamp-9 + source that is one contract, the one at the graph root at the upgrade, + named by every record. And the census bound of about 75,000 commits on + one lineage is derived from the stamp-13 overwrite publish; a stamp-8 or + stamp-9 `__manifest` keeps every row and every version, and its bound in + commits has not been derived. The amendment's heading keeps its name so + that existing links resolve. diff --git a/docs/user/cli/reference.md b/docs/user/cli/reference.md index 4ceca5352..df85dd0fc 100644 --- a/docs/user/cli/reference.md +++ b/docs/user/cli/reference.md @@ -54,7 +54,7 @@ server resolves the actor from the bearer token. Drop it, or use `--store ` | `schema plan` | Preview a schema migration | direct | | `schema upgrade-system-columns` | Respell a graph's system columns in place; needs a graph that normal open accepts and keeps its storage format | direct | | `lint` | Validate `.gq` source | local schema or direct graph | -| `upgrade` | Convert a v13 graph's storage to the format this binary serves, offline; `--check` writes nothing | direct standalone | +| `upgrade` | Convert a v8, v9 or v13 graph's storage to the format this binary serves, offline; `--check` writes nothing | direct standalone | | `optimize` | Compact data and reconcile declared indexes | direct | | `rebuild-full-text-indexes` | Replace full-text indexes on one branch | direct | | `repair` | Report each table's Lance history against its registration (`no_drift` or `foreign_drift`) | direct | @@ -135,12 +135,12 @@ omnigraph schema upgrade-system-columns ./graph.omni --check --json ``` `--store` is an alternative to the positional storage URI. `omnigraph upgrade` -converts a standalone v13 graph to the format this binary serves (v14), -offline and in place, keeping branches and commit history; stop every process -using the graph and retain a verified whole-root backup first. `--check` writes +converts a standalone v8, v9 (release 0.11.x) or v13 graph to v14, offline and +in place, keeping branches and commit history; stop every process using the +graph and retain a verified whole-root backup first. `--check` writes nothing; `--to-format` accepts 14 only. `check_passed`, `already_current` (a -v14 graph) and `completed` exit 0; `check_failed` (`unsupported_source` for a -format below 13, `newer_than_binary` for one above 14, `unsupported_target`) +v14 graph) and `completed` exit 0; `check_failed` (`unsupported_source` for any +other format below 14, `newer_than_binary` above 14, `unsupported_target`) and `recovery_required` (a pending attempt, finished as `recovery.action` says, or leftover recovery files) exit 1. The report names the formats, the route, the `work` counts, findings and the recovery action. Server and cluster diff --git a/docs/user/deployment.md b/docs/user/deployment.md index 1292e6e07..ed30ea0a0 100644 --- a/docs/user/deployment.md +++ b/docs/user/deployment.md @@ -8,7 +8,7 @@ and exposes its ready applied graphs under `/graphs/{id}/…`; use Start with [Operating a cluster](clusters/index.md) to create and apply the deployment bundle. -Servers use the v0.12 HTTP contract and open graph storage format v13. The +Servers use the v0.12 HTTP contract and open graph storage format v14. The cluster ledger has a separate version: explicitly converting it to v2 preserves graph data and history. See [cluster deployments](clusters/index.md). diff --git a/docs/user/operations/troubleshooting.md b/docs/user/operations/troubleshooting.md index bd6efb18a..d1d10fc1a 100644 --- a/docs/user/operations/troubleshooting.md +++ b/docs/user/operations/troubleshooting.md @@ -149,7 +149,7 @@ See [Operating a cluster](../clusters/index.md). - Recovery required: reopen the graph read-write or restart its server. A graph carrying a sidecar from a release before 0.12 must first be opened - read-write with that release. + read-write with that release (`omnigraph snapshot ` is such an open). - Foreign drift: `repair` reports Lance commits above a table's last linear version as `foreign_drift`; no read or write uses them and no command adopts them (below). diff --git a/docs/user/operations/upgrade.md b/docs/user/operations/upgrade.md index da42a07bb..a41bd6cbf 100644 --- a/docs/user/operations/upgrade.md +++ b/docs/user/operations/upgrade.md @@ -1,23 +1,23 @@ # Upgrading OmniGraph Normal open accepts storage format v14 and never migrates a graph. A -standalone v13 graph is converted in place by the offline -[storage upgrade](#storage-upgrade). A graph at any other format, and every -cluster-managed graph below v14, is refused and takes the +standalone graph at v8 or v9 (release 0.11.x) or at v13 is converted in place +by the offline [storage upgrade](#storage-upgrade). A graph at any other +format, and every cluster-managed graph below v14, is refused and takes the [export/import rebuild](#rebuild) with the binary that wrote the graph. Check the [release notes](../../releases/) for storage and index compatibility. ## Storage upgrade -`omnigraph upgrade` converts a standalone v13 graph to v14 in place. Branches, -commit ids, commit history and table data are kept; no table row is rewritten. -v13 was written only by main development builds from 2026-10-01 to 2026-10-04. +`omnigraph upgrade` converts a standalone v8, v9 or v13 graph to v14 in place. +Branches, commit ids, commit history and table data are kept; no table row is +rewritten. Only main builds from 2026-10-01 to 2026-10-04 wrote v13. The upgrade is offline, and it cannot verify that itself: 1. Stop every server, embedded reader or writer and maintenance process that has the graph open, and keep them stopped until the upgrade reports - `completed`. A v13 process opened earlier can still write to a branch; the + `completed`. An old process opened earlier can still write to a branch; the upgrade then stops and the root is restored from the backup. 2. Take and verify a backup of the whole graph root. There is no reverse conversion: rollback is restoring that backup with the old binary. @@ -38,7 +38,7 @@ executable. Never delete the marker or anything under `__history/`. | `outcome` | Meaning | Exit | |---|---|---| -| `check_passed` | v13 graph, every check passed, nothing written | 0 | +| `check_passed` | v8, v9 or v13 graph, every check passed, nothing written | 0 | | `already_current` | v14 graph, nothing to do | 0 | | `completed` | converted, validated and served as v14 | 0 | | `check_failed` | a finding below; nothing was written | 1 | @@ -48,17 +48,20 @@ The report's `work` object counts what the run read and writes: `live_refs`, `retired_refs` (deleted branches whose commits are kept), `orphan_writers`, `legacy_commits`, `bookkeeping_versions`, `absent_parents`, `data_files`, `id_shards`, `writer_shards`, `schema_contents`, `census_reads`, -`census_cells` and `legacy_bytes`. A resumed run that finds the legacy objects -complete reads no history again and reports `retired_refs`, `orphan_writers`, -`bookkeeping_versions`, `schema_contents`, `census_reads`, `census_cells` and -`legacy_bytes` as zero. +`census_cells`, `legacy_bytes` and `table_opens` (the tables of a v8 or v9 +graph opened to compare their columns with the root schema objects). A +resumed run that finds the legacy objects complete reads no history again and +reports `retired_refs`, `orphan_writers`, `bookkeeping_versions`, +`schema_contents`, `census_reads`, `census_cells`, `legacy_bytes` and +`table_opens` as zero. | Finding | Names | What to do | |---|---|---| | `unsupported_target` | the target | `--to-format` accepts 14 only | | `newer_than_binary` | the format | upgrade the binary | -| `unsupported_source` | the format, layout or over-budget branch | [rebuild](#rebuild) with the build that wrote the graph. For a retired branch, instead of rebuilding run `cleanup` with a retention option and `--confirm` using that build, then run `upgrade --check` again; rebuild only when the retired branch stays | -| `source_recovery_required` | leftover recovery files | open the graph read-write with the build that wrote them, then check again | +| `unsupported_source` | the format, layout, over-budget branch, or the unreadable, over-bound or table-mismatched root schema objects of a v8 or v9 graph | [rebuild](#rebuild) with the build that wrote the graph. For a retired branch, instead of rebuilding run `cleanup --keep --confirm` with that build, then `upgrade --check` again; rebuild only when the retired branch stays. For root schema objects, restore the whole root from one backup, objects and tables together, then check again; the objects alone, from an older backup, no longer match the tables and are refused. An unreadable contract refuses the 0.11.x open too, so export with 0.11.x is no way out of it | +| `source_recovery_required` | leftover recovery files, or a `.staging` root schema object of a v8 or v9 graph, each with or without a live `__schema_apply_lock__` branch | stop every writer, then open the graph read-write with 0.11.x: `omnigraph snapshot ` is such an open (`upgrade --check` is not). The open removes the recovery files and finishes or rolls back a schema apply that changed the table set; a property-only `.staging` it refuses and names the manual choice between the live and the staging schema. The lock stays live after the open; the upgrade handles it (next row). Then check again | +| `schema_apply_lock_retired` | a live `__schema_apply_lock__` branch on a root with no `.staging` object and no recovery files: the lock of a v8 or v9 schema apply killed before it released it | nothing: the upgrade handles it, and the converted graph does not hold it as a live branch; `--check` reports it, the run handles it | | `history_objects_present` | object keys | restore the whole root, `__history/` included, from the backup of the earlier attempt | | `legacy_lineage_incomplete`, `legacy_lineage_corrupt` | up to 20 commit ids or branches | rebuild | | `legacy_record_over_bound`, `legacy_census_over_bound`, `legacy_directory_over_bound` | the measured value and its limit | rebuild | @@ -66,26 +69,24 @@ complete reads no history again and reports `retired_refs`, `orphan_writers`, | `preflight_failed` | the error | fix the cause and check again; nothing was written. Above the live-branch limit: rebuild | | `pending_upgrade`, `upgrade_interrupted` | the attempt | rerun without `--check` with the same executable | | `unknown_upgrade_ownership`, `fence_publication_attempted` | the marker state | keep the root, run `--check`, finish with the executable that started the attempt | -| `legacy_plan_changed`, `legacy_objects_differ` | digests | finish with the executable that marked the graph; restore the backup if an object under `__history/legacy/` was changed | +| `legacy_plan_changed`, `legacy_objects_differ` | digests | finish with the executable that marked the graph; restore the backup if an object under `__history/legacy/`, or the schema archive under `__history/schemas/` that a fenced v8 or v9 rerun reads instead of the root schema objects, was changed | Limits, checked before anything is written: 1,024 live branches including main and 1,024 retired branches; 1,000,000 catalog rows or 64 MiB per branch -head, live or retired; 256 KiB of commit fields and 16 MiB per commit record; -a history-read budget that refuses a single branch line above about -75,000 commits; and a history-memory budget of 1 GiB that admits about -3.8 million table entries, counted for each branch as its commits plus its -head, times the tables of its head (10,000 commits over 383 tables, or -1,000 commits over 3,800). That count multiplies every commit of a branch by -the tables of its head, so a graph whose tables were added late is -over-counted. The budget counts only the fixed-size part of each table -entry: table names and paths are not counted and planning holds a second -copy, so the process needs several times 1 GiB of memory. A graph over a -limit is rebuilt. A retired branch over a limit cannot shrink: `cleanup` -with the build that wrote the graph removes it when no live branch, merge -base or tag needs it. Cleanup without `--confirm` is a dry run, so pass a -retention option and `--confirm` (for example -`omnigraph cleanup --keep 10 --confirm ./graph.omni`, see -[maintenance](maintenance.md)), then run `upgrade --check` again. +head, live or retired, and 64 MiB per root schema object of a v8 or v9 graph; +256 KiB of commit fields and 16 MiB per commit record; a history-read budget +that refuses a single v13 branch line above about 75,000 commits; and a +history-memory budget of 1 GiB that admits about 3.8 million table entries, +counted for each branch as its commits plus its head, times the tables of its +head (10,000 commits over 383 tables, or 1,000 commits over 3,800). That count +multiplies every commit of a branch by the tables of its head, so a graph +whose tables were added late is over-counted. The budget counts only the +fixed-size part of each table entry: table names and paths are not counted +and planning holds a second copy, so the process needs several times 1 GiB of +memory. A graph over a limit is rebuilt. A retired branch over a limit cannot +shrink; 0.11.x `cleanup --keep --confirm` removes it when no live branch, +merge base or tag needs it (without `--confirm` it is a dry run), then run +`upgrade --check` again. After the upgrade, commit ids from before it resolve as before, and commit listing, the change feed, merges and `cleanup` read across the upgrade. A @@ -94,10 +95,19 @@ at that version, or the nearest commit below it. A v14 development build older than this upgrade reports pre-upgrade commit ids as not found and refuses full-history reads on an upgraded graph: upgrade every binary first. -Server and cluster selectors, cluster profiles and recognized cluster-layout -roots are refused: a cluster-managed graph is exported with the build that -wrote it and [rebuilt](#rebuild). Local paths, file URIs and symlink aliases -are resolved before the cluster ownership check. +A graph written by 0.11.x (v8 or v9) has limits of its own. Its checks read every catalog version since `init`, so a long history is slow to check. + +| Subject | After the upgrade | +|---|---| +| Root schema objects | `_schema.pg`, `_schema.ir.json` and `__schema_state.json` at the graph root are read as the graph's schema and must be present and consistent. The upgrade leaves them in place. v14 never reads them, and they go stale at the next schema apply. `_graph_commit_recoveries.lance/` also stays, as an unreferenced dataset | +| Unfinished schema apply | A `.staging` copy of a root schema object, or a leftover recovery file, is refused as `source_recovery_required`, with or without a live `__schema_apply_lock__` branch: open the graph read-write with 0.11.x (`omnigraph snapshot `), which removes the recovery file and finishes or rolls back an apply that changed the table set. An apply that changed properties only leaves a `.staging` that 0.11.x refuses as well; its message names the manual choice between the live and the staging schema. 0.11.x releases the lock only inside the process that took it, so the lock stays live after that open. A live lock on an otherwise clean root is handled by the upgrade: the converted graph does not hold it as a live branch, and the next schema apply is not refused as already in progress | +| Schema of old commits | 0.11.x kept only the latest schema, so every commit from before the upgrade is recorded under the one schema live at the upgrade. The tables and rows of an old commit are exact. A read at an old commit id is planned against the live schema, as in 0.11.x: a type added later is absent there, and a type dropped later cannot be queried | +| Table versions above a commit | An interrupted 0.11.x write can leave them. The upgrade opens each table only at the version main's head pins and does not report them. Reads and writes ignore them, `repair` reports `foreign_drift` and `cleanup` lists them as foreign versions | +| `cleanup` after a 0.11.x `cleanup` or `schema apply --allow-data-loss` | Both removed table versions that old catalog versions still name. `cleanup --older-than`, or a `--keep` that reaches below the upgrade, then reports `is absent from the listing` and deletes nothing. Run `omnigraph cleanup --keep 1 --confirm` alone as the first cleanup; later runs with any retention option pass. A read at a commit id whose table version 0.11.x removed fails as it did before | + +Server and cluster selectors, cluster profiles and cluster-layout roots are +refused: a cluster-managed graph is exported with the build that wrote it and +[rebuilt](#rebuild). Paths, file URIs and symlinks resolve before that check. ## v0.9 to v0.10 @@ -228,7 +238,7 @@ mapping is: | v5 | the exact unreleased development build that wrote it | | v6 | latest 0.10.x (the refusal names 0.9.x or 0.10.x) | | v7 | the exact unreleased development build that wrote it | -| v8, v9 | latest 0.11.x (v8: legacy system column spellings) | +| v8, v9 | latest 0.11.x (v8: legacy system column spellings). A standalone graph takes the [storage upgrade](#storage-upgrade) instead | | v10 to v13 | the main development build that wrote it; the refusal names its build dates. A standalone v13 graph takes the [storage upgrade](#storage-upgrade) instead | | v14 | current binary; entity export/import is not required within this storage generation | diff --git a/scripts/check-storage-upgrade-ci.py b/scripts/check-storage-upgrade-ci.py index 33197aaec..1421b48b8 100644 --- a/scripts/check-storage-upgrade-ci.py +++ b/scripts/check-storage-upgrade-ci.py @@ -17,6 +17,10 @@ "storage_upgrade_current_binary_reports_already_current_on_a_fresh_graph", "genuine_v13_storage_upgrade_preserves_history", "storage_upgrade_refuses_cluster_path_aliases", + "genuine_v0_11_0_storage_upgrade_preserves_history", + "genuine_v0_11_0_storage_upgrade_after_predecessor_cleanup", + "genuine_v0_10_0_to_stamp_8_storage_upgrade_preserves_history", + "genuine_v0_10_0_to_stamp_9_by_default_storage_upgrade_preserves_history", ) ENGINE_CASES = ( "a_fresh_graph_is_already_current_and_nothing_is_written", @@ -25,11 +29,24 @@ "a_pending_conversion_with_a_valid_intent_is_reported_by_check_as_pending", "a_pending_conversion_with_an_unreadable_intent_is_unknown_ownership", "check_has_no_local_store_effects", - "a_stamp_13_root_is_converted_once_and_is_then_current", + "a_source_root_is_converted_once_and_is_then_current", "history_leftovers_refuse_before_fence", "over_bound_record_refuses_before_fence", "census_over_bound_refuses_before_reads", "policy_denial_precedes_effects", + "recovery_sidecars_refuse_under_the_route_of_the_stamp", + "a_staged_schema_object_at_the_root_refuses_before_the_contract_is_read", + "an_unreadable_root_contract_refuses_as_unsupported_source", + "a_store_failure_reading_the_root_contract_fails_preflight_and_a_rerun_routes", + "a_live_ref_stamped_differently_from_main_fails_preflight", + "the_schema_state_v0_11_0_writes_converts_with_or_without_an_unread_field", + "stamp_8_refuses_v3_system_columns_and_stamp_9_converts_the_legacy_ones", + "a_live_schema_apply_lock_is_retired_and_a_retired_one_converts", + "a_schema_apply_lock_beside_staging_or_a_sidecar_stays_refused", + "an_over_bound_root_schema_object_refuses_as_unsupported_source", + "a_root_contract_whose_columns_the_tables_lack_refuses_before_any_write", + "a_root_contract_that_does_not_describe_a_live_ref_refuses", + "a_stamp_9_root_reached_from_6_by_the_released_upgrade_converts_its_retained_versions", "cleanup_after_upgrade_with_keep_four_and_older_than", "merge_with_legacy_base_pins_and_collector_keeps_it", "leftover_merge_input_tag_on_bookkeeping_version_resolves", @@ -40,8 +57,14 @@ "named_and_fresh_fork_conversions_pass_equivalence", "fork_head_and_writer_head_release_equal_copies", "interrupted::interruption_boundaries_retry_without_mixed_visibility", + "interrupted::interruption_boundaries_retry_from_stamp_9", + "interrupted::interruption_boundaries_retry_from_stamp_9_with_a_schema_apply_lock", + "interrupted::interruption_boundaries_retry_from_stamp_8", + "interrupted::a_fenced_stamp_8_or_9_rerun_reads_the_archived_contract_not_the_root_objects", + "interrupted::a_fenced_stamp_9_rerun_without_its_archived_contract_asks_for_the_backup", "interrupted::partial_legacy_write_is_completed_on_retry", "interrupted::resume_after_directory_skips_census", + "interrupted::fenced_census_applies_source_bounds_and_directory_resume_reads_no_head", "interrupted::foreign_layout_version_refuses", "interrupted::a_source_changed_after_the_fence_refuses_as_plan_changed", "interrupted::modified_legacy_object_refuses_resume", @@ -98,6 +121,37 @@ cargo clean --workspace --locked \\ --target-dir "$GITHUB_WORKSPACE/target" echo "OMNIGRAPH_V13_BIN=$v13_bin" >> "$GITHUB_ENV\"""" +RELEASE_TOKENS = ( + 'echo "OMNIGRAPH_V6_BIN=$v6_dir/omnigraph" >> "$GITHUB_ENV"', + 'echo "OMNIGRAPH_V011_BIN=$v011_dir/omnigraph" >> "$GITHUB_ENV"', +) +RELEASE_SCRIPT = """\ +set -euo pipefail +v6_dir="$RUNNER_TEMP/omnigraph-v010" +v011_dir="$RUNNER_TEMP/omnigraph-v011" +# The installer downloads the official archive and verifies its +# SHA256 before extraction. An exact VERSION never falls back to edge. +# A failed download leaves nothing behind, so each release gets three +# attempts before this required context goes red. +install_release() { + for attempt in 1 2 3; do + REPO_SLUG=ModernRelay/omnigraph VERSION="$1" INSTALL_DIR="$2" \\ + bash scripts/install.sh && return 0 + echo "install of $1 failed on attempt $attempt/3; retrying" + sleep 10 + done + return 1 +} +install_release v0.10.0 "$v6_dir" +test -x "$v6_dir/omnigraph" +[[ "$("$v6_dir/omnigraph" --version)" == "omnigraph 0.10.0" ]] \\ + || { echo "::error::the storage upgrades of a 0.10.0 graph require the genuine v0.10.0 CLI"; exit 1; } +install_release v0.11.0 "$v011_dir" +test -x "$v011_dir/omnigraph" +[[ "$("$v011_dir/omnigraph" --version)" == "omnigraph 0.11.0" ]] \\ + || { echo "::error::the release storage upgrade requires the genuine v0.11.0 CLI"; exit 1; } +echo "OMNIGRAPH_V6_BIN=$v6_dir/omnigraph" >> "$GITHUB_ENV" +echo "OMNIGRAPH_V011_BIN=$v011_dir/omnigraph" >> "$GITHUB_ENV\"""" def scope_script(scope: str) -> str: @@ -136,6 +190,7 @@ def validate(workflow: str, policy: dict) -> list[str]: f"name: {CONTEXT}", "run: python3 scripts/check-storage-upgrade-ci.py --self-test", *PREDECESSOR_TOKENS, + *RELEASE_TOKENS, ): if token not in job: failures.append(f"storage compatibility is missing {token!r}") @@ -145,6 +200,8 @@ def validate(workflow: str, policy: dict) -> list[str]: scripts = {"\n".join(line[10:] for line in body.rstrip().splitlines()) for body in scripts} if PREDECESSOR_SCRIPT not in scripts: failures.append("storage compatibility requires the exact genuine stamp-13 build script") + if RELEASE_SCRIPT not in scripts: + failures.append("storage compatibility requires the exact released v0.10.0 and v0.11.0 install script") for scope in SCOPES: if scope_script(scope) not in scripts: failures.append(f"storage compatibility requires the exact fail-closed {scope} command and log check") @@ -253,15 +310,63 @@ def test_removed_or_altered_predecessor_build_line_fails(self): self.assertNotEqual(swapped, self.workflow) self.assertEqual(validate(swapped, self.policy), exact) + def storage_job(self): + start = self.workflow.index(" storage_upgrade_compatibility:\n") + end = self.workflow.index("\n v5_v10_format_fence:\n", start) + return self.workflow[start:end] + + def test_missing_release_binary_export_fails(self): + job = self.storage_job() + for token in RELEASE_TOKENS: + with self.subTest(token=token): + self.assertEqual(job.count(token), 1) + changed = self.workflow.replace(job, job.replace(token, "")) + failures = validate(changed, self.policy) + self.assertIn(f"storage compatibility is missing {token!r}", failures) + + def test_removed_or_altered_release_install_line_fails(self): + exact = ["storage compatibility requires the exact released v0.10.0 and v0.11.0 install script"] + job = self.storage_job() + lines = RELEASE_SCRIPT.splitlines() + block = "".join(f" {line}\n" for line in lines) + self.assertEqual(job.count(block), 1) + for index, line in enumerate(lines): + for replacement in ([], [f"true # {line}"], [f"{line} || true"]): + with self.subTest(line=line, replacement=replacement): + altered = lines[:index] + replacement + lines[index + 1:] + changed = self.workflow.replace(job, job.replace( + block, "".join(f" {line}\n" for line in altered) + )) + failures = validate(changed, self.policy) + self.assertTrue(set(exact) <= set(failures), failures) + for before, after in ( + ("install_release v0.11.0", "install_release v0.12.0"), + ("for attempt in 1 2 3", "for attempt in 1"), + ("bash scripts/install.sh && return 0", "bash scripts/install.sh; return 0"), + ): + with self.subTest(before=before, after=after): + moved = self.workflow.replace(job, job.replace(before, after)) + self.assertNotEqual(moved, self.workflow) + self.assertEqual(validate(moved, self.policy), exact) + def test_missing_required_context_fails(self): self.assertTrue(validate(self.workflow, {})) def test_expected_cases_name_the_genuine_journey_and_every_engine_case(self): - self.assertIn("genuine_v13_storage_upgrade_preserves_history", expected_cases("crossversion")) + crossversion = expected_cases("crossversion") + for name in CASES: + self.assertIn(name, crossversion) engine = expected_cases("engine") for name in ENGINE_CASES: self.assertIn("db::upgrade::tests::" + name, engine) + def test_every_named_case_exists_in_its_source(self): + for scope, cases in (("crossversion", CASES), ("engine", ENGINE_CASES)): + source = (ROOT / SCOPES[scope][1]).read_text() + for name in cases: + with self.subTest(name=name): + self.assertRegex(source, rf"\bfn {name.rsplit('::', 1)[-1]}\(") + def test_log_requires_every_case_and_positive_unskipped_summary(self): good = "test alpha ... ok\ntest beta ... ok\ntest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 4 filtered out; finished in 1s\n" expected = {"alpha", "beta"} @@ -271,7 +376,9 @@ def test_log_requires_every_case_and_positive_unskipped_summary(self): good.replace("test beta ... ok", "test beta ... ignored"), good.replace("2 passed", "0 passed"), good.replace("0 ignored", "1 ignored"), - good + "skipping genuine v13 storage upgrade: OMNIGRAPH_V13_BIN is unset\n", + good + "skipping genuine stamp-13 storage upgrade: OMNIGRAPH_V13_BIN is unset\n", + good + "skipping genuine v0.11.0 storage upgrade: OMNIGRAPH_V011_BIN is unset\n", + good + "skipping genuine v0.10.0 storage upgrade: OMNIGRAPH_V6_BIN is unset\n", good + good, ): with self.subTest(log=log): @@ -300,7 +407,7 @@ def main() -> int: for failure in failures: print(f"Storage upgrade CI: {failure}", file=sys.stderr) return 1 - print("Storage upgrade CI OK (genuine stamp-13 route, cluster refusal, engine protocol, Lance and registry coverage).") + print("Storage upgrade CI OK (genuine stamp-13 route, genuine v0.11.0 and v0.10.0 release route, cluster refusal, engine protocol, Lance and registry coverage).") return 0 diff --git a/skills/omnigraph/SKILL.md b/skills/omnigraph/SKILL.md index 1f3edc37d..d82677c4b 100644 --- a/skills/omnigraph/SKILL.md +++ b/skills/omnigraph/SKILL.md @@ -17,8 +17,12 @@ Check `omnigraph version` and the command's `--help` before using these instructions. This skill targets the [v0.11.0 release](https://github.com/ModernRelay/omnigraph/releases/tag/v0.11.0), which reports `internal-schema 9` and serves both v8 and v9 graphs. New graphs use v9; opening v8 preserves its physical columns. A v0.10 graph uses v6 and -requires an explicit upgrade or rebuild. Read [migration guidance](references/migrations.md) -before replacing a deployed binary. +requires an explicit upgrade or rebuild. The binary after 0.11 (main after +2026-10-04) serves storage format 14 only and converts a standalone v8, v9 or +v13 graph in place with its own `omnigraph upgrade`; see +[upgrading past v0.11](references/migrations.md#upgrade-v011-to-storage-format-14). +Read [migration guidance](references/migrations.md) before replacing a +deployed binary. ## Upcoming GQ 2.1 traversal support @@ -182,7 +186,9 @@ accept `--cluster --graph `: - `upgrade [--check] [--to-format <7|8|9>] [--json]` — offline standalone conversion; defaults to v9. `schema upgrade-system-columns [--check] [--json]` is the v8→v9 step. Read [migration preconditions](references/migrations.md) - first; cluster-managed roots refuse. + first; cluster-managed roots refuse. In the binary after 0.11 the same + `upgrade [--check] [--to-format 14] [--json]` converts a v8, v9 or v13 + graph to format 14 and accepts no other target. - `lint --query [--schema ] [] [--json]` — offline with `--schema`, graph-backed with a URI - `optimize [--json]` · `repair [--confirm] [--force] [--json]` · `cleanup [--keep ] [--older-than <7d>] --confirm [--json]` (at least one retention option; both may be combined) - `rebuild-full-text-indexes [--branch ] [--json]` — replace full-text indexes on one branch with default English analysis; custom tokenizer settings are replaced. Stop overlapping writers and retain a whole-store backup for upgrades. `--as` records attribution; direct access does not load server policy. See [maintenance commands](references/commands.md#rebuild-full-text-indexes--explicit-analyzer-upgrade). @@ -355,6 +361,7 @@ These are the traps most likely to bite. Scan this table before debugging any pa | Full-text search on indexes built before Lance 11 (v0.9 era), including after `omnigraph upgrade` | `FullTextIndexRebuildRequired` | Stop mixed-version access and run `rebuild-full-text-indexes --branch ` on every live branch that needs text search; `upgrade` does not rebuild them | | Reopening a v0.10/v6 graph with v0.11 | `__manifest is stamped at internal schema v6, but this omnigraph reads only v8 to v9` | Stop the old fleet and use the explicit standalone upgrade or cluster rebuild procedure | | Default `omnigraph upgrade` on a branched graph or one with `_` properties | `the route ends at v9, which requires a graph with only main` (or `reserves property names starting with '_'`) | Delete branches / `@rename_from` first, or pass `--to-format 8` to both check and execution | +| Opening a v8 or v9 graph with the binary after 0.11 | refused: that binary reads format 14 only and names `omnigraph upgrade` | Stop every process using the graph, back up the whole root, then run that binary's `upgrade --check` and `upgrade` on a standalone graph; a cluster-managed graph is exported with 0.11.x and rebuilt. See [upgrading past v0.11](references/migrations.md#upgrade-v011-to-storage-format-14) | | Using `data.id` for identity on a new graph | `unknown input field 'id': move data.id to the top-level 'id' field` | Put identity in top-level JSONL `id`; `data` contains user properties | | Re-running a `--mode merge` load with unkeyed edges | duplicate edges | Supply stable top-level edge `id`s, or declare `@key(@src, @dst)` when creating the edge type | | Assuming every JSON result cell has a key | absent keys for null values | Query/export JSON omits null cells; change images retain explicit nulls | diff --git a/skills/omnigraph/references/commands.md b/skills/omnigraph/references/commands.md index 57a2e5b1f..e5de976ff 100644 --- a/skills/omnigraph/references/commands.md +++ b/skills/omnigraph/references/commands.md @@ -125,12 +125,17 @@ omnigraph upgrade "$REPO" --json omnigraph schema upgrade-system-columns "$REPO" --check --json ``` -The default target is v9; `--to-format 8` retains legacy system spellings and -can preserve live branches. The v9 step requires only `main` and no user -property starting with `_`. These standalone operations require stopped writers -and a verified whole-root backup; cluster-managed roots refuse. Read +In 0.11.0 the default target is v9; `--to-format 8` retains legacy system +spellings and can preserve live branches. The v9 step requires only `main` and +no user property starting with `_`. These standalone operations require stopped +writers and a verified whole-root backup; cluster-managed roots refuse. Read [migration preconditions](migrations.md) before executing. +The binary after 0.11 runs the same two commands to convert a v8, v9 or v13 +graph to storage format 14 in place, keeping branches and commit history; +`--to-format` accepts 14 only there. See +[upgrading past v0.11](migrations.md#upgrade-v011-to-storage-format-14). + ## Lint ```bash diff --git a/skills/omnigraph/references/migrations.md b/skills/omnigraph/references/migrations.md index b5bb979aa..d8495b41f 100644 --- a/skills/omnigraph/references/migrations.md +++ b/skills/omnigraph/references/migrations.md @@ -56,6 +56,50 @@ Do not point `init --force` at the old root. The [upgrade guide](https://github.com/ModernRelay/omnigraph/blob/v0.11.0/docs/user/operations/upgrade.md) owns qualification, interruption handling, external Blob caveats and cluster cutover details. +## Upgrade v0.11 to storage format 14 + +The binary after 0.11 (main after 2026-10-04) serves storage format 14 only: +it refuses a v8 or v9 graph on open and names `omnigraph upgrade`. Its +`upgrade` converts a standalone v8, v9 or v13 graph to 14 in place and keeps +branches, commit ids and commit history; no table row is rewritten. +`--to-format` accepts 14 only. The upgrade is offline and cannot verify that +itself: stop every server, reader, writer and maintenance process, take and +verify a backup of the whole graph root (rollback is that backup with the +0.11 binary), then with the new binary: + +```bash +omnigraph upgrade ./graph.omni --check --json +omnigraph upgrade ./graph.omni --json +omnigraph commit list ./graph.omni --json +``` + +`--check` writes nothing. A run without `--check` marks the graph as pending, +which every normal open refuses, and a run that stops is resumed by the same +command with the same executable. Never delete the marker or anything under +`__history/`. What a 0.11 graph brings with it: + +- Its schema is read from `_schema.pg`, `_schema.ir.json` and + `__schema_state.json` at the graph root, which must be present, consistent + and match the tables' columns. The upgrade leaves them in place; format 14 + never reads them. Every pre-upgrade commit is recorded under the one schema + live at the upgrade, as 0.11 kept only the latest schema. +- A `.staging` schema object or a recovery file left by 0.11 is refused until + a 0.11 read-write open resolves it (`omnigraph snapshot ./graph.omni` with + the 0.11 binary is such an open; a property-only `.staging` 0.11 refuses + too and names the manual choice). A `__schema_apply_lock__` branch left + alone by a killed 0.11 schema apply is handled by the upgrade. +- On a root where 0.11 `cleanup` or `schema apply --allow-data-loss` ran, the + first `cleanup` after the upgrade is `cleanup --keep 1 --confirm` alone; + later runs with any retention option pass. +- A cluster-managed graph is not converted: export with 0.11, then `init` and + `load --mode overwrite` with the new binary, as in the section above. +- A 0.9/0.10 graph (v6) is first taken to v9 with the 0.11 `upgrade` above, + then to 14; v7 and v10 to v12 are rebuilt. + +The current +[upgrade guide](https://github.com/ModernRelay/omnigraph/blob/main/docs/user/operations/upgrade.md#storage-upgrade) +owns the findings table, the limits and the recovery actions. + ### Identity and response changes - GQ system fields are `$p.@id`, `$e.@src`, and `$e.@dst`. Bare `id`, `src`