diff --git a/docs/architecture/rfcs/shared-goal-authority-state-provider-v0.md b/docs/architecture/rfcs/shared-goal-authority-state-provider-v0.md index bac0923c6..6300a21d9 100644 --- a/docs/architecture/rfcs/shared-goal-authority-state-provider-v0.md +++ b/docs/architecture/rfcs/shared-goal-authority-state-provider-v0.md @@ -3158,6 +3158,19 @@ freshness, a background drainer, all L5 consumers or D2/D3. See the naming passed, failed and missing evidence. NoKV and PostgreSQL retain their own qualification; a pass on SQLite cannot waive another affected provider. +**D3 recovery artifact checkpoint (2026-09-22).** The +[canonical authority archive](../../reference/authority-archive.md) now defines +an explicit CLI export/verify/isolated-restore journey. One TS retained-journal +codec preserves operation/receipt identity, complete historical state and source +provenance across File/SQLite and service-owned PostgreSQL. Verified recovery +copies do not select an authority, revive executors or roll back writer fences. +This closes the portable recovery-artifact gap only; final source draining, +fenced target adoption, later-write accounting and cohort cutover remain L8. +The conditional **5–8 package** estimate below is unchanged until the remaining +caller, projection, D2, migration and default exits are qualified. Pending +reviewed-promotion and command-recovery PRs must be integrated at their accepted +heads rather than counted as merged prerequisites. + **D3 — integrate and request a whole-Goal cutover.** - Requires T1–T3, D1/D2 and qualified capture; production cutover additionally diff --git a/docs/architecture/rfcs/typescript-control-plane-migration-v0.md b/docs/architecture/rfcs/typescript-control-plane-migration-v0.md index 9fb2ce0df..fbc73e3c1 100644 --- a/docs/architecture/rfcs/typescript-control-plane-migration-v0.md +++ b/docs/architecture/rfcs/typescript-control-plane-migration-v0.md @@ -1109,6 +1109,16 @@ debit. This closes the demonstrated T3 consumer gap, not D1–D3, provider promotion, or the remaining Python transaction adapters. See the [operating contract](../../quota-allocation.md#receipt-backed-settlement-progress). +**Recovery boundary (2026-09-22).** The +[authority archive command](../../reference/authority-archive.md) places retained +history validation, delta reconstruction and resumable restore in the existing +TS coordination owner. Python only resolves CLI paths and transports compact +requests/results; it does not reconstruct domain state or duplicate a provider +migration rule. The state-log codec is reused rather than creating independent +File/SQLite/PostgreSQL export formats. This enables an independently verified +recovery copy for D3/L8, while authority adoption, executor fencing and actual +Python writer retirement remain separate exits. + **T4 — collect full-writer retirement after durability cutover.** - The 2026-09-19 command audit retires two already-typed but unconsumed diff --git a/docs/architecture/rfcs/typescript-control-plane-migration-v0.zh-CN.md b/docs/architecture/rfcs/typescript-control-plane-migration-v0.zh-CN.md index 8b1428e72..2b65ca64c 100644 --- a/docs/architecture/rfcs/typescript-control-plane-migration-v0.zh-CN.md +++ b/docs/architecture/rfcs/typescript-control-plane-migration-v0.zh-CN.md @@ -822,6 +822,12 @@ T3/D1 reader,未完成全部 Todo writer、retention/compaction 或 promotion 配额准入与结算消费者现在从统一 Todo reader 读取完整来源,在显示压缩前解析显式 Todo 选择。它删除直接追加 Markdown 候选的路径,保留 promote 前的事件适配;promote 后权威为空或不可读都不能复活展示行。结算进度由现有 TS 回执链归约,Python 负责完整身份命令及 JSON/Markdown 展示。现有幂等 writer 可补齐缺失的 spend 回执而不再次扣款。这关闭已复现的 T3 消费者缺口,不代表 D1–D3、provider promotion 或剩余 Python 事务适配已完成。操作语义见[结算进度契约](../../quota-allocation.md#receipt-backed-settlement-progress)。 +**恢复边界(2026-09-22)。** [authority archive 命令](../../reference/authority-archive.md) +由现有 TS coordination owner 负责历史校验、状态 delta 重建和可重入恢复;Python +只解析 CLI 路径、传递请求并展示紧凑结果。复用 state-log codec,避免各 provider +分别实现导出格式或在 Python 再写一份状态规则。交付的是 D3/L8 的隔离恢复副本; +正式接管 authority、执行器围栏和旧 Python writer 退役仍有独立验收条件。 + **T4 — durable cutover 后兑现完整 writer 删除。** - 2026-09-19 命令审计退役两条已经 typed、但没有实际消费者的执行面: diff --git a/docs/reference/authority-archive.md b/docs/reference/authority-archive.md new file mode 100644 index 000000000..eb972a724 --- /dev/null +++ b/docs/reference/authority-archive.md @@ -0,0 +1,111 @@ +# Canonical authority archive and isolated recovery + +`loopx authority-archive` exports a pinned prefix of the selected coordination +AuthorityStore and restores that history into a separate File or SQLite store. +It preserves the full committed projections, events, operation ids and receipts, +including archived Todos and retained leases. It does not export the registry, +quota ledger, external artifacts, host sessions or the rest of a Goal's runtime. + +This is the recovery-artifact part of shared-authority D3/L8. It does not promote +a Goal, switch providers, release a writer fence, register a restored runtime, +or qualify the default local profile. The recovered lease records are historical +state, not a new execution grant. Canonical writes made after the captured cursor +are outside that archive and must be accounted for by a separate fenced cutover. +Never reactivate an older Markdown state as a rollback after canonical writes. + +## Export and verify + +Use the registry and runtime of the source Goal. Export is explicit and reads +the selected provider; a missing or invalid selector does not fall back to a +legacy display. The archive's parent directory must exist. An existing output +file is never replaced. + +```bash +loopx --registry ./registry.json --format json authority-archive export \ + --goal-id example-goal --archive ./authority.ndjson +loopx --format json authority-archive verify --archive ./authority.ndjson +``` + +Save `archive.archive_sha256` from the compact response for recovery review. +It identifies the canonical record hash chain and terminal seal, not the raw +file bytes: use `verify`, rather than `sha256sum` of the NDJSON file, for this value. +The archive is private state, written with mode `0600`; do not publish it as a +fixture or attach it to a public PR. Checksums detect corruption and bind the +reviewed content; they do not authenticate its author. Protect the digest and +archive together under the existing local filesystem trust boundary. + +The export pins the initial head cursor, provider revision and store identity. +It reads contiguous pages only through that cursor, independently reconstructs +the archive, checks the terminal head and identity, then publishes the completed +file without overwriting another output. Concurrent appends do not force a +restart. Missing history, source replacement and a rewritten captured head fail. +A process killed before publication may leave a private `.partial` sibling; +verify any completed output before deciding whether an interrupted export needs +to be repeated. + +## Restore an isolated copy + +The destination is a **new directory**, not a runtime root or a provider selector. +Preview validates the complete archive and reviewed goal/digest without creating +the destination. Execution creates a private binding manifest and a `store/` +subdirectory, then writes and reads back every retained transaction. + +```bash +loopx --format json authority-archive restore \ + --goal-id example-goal --archive ./authority.ndjson \ + --archive-sha256 --provider sqlite \ + --destination ./recovered-authority +# Repeat the same command with --execute to restore. +``` + +Use `--provider file` for an isolated File copy. The CLI deliberately does not +accept database credentials. The same TS archive/restore contract supports +PostgreSQL through a service-owned `AuthorityStore`; authentication, tenant +scope and a separate database incarnation remain the service's responsibility. +This is portable recovery, not a deployed PostgreSQL provider-switch feature. + +An interrupted restore can be repeated with the same archive, digest, goal, +provider and destination. Its existing prefix must match every operation, +event, receipt and historical projection. Extra or conflicting target commits +reject; recovery never overwrites them. A lost commit response is resolved by +exact journal and receipt readback. If directory creation was interrupted before +its binding manifest was durable, use a fresh destination; an unbound occupied +directory is intentionally not adopted. + +`verified-restore.json` records the archive digest, target store identity and +final target provider revision only after full readback. It is a historical +verification receipt, not a permanent claim that nobody changed the copy later. +The source physical provider revisions remain in the archive for provenance; +restored transactions receive the destination provider's own revision tokens. +Business operation identities and receipt payloads remain unchanged. + +The full archive is validated before restore writes begin and again during +replay. Modifying the input during recovery fails verification; any partial +result stays isolated and must not be adopted. A separate authority transition +must own executor fencing, target adoption and the accounting for later source +writes. There is no automatic selection or Markdown rollback here. + +## Format and validation scope + +The versioned NDJSON stream contains a header, ordered transaction records and +a terminal seal. A transaction stores an exact state delta using the same +`authority_state_log` codec as SQLite, plus events and receipts. The initial +delta reconstructs from the empty object; later records avoid repeating the +whole graph. Each record is hash-linked to its predecessor. Verification checks +strict fields, goal identity, positive contiguous cursors, unique operation ids, +state digests, source head revision and a seal followed by EOF. A valid checksum +alone cannot make a missing transaction or reordered history valid. + +Processing retains one reconstructed projection and the operation-id inventory, +not every full historical projection. Individual encoded records are limited to +64 MiB; this is an explicit archive format bound, not provider capacity evidence. +Supported-runtime, retention, crash/restore and elapsed-soak qualification for a +production profile remain separate. + +Public tests cover real File/SQLite round trips, PostgreSQL in both directions, +complex native/imported graphs, checkpoint-window history, corrupted/re-signed +invalid archives, concurrent source appends and interrupted recovery. CLI tests +also prove preview, repeated recovery, independent-process readback and rejection +of an occupied active-runtime destination. Frontend and Lark settings are +unchanged: this administrative command never changes their authority selection, +configuration owner or state projection. diff --git a/loopx/cli.py b/loopx/cli.py index d0e08d4a4..442f788c8 100644 --- a/loopx/cli.py +++ b/loopx/cli.py @@ -82,6 +82,7 @@ handle_bootstrap_connect_command, handle_canary_command, handle_coordination_shadow_command, + handle_authority_archive_command, handle_capability_command, handle_doctor_command, handle_dreaming_command, @@ -120,6 +121,7 @@ register_bootstrap_connect_command, register_canary_commands, register_coordination_shadow_command, + register_authority_archive_command, register_capability_commands, register_doctor_command, register_dreaming_commands, @@ -357,6 +359,7 @@ def build_parser() -> LoopXArgumentParser: register_explore_commands(sub, add_subcommand_format) register_todo_command(sub, add_subcommand_format) register_coordination_shadow_command(sub, add_subcommand_format) + register_authority_archive_command(sub, add_subcommand_format) register_task_lease_command(sub, add_subcommand_format) register_authority_shadow_command(sub, add_subcommand_format) register_todo_continuation(sub, add_subcommand_format) @@ -886,6 +889,13 @@ def main(argv: list[str] | None = None) -> int: if explore_result is not None: return explore_result + authority_archive_result = handle_authority_archive_command( + args, registry_path=registry_path, runtime_root_arg=args.runtime_root, + output_format=output_format, print_payload=print_payload, + ) + if authority_archive_result is not None: + return authority_archive_result + coordination_shadow_result = handle_coordination_shadow_command( args, registry_path=registry_path, diff --git a/loopx/cli_commands/__init__.py b/loopx/cli_commands/__init__.py index f3ed403fa..1a4a50975 100644 --- a/loopx/cli_commands/__init__.py +++ b/loopx/cli_commands/__init__.py @@ -45,6 +45,7 @@ def _load_exports() -> None: register_bootstrap_connect_command, ) from .canary import handle_canary_command, register_canary_commands + from .authority_archive import handle_authority_archive_command, register_authority_archive_command from .coordination_shadow import ( handle_coordination_shadow_command, register_coordination_shadow_command, @@ -169,6 +170,7 @@ def _load_exports() -> None: "handle_bootstrap_connect_command", "handle_canary_command", "handle_coordination_shadow_command", + "handle_authority_archive_command", "handle_capability_command", "handle_extension_command", "handle_check_command", @@ -240,6 +242,7 @@ def _load_exports() -> None: "register_bootstrap_connect_command", "register_canary_commands", "register_coordination_shadow_command", + "register_authority_archive_command", "register_capability_commands", "register_extension_commands", "register_doctor_command", diff --git a/loopx/cli_commands/authority_archive.py b/loopx/cli_commands/authority_archive.py new file mode 100644 index 000000000..52253176c --- /dev/null +++ b/loopx/cli_commands/authority_archive.py @@ -0,0 +1,63 @@ +"""Compact CLI transport for the TS-owned retained-authority archive contract.""" +from __future__ import annotations + +import argparse +from collections.abc import Callable +from pathlib import Path + +from ..control_plane.effect_runtime import effect_runtime_result +from ..paths import resolve_runtime_root +from ..history import load_registry + + +def register_authority_archive_command( + subparsers: argparse._SubParsersAction[argparse.ArgumentParser], + add_subcommand_format: Callable[[argparse.ArgumentParser], None], +) -> None: + parser = subparsers.add_parser( + "authority-archive", help="Export, verify or restore an isolated canonical authority copy." + ) + add_subcommand_format(parser) + actions = parser.add_subparsers(dest="authority_archive_action", required=True) + for name in ("export", "verify", "restore"): + action = actions.add_parser(name) + action.add_argument("--archive", type=Path, required=True) + if name != "verify": + action.add_argument("--goal-id", required=True) + if name == "restore": + action.add_argument("--destination", type=Path, required=True) + action.add_argument("--provider", choices=("file", "sqlite"), required=True) + action.add_argument("--archive-sha256", required=True) + action.add_argument("--execute", action="store_true", + help="Restore into a new isolated directory; otherwise preview.") + + +def handle_authority_archive_command( + args: argparse.Namespace, *, registry_path: Path, runtime_root_arg: str | None, + print_payload: Callable[[dict[str, object], str, Callable[[dict[str, object]], str]], None], + output_format: Callable[..., str], +) -> int | None: + if args.command != "authority-archive": + return None + request: dict[str, object] = { + "schema_version": "loopx_authority_archive_admin_request_v0", + "action": args.authority_archive_action, + "archive": str(args.archive.expanduser().resolve()), + } + if args.authority_archive_action == "export": + request.update(goal_id=args.goal_id, runtime_root=str(resolve_runtime_root( + load_registry(registry_path), runtime_root_arg, registry_path=registry_path))) + elif args.authority_archive_action == "restore": + request.update(goal_id=args.goal_id, destination=str(args.destination.expanduser().resolve()), + provider=args.provider, archive_sha256=args.archive_sha256, execute=args.execute) + try: + result = effect_runtime_result( + "coordination.authority_archive.manage", request, timeout=300.0, retry_safe=False + ) + except (RuntimeError, ValueError) as error: + result = {"status": "failed", "reason": str(error), "authority_changed": False} + print_payload(result, output_format(args), lambda value: ( + f"Authority archive: {value.get('status')}\n" + f"{value.get('reason', 'Active authority selection is unchanged.')}" + )) + return 1 if result.get("status") == "failed" else 0 diff --git a/loopx/control_plane/coordination/authority_archive.ts b/loopx/control_plane/coordination/authority_archive.ts new file mode 100644 index 000000000..d42c4d9de --- /dev/null +++ b/loopx/control_plane/coordination/authority_archive.ts @@ -0,0 +1,271 @@ +/** Portable retained-journal recovery. A restored store is an isolated copy, + * never an authority selection, writer-fence rollback or execution grant. */ +import {randomUUID} from "node:crypto"; +import {createReadStream} from "node:fs"; +import {link, open, unlink} from "node:fs/promises"; +import {dirname} from "node:path"; +import type {JsonObject} from "../effect_program.ts"; +import type {AuthorityStore, AuthorityStoreCommittedTransaction} from "./authority_store.ts"; +import {AuthorityStoreProtocolError, canonicalAuthorityObject, canonicalAuthorityObjectList, + canonicalAuthoritySha256, hasExactAuthorityKeys, requireAuthorityStoreId} from "./authority_store_codec.ts"; +import {applyAuthorityStateDelta, authorityStateDelta, decodeAuthorityStateDelta} from "./authority_state_log.ts"; + +const SCHEMA = "loopx_authority_archive_v0"; +const MAX_LINE_BYTES = 64 * 1024 * 1024; +const HEX = /^[0-9a-f]{64}$/; +interface ArchiveHeader extends JsonObject { + kind: "header"; + schema_version: typeof SCHEMA; + goal_id: string; + source_provider: string; + store_identity: string; + cursor: string; + provider_revision: string; + projection_sha256: string; +} +export interface AuthorityArchiveSummary { + schema_version: typeof SCHEMA; + goal_id: string; + source_provider: string; + source_store_identity: string; + source_provider_revision: string; + commits: string; + projection_sha256: string; + archive_sha256: string; +} +type ArchiveRecord = {kind: "header"; header: ArchiveHeader} | + {kind: "transaction"; transaction: AuthorityStoreCommittedTransaction} | + {kind: "verified"; summary: AuthorityArchiveSummary}; + +function invalid(reason: string): never { throw new AuthorityStoreProtocolError(reason); } +function positive(value: unknown): string { + if (typeof value !== "string" || !/^[1-9]\d*$/.test(value)) invalid("archive cursor must be a positive integer string"); + return value; +} +function hash(value: unknown): string { + if (typeof value !== "string" || !HEX.test(value)) invalid("archive digest must be SHA-256"); + return value; +} +function exact(value: JsonObject, keys: readonly string[]): void { + if (!hasExactAuthorityKeys(value, keys)) invalid("archive record has missing or unknown fields"); +} +function summary(header: ArchiveHeader, digest: string): AuthorityArchiveSummary { + return {schema_version: SCHEMA, goal_id: header.goal_id, source_provider: header.source_provider, + source_store_identity: header.store_identity, source_provider_revision: header.provider_revision, + commits: header.cursor, projection_sha256: header.projection_sha256, archive_sha256: digest}; +} +function signed(value: JsonObject): JsonObject { + return {...value, sha256: canonicalAuthoritySha256(value)}; +} +function unsigned(value: JsonObject): JsonObject { + const {sha256, ...body} = value; + if (hash(sha256) !== canonicalAuthoritySha256(body)) invalid("archive record digest mismatch"); + return body; +} + +/** Bound a single physical line before JSON parsing, without buffering history. */ +async function* lines(path: string): AsyncGenerator { + let chunks: Buffer[] = []; + let size = 0; + for await (const chunk of createReadStream(path)) { + const bytes = chunk as Buffer; + let start = 0; + while (start < bytes.length) { + const end = bytes.indexOf(10, start); + const piece = bytes.subarray(start, end === -1 ? bytes.length : end); + size += piece.length; + if (size > MAX_LINE_BYTES) invalid("archive record exceeds 64 MiB"); + chunks.push(piece); + if (end === -1) break; + // Fatal UTF-8 decoding prevents replacement characters silently changing data. + yield new TextDecoder("utf-8", {fatal: true}).decode(Buffer.concat(chunks, size)); + chunks = []; size = 0; start = end + 1; + } + } + if (size !== 0) invalid("archive is truncated: final newline missing"); +} + +/** One decoder owns ordering, state reconstruction and the terminal seal for + * verify and restore. Hashes detect corruption; they are not signatures. */ +async function* records(path: string): AsyncGenerator { + let header: ArchiveHeader | null = null; + let digest: string | null = null; + let cursor = 0n; + let state: JsonObject = {}; + let revision: string | null = null; + let sealed = false; + const operations = new Set(); + let verified: AuthorityArchiveSummary | null = null; + for await (const line of lines(path)) { + if (sealed) invalid("archive has content after its terminal seal"); + const raw = canonicalAuthorityObject(JSON.parse(line) as unknown, "archive record"); + const value = unsigned(raw); + if (header === null) { + exact(value, ["kind", "schema_version", "goal_id", "source_provider", "store_identity", + "cursor", "provider_revision", "projection_sha256"]); + if (value.kind !== "header" || value.schema_version !== SCHEMA || + !["file", "sqlite", "postgresql", "nokv"].includes(String(value.source_provider))) invalid("invalid archive header"); + header = {...value, kind: "header", schema_version: SCHEMA, + goal_id: requireAuthorityStoreId(value.goal_id, "archive goal id"), + source_provider: String(value.source_provider), store_identity: requireAuthorityStoreId(value.store_identity, "store identity"), + cursor: positive(value.cursor), provider_revision: requireAuthorityStoreId(value.provider_revision, "provider revision"), + projection_sha256: hash(value.projection_sha256)}; + yield {kind: "header", header}; + } else if (value.kind === "transaction") { + exact(value, ["kind", "cursor", "provider_revision", "operation_id", "events", "receipts", + "delta", "projection_sha256", "previous_sha256"]); + const nextCursor = positive(value.cursor); + if (value.previous_sha256 !== digest || BigInt(nextCursor) !== cursor + 1n || + BigInt(nextCursor) > BigInt(header.cursor)) invalid("archive transaction lineage mismatch"); + const operation = requireAuthorityStoreId(value.operation_id, "operation id"); + if (operations.has(operation)) invalid("archive operation id is duplicated"); + operations.add(operation); + state = applyAuthorityStateDelta(state, decodeAuthorityStateDelta(value.delta)); + if (state.goal_id !== header.goal_id) invalid("archive transaction belongs to another goal"); + if (canonicalAuthoritySha256(state) !== hash(value.projection_sha256)) invalid("archive state reconstruction mismatch"); + revision = requireAuthorityStoreId(value.provider_revision, "provider revision"); + cursor += 1n; + yield {kind: "transaction", transaction: {cursor: cursor.toString(), provider_revision: revision, + operation_id: operation, events: canonicalAuthorityObjectList(value.events, "events"), + receipts: canonicalAuthorityObjectList(value.receipts, "receipts"), projection: state}}; + } else if (value.kind === "seal") { + exact(value, ["kind", "cursor", "projection_sha256", "previous_sha256"]); + if (value.previous_sha256 !== digest || value.cursor !== header.cursor || cursor.toString() !== header.cursor || + revision !== header.provider_revision || value.projection_sha256 !== header.projection_sha256 || + canonicalAuthoritySha256(state) !== header.projection_sha256) invalid("archive seal does not cover its captured head"); + sealed = true; + verified = summary(header, hash(raw.sha256)); + } else invalid("unknown archive record kind"); + digest = hash(raw.sha256); + } + if (!sealed || verified === null) invalid("archive is incomplete: terminal seal missing"); + // Verification is delivered only after EOF, including the no-trailing-data check. + yield {kind: "verified", summary: verified}; +} + +export async function verifyAuthorityArchive(path: string): Promise { + for await (const record of records(path)) if (record.kind === "verified") return record.summary; + return invalid("archive verification did not finish"); +} + +/** Pin one retained prefix. Concurrent appends are allowed; a changed store + * identity, missing interval or rewritten captured head is not. Output is + * published without replacing an existing file, only after independent readback. */ +export async function exportAuthorityArchive(store: AuthorityStore, goalId: string, output: string, + options: {pageSize?: number} = {}): Promise { + requireAuthorityStoreId(goalId, "goal id"); + const pageSize = options.pageSize ?? 64; + if (!Number.isSafeInteger(pageSize) || pageSize < 1 || pageSize > 512) invalid("archive page size must be 1..512"); + const head = await store.loadAuthority(); + if (head.status !== "loaded") invalid("archive requires a committed source head"); + const identity = await store.storeIdentity(); + if (identity.status !== "available") invalid("archive source identity is unavailable"); + if (head.head.goal_id !== goalId) invalid("archive source goal mismatch"); + const header: ArchiveHeader = {kind: "header", schema_version: SCHEMA, goal_id: goalId, + source_provider: store.providerKind ?? "file", store_identity: identity.store_identity, + cursor: positive(head.cursor), provider_revision: head.provider_revision, + projection_sha256: canonicalAuthoritySha256(head.head)}; + const temporary = `${output}.${randomUUID()}.partial`; + const handle = await open(temporary, "wx", 0o600); + let closed = false; + try { + let digest = ""; + const append = async (value: JsonObject) => { + const record = signed(value); + const text = JSON.stringify(record) + "\n"; + if (Buffer.byteLength(text) > MAX_LINE_BYTES) invalid("archive record exceeds 64 MiB"); + await handle.writeFile(text); + digest = String(record.sha256); + }; + await append(header); + let after: string | null = null; + let state: JsonObject = {}; + let revision: string | null = null; + while (after !== header.cursor) { + const remaining = BigInt(header.cursor) - BigInt(after ?? "0"); + const limit = Number(remaining < BigInt(pageSize) ? remaining : BigInt(pageSize)); + const page = await store.scanCommitted(after, limit); + if (page.status !== "page" || page.transactions.length !== limit || + page.next_cursor !== page.transactions.at(-1)?.cursor) invalid("archive source scan is incomplete"); + for (const row of page.transactions) { + if (BigInt(positive(row.cursor)) !== BigInt(after ?? "0") + 1n) invalid("archive source cursor gap"); + await append({kind: "transaction", cursor: row.cursor, provider_revision: row.provider_revision, + operation_id: row.operation_id, events: row.events, receipts: row.receipts, + delta: authorityStateDelta(state, row.projection), projection_sha256: canonicalAuthoritySha256(row.projection), + previous_sha256: digest}); + state = row.projection; after = row.cursor; revision = row.provider_revision; + } + } + const finalIdentity = await store.storeIdentity(); + if (finalIdentity.status !== "available" || finalIdentity.store_identity !== identity.store_identity || + revision !== header.provider_revision || canonicalAuthoritySha256(state) !== header.projection_sha256) { + invalid("archive source lineage changed during capture"); + } + await append({kind: "seal", cursor: after, projection_sha256: header.projection_sha256, previous_sha256: digest}); + await handle.sync(); await handle.close(); closed = true; + const verified = await verifyAuthorityArchive(temporary); + await link(temporary, output); + const directory = await open(dirname(output), "r"); + try { await directory.sync(); } finally { await directory.close(); } + return verified; + } finally { + if (!closed) await handle.close(); + await unlink(temporary).catch(error => { if (error.code !== "ENOENT") throw error; }); + } +} + +function semanticTransaction(row: AuthorityStoreCommittedTransaction): JsonObject { + const {provider_revision: _revision, ...semantic} = row; + return semantic; +} + +/** Restore only into a caller-owned isolated store. CLI enforces a private + * destination manifest/lock. Existing prefixes must match every retained + * transaction; mismatches and extra writes reject without overwriting them. + * A crash leaves an explicitly incomplete copy, resumable with the same digest. */ +export async function restoreAuthorityArchive(path: string, target: AuthorityStore, + expectedArchiveSha256: string): Promise { + const verified = await verifyAuthorityArchive(path); + if (verified.archive_sha256 !== hash(expectedArchiveSha256)) invalid("restore archive differs from the reviewed digest"); + const identity = await target.storeIdentity(); + if (identity.status !== "available" || identity.store_identity === verified.source_store_identity) { + invalid("restore requires an independent available target lineage"); + } + const head = await target.loadAuthority(); + if (head.status !== "missing" && head.status !== "loaded") invalid("restore target is unavailable"); + if (head.status === "loaded" && BigInt(positive(head.cursor)) > BigInt(verified.commits)) invalid("restore target has extra commits"); + let previous: string | null = null; + let after: string | null = null; + let completed = false; + for await (const record of records(path)) { + if (record.kind === "header") continue; + if (record.kind === "verified") { + if (record.summary.archive_sha256 !== verified.archive_sha256) invalid("archive changed during restore; target remains isolated"); + completed = true; continue; + } + const row = record.transaction; + if (BigInt(row.cursor) > BigInt(head.status === "loaded" ? head.cursor : "0")) { + // The sole write attempt may commit and lose its response. Read the exact + // retained row below, never repeat the write or infer success from state alone. + try { + await target.commitAuthority({expected_provider_revision: previous, operation_id: row.operation_id, + events: row.events, next_projection: row.projection, receipts: row.receipts}); + } catch { /* exact journal readback is the recovery proof */ } + } + const readback = await target.scanCommitted(after, 1); + if (readback.status !== "page" || readback.transactions.length !== 1 || + canonicalAuthoritySha256(semanticTransaction(readback.transactions[0])) !== + canonicalAuthoritySha256(semanticTransaction(row))) invalid("restore target transaction differs or is unavailable; resume the same archive"); + const durable = readback.transactions[0]; + const receipt = await target.readReceipt(row.operation_id); + if (receipt.status !== "found" || receipt.cursor !== row.cursor || receipt.provider_revision !== durable.provider_revision || + canonicalAuthoritySha256(receipt.receipts) !== canonicalAuthoritySha256(row.receipts)) invalid("restore receipt readback mismatch"); + previous = durable.provider_revision; after = row.cursor; + } + const final = await target.loadAuthority(); + const finalIdentity = await target.storeIdentity(); + if (!completed || finalIdentity.status !== "available" || finalIdentity.store_identity !== identity.store_identity || + final.status !== "loaded" || final.cursor !== verified.commits || final.provider_revision !== previous || + canonicalAuthoritySha256(final.head) !== verified.projection_sha256) invalid("restore final readback mismatch"); + return {...verified, status: "restored", target_store_identity: identity.store_identity, target_provider_revision: final.provider_revision}; +} diff --git a/loopx/control_plane/coordination/local_authority_archive.ts b/loopx/control_plane/coordination/local_authority_archive.ts new file mode 100644 index 000000000..86253d46e --- /dev/null +++ b/loopx/control_plane/coordination/local_authority_archive.ts @@ -0,0 +1,65 @@ +/** Administrative archive transport. Large private state stays in local files; + * the managed effect runtime returns only compact integrity/readback facts. */ +import {mkdir, readFile} from "node:fs/promises"; +import {isAbsolute, join} from "node:path"; +import type {JsonObject} from "../effect_program.ts"; +import {durableWriteJson, withFileMutationLock} from "../effect_runtime_io.ts"; +import {requireJsonObject} from "../runtime_decode.ts"; +import {canonicalAuthoritySha256, requireAuthorityStoreId} from "./authority_store_codec.ts"; +import {exportAuthorityArchive, restoreAuthorityArchive, verifyAuthorityArchive} from "./authority_archive.ts"; +import {FileAuthorityStore} from "./file_authority_store.ts"; +import {SqliteAuthorityStore} from "./sqlite_authority_store.ts"; +import {openRuntimeAuthorityStore, requireLocalAuthorityRuntimeRoot, + type LocalAuthorityProviderDependencies} from "./local_authority_provider.ts"; + +function path(value: unknown, name: string): string { + if (typeof value !== "string" || !isAbsolute(value)) throw new Error(`${name} must be absolute`); + return value; +} + +export async function manageLocalAuthorityArchive(value: unknown, + dependencies: LocalAuthorityProviderDependencies = {}): Promise { + const base = {schema_version: "loopx_authority_archive_admin_v0", authority_changed: false, + legacy_fallback_used: false, execution_authority_granted: false}; + try { + const request = requireJsonObject(value, "authority archive request"); + if (request.schema_version !== "loopx_authority_archive_admin_request_v0") throw new Error("archive request schema mismatch"); + const archive = path(request.archive, "archive path"); + if (request.action === "verify") return {...base, status: "verified", archive: await verifyAuthorityArchive(archive)}; + const goalId = requireAuthorityStoreId(request.goal_id, "goal id"); + if (request.action === "export") { + const store = await openRuntimeAuthorityStore(requireLocalAuthorityRuntimeRoot(request.runtime_root), goalId, dependencies); + return {...base, status: "exported", archive: await exportAuthorityArchive(store, goalId, archive)}; + } + if (request.action !== "restore") throw new Error("unknown authority archive action"); + const inspected = await verifyAuthorityArchive(archive); + if (inspected.goal_id !== goalId || inspected.archive_sha256 !== request.archive_sha256) { + throw new Error("restore goal or reviewed archive digest mismatch"); + } + if (request.provider !== "file" && request.provider !== "sqlite") throw new Error("isolated local restore requires file or sqlite"); + const destination = path(request.destination, "restore destination"); + if (request.execute !== true) return {...base, status: "planned", archive: inspected, + provider: request.provider, requires_execute: true, destination_is_active: false}; + // A destination is a standalone recovery artifact, never a runtime root. + // Exclusive directory creation prevents adoption of an existing runtime or store. + const binding = {schema_version: "loopx_authority_restore_destination_v0", goal_id: goalId, + archive_sha256: inspected.archive_sha256, provider: request.provider}; + let created = false; + try { await mkdir(destination, {mode: 0o700}); created = true; } + catch (error) { if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; } + if (created) await durableWriteJson(join(destination, "restore-binding.json"), binding); + const actual = JSON.parse(await readFile(join(destination, "restore-binding.json"), "utf8")) as unknown; + if (canonicalAuthoritySha256(actual) !== canonicalAuthoritySha256(binding)) throw new Error("restore destination belongs to a different archive or provider"); + return await withFileMutationLock(join(destination, "restore"), async () => { + const store = request.provider === "file" ? new FileAuthorityStore(join(destination, "store"), goalId) + : new SqliteAuthorityStore(join(destination, "store"), goalId); + const restored = await restoreAuthorityArchive(archive, store, inspected.archive_sha256); + await durableWriteJson(join(destination, "verified-restore.json"), {...restored}); + return {...base, status: "restored", archive: restored, destination_is_active: false, + requires_separate_authority_cutover: true}; + }); + } catch (error) { + return {...base, status: "failed", reason_code: "authority_archive_failed", + reason: error instanceof Error ? error.message : "authority archive unavailable"}; + } +} diff --git a/loopx/control_plane/effect_runtime_handlers.ts b/loopx/control_plane/effect_runtime_handlers.ts index 4d509c101..d2bd79b72 100644 --- a/loopx/control_plane/effect_runtime_handlers.ts +++ b/loopx/control_plane/effect_runtime_handlers.ts @@ -1,3 +1,4 @@ +import {manageLocalAuthorityArchive} from "./coordination/local_authority_archive.ts"; import {selectPeriodicReportProgress, selectPeriodicReportApprovalRetry} from "./capabilities/periodic_report_progress.ts"; import {planIssueFixMonitorReconciliation} from "./capabilities/issue_fix_monitor_reconciliation.ts"; import {projectPeerOrchestration} from "./quota/peer_orchestration.ts"; @@ -531,6 +532,7 @@ export function createEffectRuntimeHandlers( ], ["coordination.runtime_shadow.rollback", rollbackCoordinationRuntimeShadow], ["coordination.local_authority.promote", promoteLocalCoordinationAuthority], + ["coordination.authority_archive.manage", manageLocalAuthorityArchive], ["coordination.local_authority.promotion_review", reviewLocalCoordinationAuthorityPromotion], ["coordination.local_authority.todo_continuation", continueLocalTodo], ["coordination.local_authority.todo_claim", claimLocalCoordinationTodo], diff --git a/tests/control_plane/test_authority_archive.py b/tests/control_plane/test_authority_archive.py new file mode 100644 index 000000000..d64308b94 --- /dev/null +++ b/tests/control_plane/test_authority_archive.py @@ -0,0 +1,83 @@ +"""Public administrative journey: snapshot, verify, preview, restore and reopen. + +The source is canonical fixture state; promotion itself is qualified separately. +""" +import json +import subprocess +import sys +from pathlib import Path + +import pytest +from canonical_authority_fixture import initialize_canonical_authority, isolate_sqlite_runtime + +REPO = Path(__file__).resolve().parents[2] + + +@pytest.mark.parametrize("provider", ["file", "sqlite"]) +def test_archive_cli_complete_isolated_recovery(tmp_path, monkeypatch, provider): + isolate_sqlite_runtime(tmp_path, monkeypatch) + runtime, registry, state = tmp_path / "runtime", tmp_path / "registry.json", tmp_path / "state.md" + goal = "archive-goal" + state.write_text("# Synthetic canonical projection\n") + registry.write_text(json.dumps({"common_runtime_root": str(runtime), "goals": [ + {"id": goal, "repo": str(tmp_path), "state_file": state.name}]})) + projection = {"goal_id": goal, "retained": [{"id": "archived", "lease_epoch": 7}], "text": "中文原文"} + initialize_canonical_authority(runtime, goal, projection, state_path=state, provider=provider) + registry_before, state_before = registry.read_bytes(), state.read_bytes() + # File bytes across the active tree prove no fence/selector/registry rewrite. + def files(): + return {str(p.relative_to(runtime)): p.read_bytes() for p in runtime.rglob("*") + if p.is_file() and p.suffix not in {"-wal", "-shm"} and not p.name.endswith(("-wal", "-shm"))} + before = files() + + def cli(*args, exit_code=0): + process = subprocess.run([sys.executable, "-m", "loopx.cli", "--registry", str(registry), + "--format", "json", "authority-archive", *args], + cwd=REPO, capture_output=True, text=True, timeout=90, check=False) + assert process.returncode == exit_code, process.stdout + process.stderr + return json.loads(process.stdout) + + archive = tmp_path / "backup.ndjson" + try: + exported = cli("export", "--goal-id", goal, "--archive", str(archive)) + assert exported["status"] == "exported" + assert not exported["authority_changed"] + verified = cli("verify", "--archive", str(archive)) + assert verified["archive"] == exported["archive"] + assert verified["archive"]["commits"] == "1" + occupied = cli("export", "--goal-id", goal, "--archive", str(archive), exit_code=1) + assert occupied["status"] == "failed" + for target_provider in ("file", "sqlite"): + destination = tmp_path / f"restore-{target_provider}" + arguments = ("restore", "--goal-id", goal, "--archive", str(archive), + "--archive-sha256", verified["archive"]["archive_sha256"], + "--destination", str(destination), "--provider", target_provider) + preview = cli(*arguments) + assert preview["status"] == "planned" and not destination.exists() + restored = cli(*arguments, "--execute") + assert restored["status"] == "restored" and not restored["destination_is_active"] + assert not restored["execution_authority_granted"] + assert restored["requires_separate_authority_cutover"] + assert cli(*arguments, "--execute")["archive"] == restored["archive"] + proof = json.loads((destination / "verified-restore.json").read_text()) + assert proof["archive_sha256"] == verified["archive"]["archive_sha256"] + assert proof["target_store_identity"] != proof["source_store_identity"] + # Reopen the actual restored backend in a separate process. + module = REPO / f"loopx/control_plane/coordination/{target_provider}_authority_store.ts" + class_name = "FileAuthorityStore" if target_provider == "file" else "SqliteAuthorityStore" + script = (f"import {{{class_name}}} from {json.dumps(module.as_uri())};" + f"const store=new {class_name}({json.dumps(str(destination / 'store'))},{json.dumps(goal)});" + "process.stdout.write(JSON.stringify(await store.loadAuthority()));") + readback = subprocess.run(["node", "--no-warnings", "--experimental-strip-types", "--input-type=module", "-e", script], + capture_output=True, text=True, timeout=30, check=True) + assert json.loads(readback.stdout)["head"] == projection + # An arbitrary occupied directory, including the active runtime, is never reused. + rejected = cli("restore", "--goal-id", goal, "--archive", str(archive), "--archive-sha256", + verified["archive"]["archive_sha256"], "--destination", str(runtime), + "--provider", provider, "--execute", exit_code=1) + assert rejected["status"] == "failed" + assert registry.read_bytes() == registry_before and state.read_bytes() == state_before + assert files() == before + finally: + subprocess.run([sys.executable, "-c", "from loopx.control_plane.effect_runtime import effect_runtime_result; effect_runtime_result('runtime.shutdown',{},retry_safe=False)"], + cwd=REPO, capture_output=True, text=True, timeout=30, check=True) diff --git a/tests/control_plane_ts/authority_archive.test.ts b/tests/control_plane_ts/authority_archive.test.ts new file mode 100644 index 000000000..7802be18f --- /dev/null +++ b/tests/control_plane_ts/authority_archive.test.ts @@ -0,0 +1,306 @@ +import type {AuthorityStore, AuthorityStoreCommit} from "../../loopx/control_plane/coordination/authority_store.ts"; +import {canonicalAuthoritySha256} from "../../loopx/control_plane/coordination/authority_store_codec.ts"; +import {productionScaleCoordinationFixture} from "./production_scale_coordination_fixture.ts"; +import assert from "node:assert/strict"; +import {mkdtemp, readFile, rm, writeFile} from "node:fs/promises"; +import {tmpdir} from "node:os"; +import {join} from "node:path"; +import test from "node:test"; +import {FileAuthorityStore} from "../../loopx/control_plane/coordination/file_authority_store.ts"; +import {SqliteAuthorityStore} from "../../loopx/control_plane/coordination/sqlite_authority_store.ts"; +import {exportAuthorityArchive, verifyAuthorityArchive, restoreAuthorityArchive} from + "../../loopx/control_plane/coordination/authority_archive.ts"; + +// Expectations come from the retained-journal contract: exact historical state, +// operation identities and receipts survive; physical revision tokens do not. +for (const sourceKind of ["file", "sqlite"] as const) { + test(`${sourceKind}: complete archive roundtrip preserves every transaction`, async () => { + const root = await mkdtemp(join(tmpdir(), "authority-archive-")); + try { + const source = sourceKind === "file" ? new FileAuthorityStore(join(root, "source"), "goal") + : new SqliteAuthorityStore(join(root, "source"), "goal"); + let revision: string | null = null; + for (let i = 1; i <= 7; i++) { + const result = await source.commitAuthority({expected_provider_revision: revision, + operation_id: `op-${i}`, events: [{kind: "change", i}], + next_projection: {goal_id: "goal", i, archived: ["todo_old"], unicode: "复杂目标"}, + receipts: [{request_sha256: `request-${i}`, changed: i % 2 === 0}]}); + assert.equal(result.status, "applied"); + if (result.status === "applied") revision = result.provider_revision; + } + const archive = join(root, "backup.ndjson"); + const result = await exportAuthorityArchive(source, "goal", archive, {pageSize: 2}); + assert.equal(result.commits, "7"); + assert.deepEqual(await verifyAuthorityArchive(archive), result); + for (const kind of ["file", "sqlite"] as const) { + const target = kind === "file" ? new FileAuthorityStore(join(root, kind), "goal") + : new SqliteAuthorityStore(join(root, kind), "goal"); + const restored = await restoreAuthorityArchive(archive, target, result.archive_sha256); + assert.equal(restored.status, "restored"); + assert.equal(restored.commits, "7"); + const original = await source.scanCommitted(null, 10); + const copy = await target.scanCommitted(null, 10); + assert.equal(original.status, "page"); assert.equal(copy.status, "page"); + if (original.status !== "page" || copy.status !== "page") throw new Error("scan failed"); + assert.deepEqual(copy.transactions.map(({provider_revision, ...t}) => t), + original.transactions.map(({provider_revision, ...t}) => t)); + assert.notEqual(copy.transactions[0].provider_revision, original.transactions[0].provider_revision); + assert.equal((await restoreAuthorityArchive(archive, target, result.archive_sha256)).status, "restored"); + } + } finally { await rm(root, {recursive: true, force: true}); } + }); +} + +test("missing source, occupied output, truncation and changed payload fail closed", async () => { + const root = await mkdtemp(join(tmpdir(), "authority-archive-")); + try { + const source = new FileAuthorityStore(join(root, "source"), "goal"); + const archive = join(root, "backup.ndjson"); + await assert.rejects(exportAuthorityArchive(source, "goal", archive)); + await source.commitAuthority({expected_provider_revision: null, operation_id: "op", events: [], + next_projection: {goal_id: "goal", value: 1}, receipts: [{done: true}]}); + await exportAuthorityArchive(source, "goal", archive); + const saved = await readFile(archive, "utf8"); + await assert.rejects(exportAuthorityArchive(source, "goal", archive)); + assert.equal(await readFile(archive, "utf8"), saved); + for (const [name, content] of [["truncated", saved.slice(0, saved.lastIndexOf('\n', saved.length - 2) + 1)], + ["changed", saved.replace('"done":true', '"done":false')], ["trailing", saved + '{}\n']]) { + const path = join(root, name); + await writeFile(path, content); + await assert.rejects(verifyAuthorityArchive(path)); + const target = new FileAuthorityStore(join(root, name + "-target"), "goal"); + await assert.rejects(restoreAuthorityArchive(path, target, "0".repeat(64))); + assert.equal((await target.loadAuthority()).status, "missing"); + } + } finally { await rm(root, {recursive: true, force: true}); } +}); + +// These adapters interrupt actual durable stores at a named effect boundary. +// They do not replace persistence with an in-memory implementation. + +function view(store: AuthorityStore, overrides: Partial): AuthorityStore { + return {providerKind: store.providerKind, storeIdentity: () => store.storeIdentity(), + loadAuthority: () => store.loadAuthority(), commitAuthority: c => store.commitAuthority(c), + readReceipt: id => store.readReceipt(id), scanCommitted: (cursor, limit) => store.scanCommitted(cursor, limit), + ...overrides}; +} +async function seed(store: AuthorityStore, count = 3) { + let previous: string | null = null; + for (let i = 1; i <= count; i++) { + const row = await store.commitAuthority({expected_provider_revision: previous, operation_id: `op-${i}`, + events: [{i}], receipts: [{decision: i}], next_projection: {goal_id: "goal", value: i}}); + assert.equal(row.status, "applied"); + if (row.status === "applied") previous = row.provider_revision; + } + return previous; +} + +test("capture pins its original prefix while real source receives later commits", async () => { + const root = await mkdtemp(join(tmpdir(), "authority-archive-")); + try { + const source = new SqliteAuthorityStore(join(root, "source"), "goal"); + const previous = await seed(source); + let appended = false; + const wrapped = view(source, {scanCommitted: async (after, limit) => { + if (!appended) { + appended = true; + assert.equal((await source.commitAuthority({expected_provider_revision: previous, operation_id: "later", + events: [], receipts: [], next_projection: {goal_id: "goal", value: 4}})).status, "applied"); + } + return source.scanCommitted(after, limit); + }}); + const report = await exportAuthorityArchive(wrapped, "goal", join(root, "archive"), {pageSize: 1}); + assert.equal(report.commits, "3"); + const head = await source.loadAuthority(); + assert.equal(head.status, "loaded"); if (head.status === "loaded") assert.equal(head.cursor, "4"); + assert.equal(report.projection_sha256, canonicalAuthoritySha256({goal_id: "goal", value: 3})); + } finally { await rm(root, {recursive: true, force: true}); } +}); + +for (const fault of ["gap", "missing-page", "identity", "changed-head"] as const) { + test(`capture rejects ${fault} without publishing an archive`, async () => { + const root = await mkdtemp(join(tmpdir(), "authority-archive-")); + try { + const source = new FileAuthorityStore(join(root, "source"), "goal"); + await seed(source); + let identityReads = 0; + const wrapped = view(source, { + storeIdentity: async () => fault === "identity" && identityReads++ > 0 + ? {status: "available", store_identity: "file:" + "f".repeat(32)} : source.storeIdentity(), + scanCommitted: async (after, limit) => { + const result = await source.scanCommitted(after, limit); + if (result.status === "page") { + if (fault === "gap") result.transactions[0].cursor = "9"; + if (fault === "missing-page") result.transactions = []; + if (fault === "changed-head" && result.transactions.at(-1)?.cursor === "3") { + result.transactions.at(-1)!.projection.value = 999; + } + } + return result; + }, + }); + const archive = join(root, "archive"); + await assert.rejects(exportAuthorityArchive(wrapped, "goal", archive, {pageSize: 1})); + await assert.rejects(readFile(archive), {code: "ENOENT"}); + assert.equal((await source.readReceipt("op-3")).status, "found"); + } finally { await rm(root, {recursive: true, force: true}); } + }); +} + +for (const kind of ["file", "sqlite"] as const) { + test(`${kind}: interrupted restore resumes the exact retained prefix; lost ack recovers`, async () => { + const root = await mkdtemp(join(tmpdir(), "authority-archive-")); + try { + const source = new FileAuthorityStore(join(root, "source"), "goal"); + await seed(source); + const archive = join(root, "archive"); + const report = await exportAuthorityArchive(source, "goal", archive); + const target = kind === "file" ? new FileAuthorityStore(join(root, "target"), "goal") + : new SqliteAuthorityStore(join(root, "target"), "goal"); + let interrupted = false; + const broken = view(target, { + commitAuthority: async c => { + const result = await target.commitAuthority(c); + if (c.operation_id === "op-2") { interrupted = true; throw new Error("process response lost"); } + return result; + }, + scanCommitted: async (after, limit) => interrupted + ? {status: "unavailable", reason_code: "stopped", reason: "stopped"} : target.scanCommitted(after, limit), + }); + await assert.rejects(restoreAuthorityArchive(archive, broken, report.archive_sha256)); + const partial = await target.loadAuthority(); + assert.equal(partial.status, "loaded"); if (partial.status === "loaded") assert.equal(partial.cursor, "2"); + const resumed = view(target, {commitAuthority: async c => { + await target.commitAuthority(c); throw new Error("lost ack with available readback"); + }}); + assert.equal((await restoreAuthorityArchive(archive, resumed, report.archive_sha256)).status, "restored"); + const end = await target.loadAuthority(); + assert.equal(end.status, "loaded"); if (end.status === "loaded") assert.equal(end.cursor, "3"); + } finally { await rm(root, {recursive: true, force: true}); } + }); +} + +test("restore rejects occupied divergent or longer target and changed reviewed digest", async () => { + const root = await mkdtemp(join(tmpdir(), "authority-archive-")); + try { + const source = new SqliteAuthorityStore(join(root, "source"), "goal"); + await seed(source); + const archive = join(root, "archive"); + const report = await exportAuthorityArchive(source, "goal", archive); + for (const mode of ["divergent", "longer", "digest", "source-itself"] as const) { + const target = mode === "source-itself" ? source : new SqliteAuthorityStore(join(root, mode), "goal"); + if (mode === "longer") await seed(target, 4); + if (mode === "divergent") await target.commitAuthority({expected_provider_revision: null, operation_id: "unrelated", + events: [], receipts: [], next_projection: {goal_id: "goal"}}); + const before = await target.loadAuthority(); + await assert.rejects(restoreAuthorityArchive(archive, target, mode === "digest" ? "0".repeat(64) : report.archive_sha256)); + assert.deepEqual(await target.loadAuthority(), before); + } + } finally { await rm(root, {recursive: true, force: true}); } +}); + +// Re-sign corrupted records to prove semantic checks independently of checksums. +function resign(rows: Record[]) { + let previous: string | null = null; + for (const row of rows) { + delete row.sha256; + if (row.kind !== "header") row.previous_sha256 = previous; + row.sha256 = canonicalAuthoritySha256(row); + previous = row.sha256 as string; + } + return rows.map(row => JSON.stringify(row)).join("\n") + "\n"; +} +for (const corruption of ["order", "duplicate", "head", "goal", "unknown-field", "missing-transaction", "missing-seal"] as const) { + test(`verified digests cannot hide ${corruption}`, async () => { + const root = await mkdtemp(join(tmpdir(), "authority-archive-")); + try { + const source = new FileAuthorityStore(join(root, "source"), "goal"); + await seed(source); + const archive = join(root, "archive"); + await exportAuthorityArchive(source, "goal", archive); + const rows = (await readFile(archive, "utf8")).trim().split("\n").map(line => JSON.parse(line)); + if (corruption === "order") [rows[1], rows[2]] = [rows[2], rows[1]]; + if (corruption === "duplicate") rows[2].operation_id = rows[1].operation_id; + if (corruption === "head") rows[0].provider_revision = "other"; + if (corruption === "goal") rows[0].goal_id = "other"; + if (corruption === "unknown-field") rows[1].ignored_payload = true; + if (corruption === "missing-transaction") rows.splice(2, 1); + if (corruption === "missing-seal") rows.pop(); + await writeFile(archive, resign(rows)); + await assert.rejects(verifyAuthorityArchive(archive)); + } finally { await rm(root, {recursive: true, force: true}); } + }); +} + +for (const schema of ["native", "legacy"] as const) { + test(`${schema}: mixed complete graph retains archived records, leases, ordering and unknown metadata`, async () => { + const root = await mkdtemp(join(tmpdir(), "authority-archive-")); + try { + const source = new SqliteAuthorityStore(join(root, "source"), "goal"); + const fixture = productionScaleCoordinationFixture("goal", schema); + const projection = {...fixture.projection, extra: JSON.parse('{"__proto__":{"keep":true},"":1}')}; + const first: AuthorityStoreCommit = {expected_provider_revision: null, operation_id: "import", events: [{kind: "import"}], + next_projection: projection, receipts: [{kind: "reviewed-import", unchanged: false}]}; + const result = await source.commitAuthority(first); + assert.equal(result.status, "applied"); + if (result.status !== "applied") throw new Error("seed failed"); + // A state-only copy would lose this first receipt and the exact older projection. + assert.equal((await source.commitAuthority({...first, expected_provider_revision: result.provider_revision, + operation_id: "observation", next_projection: {...projection, observed: true}, receipts: []})).status, "applied"); + const archive = join(root, "archive"); + const report = await exportAuthorityArchive(source, "goal", archive, {pageSize: 1}); + const target = new FileAuthorityStore(join(root, "target"), "goal"); + await restoreAuthorityArchive(archive, target, report.archive_sha256); + const rows = await target.scanCommitted(null, 2); + assert.equal(rows.status, "page"); + if (rows.status !== "page") throw new Error("readback failed"); + assert.deepEqual(rows.transactions[0].projection, projection); + assert.deepEqual(rows.transactions[1].projection, {...projection, observed: true}); + assert.deepEqual(rows.transactions[0].receipts, first.receipts); + // Delta archive should not retain the complete graph twice. + assert.ok((await readFile(archive)).length < JSON.stringify(projection).length * 1.4); + } finally { await rm(root, {recursive: true, force: true}); } + }); +} + +test("SQLite retained history crosses checkpoint windows and preserves an old same-key receipt", async () => { + const root = await mkdtemp(join(tmpdir(), "authority-archive-history-")); + try { + const source = new SqliteAuthorityStore(join(root, "source"), "goal"); + await seed(source, 70); + const archive = join(root, "archive"); + const report = await exportAuthorityArchive(source, "goal", archive, {pageSize: 7}); + assert.equal(report.commits, "70"); + const target = new SqliteAuthorityStore(join(root, "target"), "goal"); + await restoreAuthorityArchive(archive, target, report.archive_sha256); + const receipt = await target.readReceipt("op-1"); + assert.equal(receipt.status, "found"); + if (receipt.status === "found") assert.deepEqual(receipt.receipts, [{decision: 1}]); + const page = await target.scanCommitted("63", 3); + assert.equal(page.status, "page"); + if (page.status === "page") assert.deepEqual(page.transactions.map(row => row.projection.value), [64, 65, 66]); + } finally { await rm(root, {recursive: true, force: true}); } +}); + +test("a changed archive during the second pass cannot claim a verified recovery", async () => { + const root = await mkdtemp(join(tmpdir(), "authority-archive-change-")); + try { + const source = new FileAuthorityStore(join(root, "source"), "goal"); + await seed(source); + const archive = join(root, "archive"); + const report = await exportAuthorityArchive(source, "goal", archive); + const target = new SqliteAuthorityStore(join(root, "target"), "goal"); + const wrapped = view(target, {storeIdentity: async () => { + // Change a syntactically valid archive after initial full verification. + const rows = (await readFile(archive, "utf8")).trim().split("\n").map(line => JSON.parse(line)); + rows[1].receipts = [{changed: true}]; + await writeFile(archive, resign(rows)); + return target.storeIdentity(); + }}); + await assert.rejects(restoreAuthorityArchive(archive, wrapped, report.archive_sha256), /archive changed/); + // It remains an isolated recovery store; no source transaction was rewritten. + const receipt = await source.readReceipt("op-1"); + assert.equal(receipt.status, "found"); + if (receipt.status === "found") assert.deepEqual(receipt.receipts, [{decision: 1}]); + } finally { await rm(root, {recursive: true, force: true}); } +}); diff --git a/tests/control_plane_ts/authority_archive_postgresql.integration.test.ts b/tests/control_plane_ts/authority_archive_postgresql.integration.test.ts new file mode 100644 index 000000000..f9aacc1e4 --- /dev/null +++ b/tests/control_plane_ts/authority_archive_postgresql.integration.test.ts @@ -0,0 +1,69 @@ +import assert from "node:assert/strict"; +import {randomUUID} from "node:crypto"; +import {mkdtemp, rm} from "node:fs/promises"; +import {tmpdir} from "node:os"; +import {join} from "node:path"; +import test from "node:test"; +import {Pool} from "pg"; +import type {AuthorityStore} from "../../loopx/control_plane/coordination/authority_store.ts"; +import {FileAuthorityStore} from "../../loopx/control_plane/coordination/file_authority_store.ts"; +import {SqliteAuthorityStore} from "../../loopx/control_plane/coordination/sqlite_authority_store.ts"; +import {installPostgreSqlAuthorityStoreSchema, PostgreSqlAuthorityStore, + type PostgreSqlAuthorityDatabase} from "../../loopx/control_plane/coordination/postgresql_authority_store.ts"; +import {exportAuthorityArchive, restoreAuthorityArchive} from "../../loopx/control_plane/coordination/authority_archive.ts"; +import {productionScaleCoordinationFixture} from "./production_scale_coordination_fixture.ts"; + +const url = process.env.LOOPX_TEST_POSTGRES_URL; +for (const local of ["file", "sqlite"] as const) { + for (const direction of ["to-postgresql", "from-postgresql"] as const) { + test(`${local} ${direction}: portable complete graph and retained receipts`, {skip: !url}, async () => { + const pool = new Pool({connectionString: url, max: 3}); + const database: PostgreSqlAuthorityDatabase = {connect: async () => { + const client = await pool.connect(); + return {query: (sql, params) => client.query(sql, params ? [...params] : undefined), release: () => client.release()}; + }}; + const root = await mkdtemp(join(tmpdir(), "authority-archive-pg-")); + try { + await installPostgreSqlAuthorityStoreSchema(database, "postgresql:" + "b".repeat(32)); + const options = {tenant_id: `archive-${randomUUID()}`, goal_id: "goal"}; + const pg = new PostgreSqlAuthorityStore(database, options); + const file = local === "file" ? new FileAuthorityStore(join(root, "store"), "goal") + : new SqliteAuthorityStore(join(root, "store"), "goal"); + const [source, target]: [AuthorityStore, AuthorityStore] = direction === "to-postgresql" ? [file, pg] : [pg, file]; + const projection = productionScaleCoordinationFixture("goal", "native").projection; + let previous: string | null = null; + for (let i = 1; i <= 3; i++) { + const committed = await source.commitAuthority({expected_provider_revision: previous, + operation_id: `historical-${i}`, events: [{kind: "observation", i}], + next_projection: {...projection, observation: i}, receipts: [{request: `same-key-${i}`, changed: false}]}); + assert.equal(committed.status, "applied"); + if (committed.status === "applied") previous = committed.provider_revision; + } + const archive = join(root, "archive.ndjson"); + const exported = await exportAuthorityArchive(source, "goal", archive, {pageSize: 1}); + const restored = await restoreAuthorityArchive(archive, target, exported.archive_sha256); + assert.equal(restored.status, "restored"); + const reopened = direction === "to-postgresql" ? new PostgreSqlAuthorityStore(database, options) : target; + const rows = await reopened.scanCommitted(null, 10); + assert.equal(rows.status, "page"); + if (rows.status !== "page") throw new Error("readback failed"); + assert.equal(rows.transactions.length, 3); + for (const [index, row] of rows.transactions.entries()) { + assert.deepEqual(row.projection, {...projection, observation: index + 1}); + const receipt = await reopened.readReceipt(`historical-${index + 1}`); + assert.equal(receipt.status, "found"); + if (receipt.status === "found") assert.deepEqual(receipt.receipts, + [{request: `same-key-${index + 1}`, changed: false}]); + } + // Prove the restored native provider can continue its own CAS chain. + assert.equal((await reopened.commitAuthority({expected_provider_revision: restored.target_provider_revision, + operation_id: "isolated-recovery-probe", events: [], receipts: [{probe: true}], + next_projection: {...projection, observation: 4}})).status, "applied"); + await assert.rejects(restoreAuthorityArchive(archive, reopened, exported.archive_sha256), /extra commits/); + const unchanged = await source.loadAuthority(); + assert.equal(unchanged.status, "loaded"); + if (unchanged.status === "loaded") assert.equal(unchanged.cursor, "3"); + } finally { await pool.end(); await rm(root, {recursive: true, force: true}); } + }); + } +}