feat: production operability + data-integrity hardening — maintenance mode, durable ingest, fenced blob-reclaim GC & out-of-band graph-repair CLI (v6.7.3) by colombod · Pull Request #79 · microsoft/amplifier-context-intelligence

Diego Colombo (colombod) · 2026-08-23T20:32:16Z

What this delivers

Context Intelligence becomes operable in production and self-defending about data integrity — re-homed onto the backend-neutral storage layer (#81) and released as v6.7.3.

Operability (new)

Maintenance mode — an /admin/maintenance endpoint + gate middleware lets an operator quiesce the server for graph work. Auto-repair runs on boot only when this replica holds the writer lease (single-flight; re-arms the global gate; a schema failure no longer skips the retry sweep).
Out-of-band graph-rectification CLI — migrations/run.py --status|--apply diagnoses and repairs graph state without ever touching the request path, reusing the same run_repair/diagnose the server itself uses. Idempotent by construction — re-running --apply is a no-op.

Data integrity (hardening + fixes)

Durable ingest cursor + retry-dedup + run-id tiebreaker — the cursor restores off the idle path before the first real batch and survives a worker rebuild, so a crash or retry can no longer silently drop, duplicate, or reorder events. commit() now requires the cursor.
Session self-heal — IncompleteSession nodes heal forward on start/fork, with relabel scripts for existing graphs.
Schema-version subsystem, working_dir DB-level non-overwrite (a late empty write can't clobber a real working_dir), and a blob-carrier allowlist tripwire.

Safe cleanup (new, with guardrails)

Blob-reclaim GC (/admin/blobs/reclaim) — dry-run by default; a destructive apply is single-flighted (409 on a concurrent apply) and fenced (delete(uri, if_unmodified=ref) refuses if the blob changed since it was scanned). Concurrent operators can't over-delete, and a re-minted blob is never destroyed.

Storage stays backend-neutral — this extends the #81 boundary, never reaches around it

Boot-reclaim enumerates the queue via a new QueueManager.session_keys() protocol method — no directory globbing.
The writer lease moved behind a new lease_store/ package — the 4th storage backend alongside blob/queue/identity.
A standing AST guard test proves no module outside the four backend packages performs a storage-artifact file op or reads a storage root path — zero violations tree-wide.

Net result: a new backend (e.g. Azure) can be added with no consumer changes.

Verification — real, no mocks at the gate

Full non-neo4j suite: 2178 passed, 7 skipped.
Full neo4j suite (real containers): 109 passed, including two live-Neo4j end-to-end tests — blob-reclaim physically deletes an orphan file from disk while a referenced blob survives; the migration CLI rectifies a genuinely dirty graph. Every mock-based test is reconciled against real behaviour.

Stack & compatibility

#78 harden-durable-ingestion   (issue #473, v6.7.1)
  └─ #81 storage-api-on-78      (blob/queue/identity behind protocols)   ← this PR's base
       └─ this PR               (operability & data-integrity, v6.7.3)

A genuine transplant, not a rebase — #78 and #76 changed overlapping queue/blob APIs, so the work was gene-transferred onto the new protocols rather than replayed. #78's ensure_neo4j_schema(fail_on_data_conflict=True) and the schema_ready gate are untouched; #70's deploy-safe-boot behaviour was deliberately not brought. Merge order: #78 → #81 → this PR. Supersedes the work previously tracked in #70.

🤖 Generated with Amplifier

Co-Authored-By: Amplifier 240397093+microsoft-amplifier@users.noreply.github.com

…ain, safe boot, self-shrinking storage, operator GC) Hardens the event-ingestion server so it durably persists events under real shared-network-storage deployment conditions, recovers safely on restart, keeps its own on-disk storage bounded, and gives operators a safe reclaim tool. Public API contract is unchanged except for additive endpoints and /status fields. - Durable, corruption-free append: queue writes are atomic and single-writer-serialised per session under a per-key file lock, so concurrent/interrupted writes can no longer produce torn or merged log lines; a partial write is discarded and surfaced, never left to corrupt the tail. - Supervised draining: a dead drain worker is logged and its session recovered instead of silently stranded; poison/unparseable lines are dead-lettered and draining continues. - Safe fast boot: /status and /version answer from the first boot phase; pre-existing on-disk data is classified (resume-vs-remove), never crash-loops, and already-drained data is reclaimed at a bounded rate; boot progress is on /status.boot. - Self-shrinking queue storage: committed prefixes are reclaimed continuously as events drain (not only at session end); dead-letter files have bounded retention. - Writer-lease detector: detects two revisions briefly writing the same data directory during a rolling deploy and surfaces it on /status.writer_lease within a heartbeat (detector mode by default; never refuses boot). - Exactly-once idempotency fix: an idempotency key is recorded only after a durable write, so a failed write + client retry is honoured instead of being falsely refused as a duplicate. - Consistent graph-write concurrency: terminal graph flushes now use the same concurrency gate as all other writes. - Operator observability: previously-silent worker death/reap/cancel/lease-conflict/orphan transitions now emit structured, session-tagged log lines. - Operator storage GC: GET /queues/gc previews safe-to-delete fully-drained queue logs and expired dead-letters (deletes nothing, read scope); POST /queues/gc/apply performs a bounded, per-item re-verified deletion (write scope, refused until boot completes). Runs server-side — no disk or storage-key access needed. - Config: adds compaction/dead-letter-retention/GC/writer-lease settings; bounds the crash-recovery respawn cap and shortens the sweep interval. Deployment disk raised to 1 TB. Full test suite green (2070 non-Neo4j plus isolated-Neo4j), including adverse-state and crash-window cases. Docs updated (README API + settings, operational hardening runbook, Azure deployment, architecture overview + diagram). 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

The GC preview/apply HTTP endpoints were not needed to address the durability work and exposed a destructive operation on the data-plane router without a proper security design. Removed the two routes, the scan/candidate enumeration, their config/model, and their tests. The automatic in-loop reclaim (continuous compaction + dead-letter retention) is unchanged and needs no endpoint. On-demand reclamation of a drained backlog, if ever needed, is an out-of-band maintenance operation run inside the container, not an exposed API. Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

Deletes an Azure Files deployed-mount smoke harness (for a test that is deliberately not run) and a throughput benchmark; neither is a correctness test for this change. Rewrites the comments and docstrings added by this branch into proper documentation -- describing what each test and code path does and why -- instead of tracing the internal design discussion. Comment/docstring and test-file-set changes only; no product logic changed. Full non-Neo4j suite green (2046 passed); ruff format/lint clean. Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

Cuts verbose explanatory comments down to what a reviewer needs: removes internal codenames, source-location references, change-history narration, and future-work notes; rewrites the two Neo4j reclaim/flush test files with concise docstrings. Comment/docstring text only -- no logic, names, or behaviour changed. Full non-Neo4j suite green (2046 passed); ruff format/lint clean. Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

Removes ~3600 lines of narrative/incident-history/codename/spec-tracking prose from comments and docstrings across the server and its tests, leaving terse purpose docstrings and only the load-bearing invariant/why notes a reader cannot infer from the code. Comment and docstring text only -- no logic, symbol names, string literals, assertions, or behaviour changed. Full non-Neo4j suite green (2046 passed); ruff format/lint clean; type:ignore/noqa pragmas preserved. Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

Bumps context-intelligence-server from 6.7.0 to 6.7.2 to reflect the durable ingestion hardening work on this branch: - Crash-safe queue - Supervised drain - Safe boot This new build is already installed and running as the live systemd daemon. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

…rrency - offset reader accepts both the legacy JSON and bare-int forms; commit() unchanged; an unusable offset never silently re-drains from 0 - per-key idempotency lock: concurrent same-key POSTs append exactly once, store-after-append ordering preserved - exhausted poison batch containing session:end now finalizes (CompletedSession + delete_drained); no leaked drained log - compaction reclaims the committed prefix regardless of tail size; remove the tail-size cap - boot auto-reclaims provably-drained logs; an unreadable offset never deletes an intact log - partial-write rollback never truncates the queue - writer lease defaults to enforce, releases on clean shutdown, takes over a stale lease - add concurrent-append stress coverage (multi/single-session, >1MiB records) 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

- writer lease documented as enforce-by-default single-writer guard (refuses a live foreign writer, releases on clean shutdown, takes over a stale lease); drop the detect-default / "do not enable enforce" guidance - remove references to the removed /queues/gc operator endpoints - fully-drained logs are reclaimed automatically at boot; unresumable/reset-offset actions remain gated on reclaim_enabled - correct offset handling: an unparseable offset re-drains from byte 0 at any size, never deleted - drop the removed compaction tail-size cap; regenerate 05-durable-ingest-queue.png 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

…recover finalize-orphans; keep dead-letters by default - boot serves /status immediately even when Neo4j is unreachable at startup: schema init moves into the guarded background boot as a retried phase, drainer start is gated on schema-ready, and an un-migrated graph is a visible failed boot rather than an ASGI-startup abort - per-phase boot timeout (boot_phase_timeout_seconds, default 300s) so a hung mount fails visibly instead of leaving /status.spool/metrics null forever - a finalize-orphan (tail-flush failure) now closes and deregisters the worker so a fresh event re-drains from the retained log, instead of a registered zombie only an opt-in sweep could recover - dead-letter auto-expiry ships off by default so an un-recovered event's last copy is never silently deleted 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

…on /status A prior change deregistered and closed the worker on the two finalize-orphan paths (tail-flush failure). That broke the deliberate orphan-visibility design: a finalize-path orphan must stay registered so orphaned_sessions() surfaces it on /status (orphaned: true) and boot recover() re-enters it after restart. Restore the original behavior on both paths (no _safe_close / _deregister on orphan) and revert the unit tests that had been flipped to assert deregistration. Regression was only caught by the memory-capped Neo4j integration tests (tests/neo4j), which now pass. Also give the deterministic ~30s OOM recipe test explicit timeout headroom so it does not trip the global 30s per-test timeout during teardown. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

…seat of #79 onto storage-API layer) Re-seat of PR #70's ingest-correctness work (originally #79 commit b946d5d) onto the storage-API-isolation layer. The durable per-record cursor is now persisted atomically with the offset ({"v":1,"offset","cursor"}); retry-dedup and the run-id tiebreaker on parallel handlers carry over unchanged. Seam adaptation for the storage-API layer: - The queue cursor methods (_write_offset_record, _read_offset_record, _read_committed_offset, commit(session_id, offset, cursor), read_cursor, is_fully_drained) land in the FileSystemQueueManager package (queue_manager/filesystem.py) rather than the former flat module; method bodies are byte-identical to #79's originals (verified by diff). - commit() gains a required cursor argument; all four registry.py call sites pass worker.services.snapshot_cursor(), and the whole tree was swept to confirm no 2-arg caller remains. - The offset record stays forward- and backward-compatible: the layer's existing tolerant reader extracts "offset" from a v1 record and ignores the cursor, so a rollback to the prior build reads the committed position correctly. Verified: 219 queue/cursor/services tests + 39 neo4j durability/tiebreaker tests pass. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

#79) Re-seat of PR #70's session-recovery work (originally #79 commit 7059e4b) onto the storage-API layer. Sessions left Incomplete by an earlier crash are healed forward on the next start/fork, and two graph-backfill scripts (relabel_incomplete_sessions, tag_legacy_pooled_iterations) rectify legacy state. Clean cherry-pick: no overlap with the storage-API seam. The two scripts are graph-only (Cypher) — no storage-artifact file operations, consistent with the storage-agnosticism rule. Verified: 154 session-handler tests + 14 neo4j heal-forward/relabel/tag tests pass. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

…rrier-allowlist (re-seat of #79) Re-seat of PR #70's schema/data-integrity work (originally #79 commit 236fa69) onto the storage-API layer: - BLOB_REF_CARRIER_PROPERTIES in blob_processor is the single source of truth for which graph properties may carry a ci-blob:// reference; validated at import. - working_dir is never silently overwritten once set. - SCHEMA_VERSION marker + drift reporting (reported, not destructively enforced). Seam adaptation: blob_processor.py reconciled as disjoint regions — the layer's write()->BlobReference mint line and #79's carrier-allowlist block coexist untouched. No storage-artifact file operations introduced. The carrier-allowlist end-to-end test (tests/test_blob_carrier_allowlist.py) imports routers.admin's reference-scan and lands with the maintenance/reclaim commit that introduces admin.py. Verified: 42 status/blob_processor tests + 7 neo4j schema-version/working_dir tests pass. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

…tral writer lease (re-seat of #79) Re-seat of PR #70's maintenance/reclaim work (originally #79 commit 6b3a6a0) onto the storage-API layer, with the storage-agnosticism gaps it exposed fixed rather than carried: - Blob-reclaim GC (/admin/blobs/reclaim) is now 100% protocol-based: scan() for discovery, BlobReference (no Path), fenced delete(uri, if_unmodified=ref). The raw glob / os.unlink / Path(settings.blob_path) / _OnDiskBlob are gone. - The destructive apply is single-flighted: a second concurrent apply is refused (409) before it scans, so two applies can never jointly exceed one operator's max_delete blast radius. dry-run is never blocked. - Boot reclaim enumerates the queue through a new QueueManager.session_keys() protocol method instead of globbing the queue directory, so the sweep works unchanged against any queue backend. - The writer-lease detector no longer does raw file I/O: lease persistence moves behind a new lease_store backend (protocol + filesystem + factory), the fourth storage backend alongside blob/queue/identity. The detector keeps its bounded single-thread I/O executor and reaches the lease only through the store. - A standing AST guard test asserts no module outside the four storage backend packages performs a storage-artifact file operation or reads a storage root path; it is a best-effort tripwire (proven red on a planted leak), not the proof. - Maintenance mode (gate + lease-armed auto-repair + /admin/maintenance), carrier-allowlist end-to-end test, and /status re-seat carry over from #79. Verified: 474 non-neo4j + 12 neo4j tests pass; the storage-boundary guard reports zero violations across the tree. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

Re-seat of PR #70's migration CLI (originally #79 commit 8e7d9a0) onto the storage-API layer. `migrations/run.py` is an operator tool to diagnose (`--status`) or rectify (`--apply`) graph state out of band, reusing the SAME `run_repair` the server's `/admin/maintenance` endpoint and `doctor --fix` call (dedup -> :Node backfill -> constraint create). Safety by construction rather than by manifest: - The rectification is stateless and idempotent (all IF NOT EXISTS / MERGE), so a run killed midway is completed simply by re-running -- there is no manifest to tear and no resume state to corrupt. - It is a structural rectification only; SCHEMA_VERSION stays 1 -> 1, so a partial run can never leave status/version reporting a phantom-advanced version. - Graph-only (Cypher through the driver); no storage-artifact file operations. Verified: 13 CLI tests pass; the real-neo4j repair path is covered by tests/neo4j/test_node_identity_migration.py (3 passed). 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

Final version of the stacked re-seat. #78 moved the server to 6.7.1; the storage-API layer holds it; this stacked head lands at 6.7.3. Adds the machine-readable migration manifest with the 6.7.3-maintenance-mode entry (forward-only structural rectification, no rollback machinery). SCHEMA_VERSION stays 1 -- the change is structural/operational, not a stored-shape change. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

The drain path built a new AsyncGraphDatabase driver for every session_id with no pool bound, so bolt connections accumulated without limit until the server's bolt thread pool starved and ingest backpressured to clients. SessionRegistry now builds one shared, pool-bounded driver (lazily, on first session) and hands it to every Neo4jGraphStore it constructs. Neo4jGraphStore accepts an optional pre-built driver and tracks whether it owns it; close() only closes a driver it owns, so a per-session finalize can never take down the driver other live sessions are still using. The shared driver itself is closed exactly once, at lifespan shutdown. The pool-bounding kwargs (max_connection_pool_size, max_connection_lifetime) live in one helper in neo4j_store.py so the lifespan admin driver, the doctor CLI, and the registry's shared driver can never diverge -- this also avoids a registry->main import cycle, since main already imports registry. Config gains neo4j_max_connection_pool_size (default 50, well under the server's bolt thread-pool size) and neo4j_max_connection_lifetime (default 3600s, so idle connections recycle). 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

Route the process-wide cypher-query driver through the same bounded-driver helper as the admin and per-session drivers, so every process-wide pool shares one cap instead of leaving the query driver on an unbounded default pool. Driver construction now funnels entirely through the store module, so lifespan tests target that single construction site. Add evidence that the per-session driver leak is gone: structural tests prove N sessions build exactly one shared driver, that it is built with the bounded pool kwargs, that concurrent first-sessions cannot race into a second build, and that the shared driver is reclaimed exactly once and is idempotent to re-close. A live-Neo4j test drives many sessions through the real registry write path and asserts, via the server's own connection list, that open bolt connections stay bounded by the pool and are released on driver close. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

…ackend-neutral protocols Transplant PR #76's storage-API isolation onto PR #78's hardened durable queue so the two stack cleanly (78 -> 76). Every store is now reached only through a backend-neutral Protocol + factory; no consumer constructs a concrete backend or touches an on-disk path. queue_manager/ FileSystemQueueManager implements the QueueManager Protocol. Built on #78's authoritative body (the durable per-record cursor model): the class body is byte-identical to #78's queue_manager.py apart from the class rename and its self- references. protocol.py carries #78's Record/Batch verbatim; factory.create_queue_manager is the only queue backend selector. blob_store/ FileSystemBlobStore implements the BlobStore Protocol. write() returns a BlobReference (uri + size + last_modified); the store gains scan()/list() (async BlobReference iterators) and a fenced delete(uri, if_unmodified=ref). Adds settings.blob_backend. identity_store/ FileSystemIdentityStore implements the IdentityStore Protocol; the commit-order and fail-closed-load contract lives in the protocol. The backing path is private -- callers use exists(). Also folds in the blob-key fix: process_event includes tool_call_id in the blob-key node_id, so two same-millisecond parallel events no longer collide on one blob and silently overwrite each other. Consumers (registry, main, blob_processor, pipeline) build stores via the factories and pass URIs/references, never paths. The boot-reclaim dry-run log derives the blob path from the QueueManager's own queues_dir (fixing a latent mismatch when it differs from settings.queues_path); the identity first-boot warnings no longer echo the store path. The sole remaining config-path read outside the storage layer is registry.queues_dir_path -- the resolver the WriterLease boot detector uses precisely because it must not construct a QueueManager. Full non-neo4j suite: 2082 passed. Neo4j subsets (queue durability, blob ingest, identity auth): green. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

…seat of #79 onto storage-API layer) Re-seat of PR #70's ingest-correctness work (originally #79 commit b946d5d) onto the storage-API-isolation layer. The durable per-record cursor is now persisted atomically with the offset ({"v":1,"offset","cursor"}); retry-dedup and the run-id tiebreaker on parallel handlers carry over unchanged. Seam adaptation for the storage-API layer: - The queue cursor methods (_write_offset_record, _read_offset_record, _read_committed_offset, commit(session_id, offset, cursor), read_cursor, is_fully_drained) land in the FileSystemQueueManager package (queue_manager/filesystem.py) rather than the former flat module; method bodies are byte-identical to #79's originals (verified by diff). - commit() gains a required cursor argument; all four registry.py call sites pass worker.services.snapshot_cursor(), and the whole tree was swept to confirm no 2-arg caller remains. - The offset record stays forward- and backward-compatible: the layer's existing tolerant reader extracts "offset" from a v1 record and ignores the cursor, so a rollback to the prior build reads the committed position correctly. Verified: 219 queue/cursor/services tests + 39 neo4j durability/tiebreaker tests pass. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

#79) Re-seat of PR #70's session-recovery work (originally #79 commit 7059e4b) onto the storage-API layer. Sessions left Incomplete by an earlier crash are healed forward on the next start/fork, and two graph-backfill scripts (relabel_incomplete_sessions, tag_legacy_pooled_iterations) rectify legacy state. Clean cherry-pick: no overlap with the storage-API seam. The two scripts are graph-only (Cypher) — no storage-artifact file operations, consistent with the storage-agnosticism rule. Verified: 154 session-handler tests + 14 neo4j heal-forward/relabel/tag tests pass. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

…rrier-allowlist (re-seat of #79) Re-seat of PR #70's schema/data-integrity work (originally #79 commit 236fa69) onto the storage-API layer: - BLOB_REF_CARRIER_PROPERTIES in blob_processor is the single source of truth for which graph properties may carry a ci-blob:// reference; validated at import. - working_dir is never silently overwritten once set. - SCHEMA_VERSION marker + drift reporting (reported, not destructively enforced). Seam adaptation: blob_processor.py reconciled as disjoint regions — the layer's write()->BlobReference mint line and #79's carrier-allowlist block coexist untouched. No storage-artifact file operations introduced. The carrier-allowlist end-to-end test (tests/test_blob_carrier_allowlist.py) imports routers.admin's reference-scan and lands with the maintenance/reclaim commit that introduces admin.py. Verified: 42 status/blob_processor tests + 7 neo4j schema-version/working_dir tests pass. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

…tral writer lease (re-seat of #79) Re-seat of PR #70's maintenance/reclaim work (originally #79 commit 6b3a6a0) onto the storage-API layer, with the storage-agnosticism gaps it exposed fixed rather than carried: - Blob-reclaim GC (/admin/blobs/reclaim) is now 100% protocol-based: scan() for discovery, BlobReference (no Path), fenced delete(uri, if_unmodified=ref). The raw glob / os.unlink / Path(settings.blob_path) / _OnDiskBlob are gone. - The destructive apply is single-flighted: a second concurrent apply is refused (409) before it scans, so two applies can never jointly exceed one operator's max_delete blast radius. dry-run is never blocked. - Boot reclaim enumerates the queue through a new QueueManager.session_keys() protocol method instead of globbing the queue directory, so the sweep works unchanged against any queue backend. - The writer-lease detector no longer does raw file I/O: lease persistence moves behind a new lease_store backend (protocol + filesystem + factory), the fourth storage backend alongside blob/queue/identity. The detector keeps its bounded single-thread I/O executor and reaches the lease only through the store. - A standing AST guard test asserts no module outside the four storage backend packages performs a storage-artifact file operation or reads a storage root path; it is a best-effort tripwire (proven red on a planted leak), not the proof. - Maintenance mode (gate + lease-armed auto-repair + /admin/maintenance), carrier-allowlist end-to-end test, and /status re-seat carry over from #79. Verified: 474 non-neo4j + 12 neo4j tests pass; the storage-boundary guard reports zero violations across the tree. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

Re-seat of PR #70's migration CLI (originally #79 commit 8e7d9a0) onto the storage-API layer. `migrations/run.py` is an operator tool to diagnose (`--status`) or rectify (`--apply`) graph state out of band, reusing the SAME `run_repair` the server's `/admin/maintenance` endpoint and `doctor --fix` call (dedup -> :Node backfill -> constraint create). Safety by construction rather than by manifest: - The rectification is stateless and idempotent (all IF NOT EXISTS / MERGE), so a run killed midway is completed simply by re-running -- there is no manifest to tear and no resume state to corrupt. - It is a structural rectification only; SCHEMA_VERSION stays 1 -> 1, so a partial run can never leave status/version reporting a phantom-advanced version. - Graph-only (Cypher through the driver); no storage-artifact file operations. Verified: 13 CLI tests pass; the real-neo4j repair path is covered by tests/neo4j/test_node_identity_migration.py (3 passed). 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

Final version of the stacked re-seat. #78 moved the server to 6.7.1; the storage-API layer holds it; this stacked head lands at 6.7.3. Adds the machine-readable migration manifest with the 6.7.3-maintenance-mode entry (forward-only structural rectification, no rollback machinery). SCHEMA_VERSION stays 1 -- the change is structural/operational, not a stored-shape change. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

… filesystem No mocks: writes real blobs to a real BlobStore, references one in a live graph, runs the actual reclaim endpoint dry-run then apply, and asserts the orphan file is physically deleted from disk while the referenced blob survives. Proves the reshaped reference-scan (Cypher), orphan selection (real QueueManager drain state), and fenced BlobStore.delete for real -- backing the mock-based unit test of the same endpoint. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

No mocked driver: the migration CLI builds its own real driver from --neo4j-* flags, runs --status (read-only) then --apply against a genuinely dirty graph (duplicate untagged nodes + a legacy :Session), and the graph is actually rectified (untagged -> 0, duplicate merged, :Node constraint present). Re-running --apply is a real idempotent no-op. Reconciles the mocked-driver unit test in tests/test_migrations_run.py against real behaviour. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>

Diego Colombo (colombod) · 2026-08-26T11:56:22Z

Being split into smaller, independently-reviewable PRs for the review/merge process. PR1 (ingest correctness) is now open at #85 — see that PR for the first layer of the split. This monolithic PR will be closed as the split (PR1..PR6) lands.

Diego Colombo (colombod) and others added 10 commits August 21, 2026 21:39

Diego Colombo (colombod) force-pushed the stack/pr70-on-pr78 branch from d7b35a8 to 8c82734 Compare August 24, 2026 18:41

Diego Colombo (colombod) changed the base branch from harden-durable-ingestion to storage-api-on-78 August 24, 2026 18:41

Diego Colombo (colombod) changed the title ~~Stack: transplant PR#70 value onto PR#78 (durable-ingestion) at 6.7.3~~ feat: operability & data-quality — maintenance mode, durable ingest cursor, protocol-based blob reclaim & out-of-band graph-rectification CLI (v6.7.3) Aug 24, 2026

Diego Colombo (colombod) mentioned this pull request Aug 24, 2026

fix: persist agent name on sub-session nodes across a delivery-order race (v6.7.4) #83

Closed

Diego Colombo (colombod) and others added 3 commits August 25, 2026 13:29

Diego Colombo (colombod) force-pushed the storage-api-on-78 branch from e53acb7 to e8954df Compare August 26, 2026 10:43

Diego Colombo (colombod) and others added 3 commits August 26, 2026 10:53

Diego Colombo (colombod) and others added 5 commits August 26, 2026 10:54

Diego Colombo (colombod) force-pushed the stack/pr70-on-pr78 branch from 8c82734 to fc1fc0f Compare August 26, 2026 10:58

Diego Colombo (colombod) mentioned this pull request Aug 26, 2026

PR1: Ingest correctness — durable per-record cursor + retry-dedup + run-id tiebreaker #85

Closed

Diego Colombo (colombod) marked this pull request as draft August 26, 2026 11:56

This was referenced Aug 26, 2026

PR2: Session-node data quality — working_dir + agent + heal-forward #87

Closed

PR3: Schema-version observability #88

Closed

PR4: Storage-protocol hygiene & guard #89

Closed

PR6: Blob-storage reclaim GC #90

Closed

Diego Colombo (colombod) force-pushed the storage-api-on-78 branch from e8954df to 301e7ef Compare August 26, 2026 17:06

Diego Colombo (colombod) closed this Sep 1, 2026

Navigation Menu

Uh oh!

feat: production operability + data-integrity hardening — maintenance mode, durable ingest, fenced blob-reclaim GC & out-of-band graph-repair CLI (v6.7.3) - #79

feat: production operability + data-integrity hardening — maintenance mode, durable ingest, fenced blob-reclaim GC & out-of-band graph-repair CLI (v6.7.3)#79
Diego Colombo (colombod) wants to merge 21 commits into
storage-api-on-78from
stack/pr70-on-pr78

Diego Colombo (colombod) commented Aug 23, 2026 •

edited

Loading

Uh oh!

Diego Colombo (colombod) commented Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Diego Colombo (colombod) commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this delivers

Operability (new)

Data integrity (hardening + fixes)

Safe cleanup (new, with guardrails)

Storage stays backend-neutral — this extends the #81 boundary, never reaches around it

Verification — real, no mocks at the gate

Stack & compatibility

Uh oh!

Diego Colombo (colombod) commented Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Diego Colombo (colombod) commented Aug 23, 2026 •

edited

Loading