feat: production operability + data-integrity hardening — maintenance mode, durable ingest, fenced blob-reclaim GC & out-of-band graph-repair CLI (v6.7.3)

AGENTS.md

@@ -82,19 +82,22 @@ local runs. Scope is intentionally narrow — do this and nothing else.
 context_intelligence_server/      # FastAPI ingestion server
 ├── main.py                       # App factory, routes, lifespan
 ├── config.py                     # Settings (YAML + env vars via Pydantic)
-├── queue_manager.py              # Durable per-session append-log (persist-then-202)
-├── registry.py                   # Per-session drainers (drain_worker, write semaphore, retry/dead-letter)
+├── queue_manager.py              # Durable per-session append-log (persist-then-202); per-key _KeyGuard file lock (atomic single-writer append), compaction, dead-letter expiry, GC candidate scan
+├── registry.py                   # Per-session drainers (drain_worker, write semaphore, retry/dead-letter, drain-task done-callback supervision)
 ├── pipeline.py                   # Per-event dispatch spine (invoked by the drainer)
 ├── neo4j_store.py                # Managed-transaction Neo4j writes
 ├── blob_store.py                 # Async disk blob storage
+├── idempotency.py                # Request dedupe (seen/store split — a key is stored only AFTER a durable append)
+├── writer_lease.py               # Single-writer lease guard for the queue directory (heartbeat + staleness; enforce/detect/off, default enforce; surfaces on /status.writer_lease)
 ├── handlers/                     # Event handlers (data_layer_1/2/3)
 │   ├── data_layer_1/             # Session/tool-call handlers
 │   ├── data_layer_2/             # Graph enrichment handlers
 │   └── data_layer_3/             # High-level insight handlers
-├── routers/                      # API routers (queues.py = dead-letter inspect/replay/purge; admin.py = /admin/* identity-map CRUD)
+├── routers/                      # API routers (queues.py = dead-letter inspect/replay/purge only \u2014 no operator GC endpoint; admin.py = /admin/* identity-map CRUD; version.py)
 ├── auth.py                       # Bearer-token auth middleware (StaticKeyResolver / EntraResolver via PrincipalResolver; BearerTokenMiddleware; admin-key recognition)
+├── authz.py                      # Per-route capability gates (require_read / require_write)
 ├── identity_store.py             # Durable JSON identity map (write-file-then-swap, fail-closed load, live flat_dict)
-├── status.py                     # Status/version plumbing (EventRingBuffer, build_status_response, SERVER_VERSION, ring_buffer)
+├── status.py                     # Status/version plumbing (EventRingBuffer, build_status_response, SERVER_VERSION, ring_buffer, BootState/boot_state — the boot-phase singleton behind /status.boot)
 └── models.py                     # Pydantic request/response models
 
 docs/                             # ⚠️ PRODUCT DOCUMENTATION ONLY
@@ -432,6 +435,11 @@ with admin"). Runtime runbook: `docs/identity-management.md`.
 ## Key Concepts
 
 - **Event pipeline** — `POST /events` persists the raw event to a durable per-session append-log (`queue_manager.py`) and returns `202` immediately (persist-then-202). A single drainer per session (`registry.py`) processes batches and flushes them to Neo4j under a global write semaphore, with transient/deadlock retry, dead-letter isolation of poison events, and crash recovery (replay + counter re-seed) on startup. Each handler invoked by the per-event dispatch spine is a Python class in `handlers/data_layer_*/`.
+- **Append framing** — appends to one worker key are serialized by a per-key file lock (`_KeyGuard.file_lock`, held by the writing *thread*, not the coroutine), so a record lands as one contiguous newline-terminated line or not at all; a partial write is discarded and the error raised. The idempotency key is stored **only after** the durable append succeeds (`idempotency.py`: `seen()` / `store()` are separate) so a failed write plus a client retry is honoured, not falsely refused.
+- **Drain supervision** — a drain task's done-callback (`registry.py`) is the single supervision point: a task that dies logs `drain_worker_died` at ERROR with the session id + traceback, deregisters the session, and closes its store. A poison line is dead-lettered and the offset advanced past it, so draining continues.
+- **Boot phases** — boot recovery runs as a supervised background task (`main._boot_reconcile`), so `/status` and `/version` answer from the first phase. `status.BootState` (module singleton `boot_state`) tracks `recovering → heal → reclaim → expire → reconcile → seed → topup → sweep → ready`, or terminates at `failed` — and `failed` keeps serving. The phase is surfaced additively on `/status.boot`; while booting, `/status` does zero disk reads and reports `metrics`/`spool` as `null` with `status_detail.reason == "booting"`.
+- **Self-shrinking queue storage** — a live session's committed prefix is reclaimed continuously (compaction); a fully-drained log is reclaimed automatically at boot regardless of `reclaim_enabled`; log-less dead-letters expire on an mtime window, opt-in (`dead_letter_expiry_enabled`, default off — a dead-letter may be the last surviving copy of an un-recovered event). There is no operator GC endpoint — reclamation is automatic. `reclaim_enabled` (default `false`) gates only the more aggressive boot reclaim of unresumable/reset-offset logs.
+- **Writer lease** — `writer_lease.py` is a single-writer lease guard on the queue directory: a backstop against an accidental second writer (not a rolling-deploy coordinator; deployment is single-replica). Default mode `enforce` refuses to boot against a *live* foreign lease, releases the lease on clean shutdown, and takes over a *stale* foreign lease automatically. `detect` only surfaces a conflict on `/status.writer_lease` within a heartbeat, never refusing to boot.
 - **Graph model** — session sub-labels: `RootSession`, `SubSession`, `ForkedSession`, `IncompleteSession` (health marker; not a terminal). Full schema with all node and edge types: see `docs/architecture/03-graph-model.dot` and `docs/architecture/README.md`.
 - **Blob storage** — Large event payloads are written to disk and referenced by URI to avoid graph bloat.
 - **Configuration** — Pydantic Settings reads from `server-config.yaml` first, then environment variables. See `config.py`.

README.md

@@ -18,6 +18,11 @@ Amplifier CLI sessions
 | - Retry + dead-letter + crash recovery   |        | 5 node / 8 edge types|
 | - Blob storage (large payloads to disk)  |        +----------------------+
 | - API docs (/docs) + Cypher proxy        |
+| - Atomic single-writer append (per-key   |
+|   file lock) + supervised drain workers  |
+| - Self-shrinking queue storage           |
+|   (continuous compaction + dead-letter   |
+|    retention)                            |
 +------------------------------------------+
 ```
 
@@ -27,6 +32,20 @@ under a global write semaphore, retrying transient/deadlock failures and dead-le
 events. See [docs/architecture/05-durable-ingest-queue.png](docs/architecture/05-durable-ingest-queue.png)
 for the full ingest/drain flow.
 
+Each append is serialised by a per-key file lock, so one worker key is written by exactly one
+writer at a time and a record lands as one contiguous, newline-terminated line or not at all —
+a write that fails part-way is discarded (and the failure surfaced to the caller) rather than
+left as a torn or merged line. Drain workers are supervised: a worker that dies is logged at
+`ERROR` with its session id and traceback, its session is deregistered and its store closed,
+and an unparseable/poison line is written to that key's dead-letter file so draining continues
+past it instead of halting.
+
+Queue storage shrinks itself. A live session's already-committed prefix is reclaimed
+continuously (compaction) rather than only when the session ends, so a `.log` tracks the
+undrained tail rather than the whole session history; dead-letter files that no longer have a
+`.log` beside them expire on an mtime-based retention window. Both run automatically as part of
+normal operation and need no operator action.
+
 ---
 
 ## Upgrading: Cold Start No Longer Auto-Migrates
@@ -311,7 +330,7 @@ Full runtime onboarding/offboarding runbook and the `/admin/*` API:
 | Method | Path | Description |
 |--------|------|-------------|
 | `POST` | `/events` | Ingest a session event (returns 202 immediately) |
-| `GET` | `/status` | Server health, active sessions, completed history, error counts, `neo4j_connected`, `neo4j_query_connected` (reflects the read/cypher_query driver's connection health), `neo4j_url`, `neo4j_browser_url` |
+| `GET` | `/status` | Server health, active sessions, completed history, error counts, `neo4j_connected`, `neo4j_query_connected` (reflects the read/cypher_query driver's connection health), `neo4j_url`, `neo4j_browser_url`, plus `boot` (boot-reconciliation phase + counters), `writer_lease` (queue-directory single-writer lease guard), `spool` (aggregate spool footprint) and `metrics` (aggregate conservation counters). `boot` and `writer_lease` are always present; **`metrics` and `spool` are `null` while the server is still booting**, and `status_detail` is then `{"reason": "booting"}`. HTTP `200` and `status: "ok"` at every boot phase — the boot phase is informational, never a liveness signal |
 | `GET` | `/version` | Server version (`{"version": "..."}`) — always unauthenticated |
 | `GET` | `/docs` | Swagger UI — always on (auth-exempt) |
 | `GET` | `/openapi.json` | OpenAPI spec — always on (auth-exempt) |
@@ -399,6 +418,20 @@ Values are resolved with this priority (highest first):
 | `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_QUEUES_PATH` | `queues_path` | `/data/queues` | Directory for the durable per-session append-logs (persist-then-202 ingest); mirrors `blob_path`. |
 | `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_WRITE_CONCURRENCY` | `write_concurrency` | `8` | Max concurrent Neo4j write flushes across all session drainers (starvation guard). |
 | `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_MAX_DELIVERY_ATTEMPTS` | `max_delivery_attempts` | `5` | Flush retries for one batch before its offending line is dead-lettered. |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_CRASH_RECOVERY_RESPAWN_LIMIT` | `crash_recovery_respawn_limit` | `8` | **Behaviour change for existing installs — this default was previously unbounded.** Ceiling on how many recovered sessions get a drainer respawned per boot/sweep pass. The remainder is *deferred*: left untouched on disk, still durable, and drained by a later sweep pass or the moment a new event for that session arrives. Deferral is never silent (a `WARNING` names the respawned/deferred counts). `null` restores unbounded; `0` disables automatic respawn at boot. Negative values are a hard startup error. |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_CRASH_RECOVERY_SWEEP_INTERVAL_SECONDS` | `crash_recovery_sweep_interval_seconds` | `60` | **Behaviour change for existing installs — this default was previously 300.** Interval at which the deferred backlog is re-scanned and the recovered-drainer pool topped back up to the ceiling. Only runs when `crash_recovery_respawn_limit` is finite — change the two together or neither. `0` disables the sweep (the deferred tail then drains only on restart or on a new event). |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_RECLAIM_ENABLED` | `reclaim_enabled` | `false` | Whether the boot-time reclaim pass may actually delete / reset anything. **Ships disabled**: boot still classifies every pre-existing queue key and logs the same audit line with `action=dry_run`, but nothing is unlinked. Boot once with this off, read the reclaim summary in the logs, reconcile it against expectations, then opt in. Surfaced on `/status` as `boot.reclaim_enabled`. |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_RECLAIM_REDRAIN_MAX_BYTES` | `reclaim_redrain_max_bytes` | `67108864` (64 MiB) | Size ceiling for a `.log` with a **negative or past-EOF** `.offset`: below the ceiling it is **re-drained from byte 0** instead of deleted — bounded, and idempotent by construction, so re-driving costs nothing. At or above the ceiling the `.log` is deleted instead. `0` means always delete. An **unparseable** `.offset` is a separate case and always re-drains from byte 0 at any size — never deleted. |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_QUEUE_COMPACT_ENABLED` | `queue_compact_enabled` | `true` | Continuously reclaim a live session's already-committed prefix, so a `.log` holds the undrained tail rather than the whole session history. Turning it off is a config change + restart (a kill switch, not a live toggle). |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_QUEUE_COMPACT_MIN_PREFIX_BYTES` | `queue_compact_min_prefix_bytes` | `8388608` (8 MiB) | Bounds compaction **frequency** on a continuously busy session: below this many committed bytes the rewrite is skipped. The idle path closes the remaining gap for free the moment the session goes quiet. `0` is a valid explicit opt-out; negative is a hard startup error. |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_DEAD_LETTER_EXPIRY_ENABLED` | `dead_letter_expiry_enabled` | `false` | Expire dead-letter files that have **no `.log` beside them** once they age past the retention window below. **Opt-in (default off)**: a dead-letter may be the only surviving copy of an un-recovered event, so it is never auto-deleted out of the box. Deliberately independent of `reclaim_enabled`: the predicate here is two plain filesystem facts, not a heuristic classification. |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_DEAD_LETTER_RETENTION_SECONDS` | `dead_letter_retention_seconds` | `2592000.0` (30 days) | How long a log-less `.dead.jsonl` survives (by mtime) before expiry — long enough to notice it via `GET /queues/dead-letter` and purge or replay it, short enough that it cannot accumulate forever. `<= 0` disables expiry outright; negative is a hard startup error. |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_WRITER_LEASE_MODE` | `writer_lease_mode` | `enforce` | Single-writer lease guard for the queue directory — a backstop against an accidental second writer, not a rolling-deploy coordinator (deployment is single-replica). `enforce` (default) **refuses to boot** against a *live* foreign lease, releases the lease on clean shutdown so a restart reacquires immediately, and takes over a *stale* foreign lease automatically once it ages past the staleness window. `detect` acquires the lease best-effort, heartbeats it, and only surfaces a conflict on `/status.writer_lease` — it never refuses to boot. `off` disables the guard. See [docs/operational-hardening.md](docs/operational-hardening.md). |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_WRITER_LEASE_HEARTBEAT_SECONDS` | `writer_lease_heartbeat_seconds` | `5.0` | Lease renew + re-read interval. Under `detect`, sets how fast a conflict becomes visible on `/status` (within one heartbeat); under `enforce`, sets how fast a foreign takeover is noticed. Must be `> 0`. |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_WRITER_LEASE_STALENESS_MULTIPLIER` | `writer_lease_staleness_multiplier` | `3.0` | Staleness window = heartbeat x this (15.0s at defaults). Must survive two consecutive missed ticks without a false "stale" verdict, so values **below `2.0` are a hard startup error**. |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_WRITER_LEASE_CONFIRM_DELAY_SECONDS` | `writer_lease_confirm_delay_seconds` | `1.0` | Settle delay between writing the lease and the confirming re-read, to exceed write→other-reader visibility latency on a shared SMB mount. Must be `>= 0`. |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_WRITER_LEASE_ACQUIRE_TIMEOUT_SECONDS` | `writer_lease_acquire_timeout_seconds` | `5.0` | Hard bound on an entire acquire (and, per tick, an entire renew), so a hung mount can never block startup or wedge the heartbeat. **Must exceed `writer_lease_confirm_delay_seconds`** — otherwise every acquire would time out and silently disarm the guard, so that combination is a hard startup error. |
+| `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_WRITER_LEASE_FORCE_ACQUIRE` | `writer_lease_force_acquire` | `false` | One-boot operator escape hatch: take over a **fresh** (non-stale) foreign lease instead of refusing (`enforce`) or merely latching (`detect`). Logs a `WARNING` on every boot while set and is surfaced on `/status.writer_lease`, so leaving it on by accident is never silent. |
 | `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_LOG_PATH` | `log_path` | `/data/logs/server.jsonl` | Structured log file path |
 | `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_LOG_LEVEL` | `log_level` | `INFO` | Log level (`DEBUG`/`INFO`/`WARNING`/`ERROR`) |
 | `AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_SERVER_HOST` | `server_host` | `0.0.0.0` | Bind host |
@@ -433,6 +466,17 @@ recovered rather than lost across a restart. The durable per-session logs — no
 just Neo4j — are the record for events that have been accepted but not yet written
 to the graph.
 
+**A `.log` is a transient buffer, not a session archive.** Once a batch is
+committed, its bytes have reached the graph and the queue no longer needs them: the
+already-committed prefix of a *live, still-open* session's `.log` is reclaimed
+continuously (compaction), so the file tracks the undrained tail rather than the
+whole session history. A fully-drained log is removed at session finalize.
+Dead-letter files are likewise bounded: a
+`.dead.jsonl` with no `.log` beside it expires once it ages past
+`dead_letter_retention_seconds`. Read the queue directory as "what has not reached
+the graph yet", never as "everything this server ever ingested" — the graph and the
+blob store are the durable record.
+
 For persistence on Azure Container Apps (persistent storage + Neo4j on AuraDB),
 see [docs/azure-deployment.md](docs/azure-deployment.md).
 

amplifier-online.yaml

@@ -47,7 +47,10 @@ services:
     # blobs, and logs. Survives restarts/revision swaps. Single-instance enforced.
     volume:
       mount_path: /data
-      size_gib: 100      # Premium minimum is 100 GiB (billed on provisioned size)
+      size_gib: 1024     # Standing size 1 TiB — data grows; live quota was raised to
+                         # 1 TiB out-of-band, so declare it here to keep manifest == actual
+                         # (a re-apply then grows/holds, never shrinks below live usage).
+                         # (Premium minimum is 100 GiB; billed on provisioned size.)
       tier: premium      # Premium_LRS SSD share. Standard Azure Files (SMB) per-op
                          # latency caps the write-heavy queue/blob throughput and
                          # builds a standing write backlog under pressure.