Context Intelligence data-quality & operability (Phase 2): trustworthy metrics, session attribution, durable fix, incomplete-label heal, blob reclaim

[Diego Colombo (colombod)]

Why this PR

Context Intelligence is meant to be the trustworthy record of what AI sessions did. This PR fixes a set of defects where that record was silently wrong — and got worse the more a session was used (more runs, more forks, more restarts) — plus gives operators a safe way to reclaim disk. Server-side only; the companion bundle work merged separately (#414, closed). Builds on #413 (closed — headless server, PR #67).

This PR now also incorporates the fixes for the code-review feedback (see the next section). The original Phase-2 sections follow it unchanged.

---

Review remediation — response to code review

The review found that two things previously reported as fixed did not actually hold, plus a cap-inversion and test gaps. Every finding was independently re-verified against the code, implemented, and validated against a real server + real Neo4j (real restart). All blocking and all worth-fixing items are now addressed in code — only the full schema-version enforcement/migration design remains out of scope (see W-4).

Blocking

• B-1 — the server was survivable but not safe on degraded/un-migrated graph state. The deploy-safe-boot change stopped the crash-loop but installed no control at the point where corruption happens: drain workers wrote to Neo4j regardless of schema health, and the health signal was computed once at boot and never re-probed. Fix (maintenance mode): a first-class maintenance mode that gates both ingest and query with a structured 503 + Retry-After while the :Node uniqueness constraint is absent. The gate sits in the drain loop (never latches; a refused write never advances its queue offset → exactly-once replay). Health is a live, per-request TTL probe that self-clears without a restart once the graph is rectified. /status advertises mode / maintenance_started_at / maintenance_elapsed_seconds; enter/complete are logged. /status, /version, and /admin/maintenance stay reachable through the gate.
• Rectification is explicitly triggered, never at startup. To resolve the cloud deadlock (private Neo4j unreachable, no operator shell), a network-reachable POST /admin/maintenance endpoint runs the rectification on demand (atomic CAS single-flight, returns promptly with a run_id; GET reports progress) via a private admin-driver path. It shares one mechanism (neo4j_store.run_repair) with the standalone migrations/run.py --apply — one repair, two transports.
• B-2 — the flush retry loop reproduced the duplicate-Iteration bug. On a transient flush failure within the retry budget, the batch was replayed without restoring cursor state, so IterationHandler's mutable counter advanced each attempt (::iteration::1, ::iteration::2, …). The phantom-cursor guard existed but was wired only into the post-budget path. Fix: the snapshot/restore guard now wraps the common retry branch too; a transient-then-succeeding flush produces exactly one Iteration node (test proven non-vacuous by revert-check).
• B-3 — max_delete was not a cap. Unvalidated, a negative value sliced candidates[:-1] = delete-all-but-N. Fix: Field(ge=1); <1 rejected with 422 at the schema boundary (verified live: -1/0 → 422, 1 → 200).
• B-4 — tests certified liveness, not integrity. Added integrity tests: writes refused while degraded, the retry-loop no-duplicate case (through the main loop), max_delete<=0, the docs-entrypoint auth tripwire, and the maintenance state machine. tag_legacy_pooled_iterations.py (374-line live-graph mutation, previously zero coverage) now has tests/neo4j/test_tag_legacy_pooled_iterations.py proving against real Neo4j that its >=2 distinct parents selector tags a pooled Iteration but never a clean single-parent one, that --apply gates writes (a dry run mutates nothing), and idempotence.

Worth fixing — all addressed

• W-1 — undo log written after the mutation. Fix: relabel_incomplete_sessions.py --apply now reads the full candidate node_id set (read-only, same selector), writes the undo log, then mutates. A mid-run crash therefore leaves a complete undo record; the pre-mutation set is a superset and restore_ids is idempotent over it (clean run: candidate == touched, count unchanged). Real-Neo4j tests prove the log is sourced from the pre-mutation read, not apply_relabel's return value.
• W-2 — conflated failure signal. Fix: queue_health is now a separate app.state signal; a recovery_reconcile_dead/recover() failure sets queue_health="degraded" instead of masquerading as a schema problem.
• W-3 — working_dir "never overwritten" was Python-layer only. Fix: a blank/whitespace validator on the model, and a DB-level coalesce(n.working_dir, row.working_dir) on the Session MERGE so an existing value is never clobbered cross-writer. The generic hot-path node MERGE is untouched (working_dir only flows through the Session MERGE), preserving the index seek — proven with real-Neo4j tests.
• W-4 — SCHEMA_VERSION was decorative; a mismatch was undetectable. Fix: a read-only neo4j_store.read_graph_schema_version() (separate from the write path — ensure_schema_version_baseline untouched) now surfaces the stored :SchemaMeta.schema_version on GET /status as graph_schema_version plus an advisory schema_version_current, so automation can detect server/graph drift. This is advisory telemetry, not a guard — it does not gate, refuse, or migrate; full mismatch enforcement/migration comparison remains out of scope. /version is unchanged (still the cheap compiled-in constant).
• W-5 — blob-carrier allowlist had no runtime tripwire. Fix: a fail-closed tripwire at the blob-ref mint site (an unregistered carrier property raises before any write), and the allowlist is now the single source of truth shared by the mint path and the reclaim scan (removing a second hand-duplicated copy in the scan Cypher).
• W-6 — no test that idx_node_universal is dropped after a successful run_repair. Added: a non-vacuous real-Neo4j test (index present before → absent after the constraint is established).
• /status.degraded_reason self-consistency (found during remediation testing). Fix: degraded_reason now derives from the same live probe as mode/schema_health (instead of a boot snapshot), so it clears after an in-server repair instead of asserting a stale "constraint absent."

Remediation validation (real server + real Neo4j)

• Headline proven end-to-end: a degraded graph (constraint dropped + duplicates seeded) → server boots into maintenance (no crash-loop) → POST /events and POST /cypher → 503 + Retry-After + structured body; /admin/maintenance reachable through the gate → POST /admin/maintenance rectifies (records_affected reported) → same process, no restart, ingest resumes 202 and query 200.
• Offset exactly-once across kill -9 during maintenance (no loss, no dup, offset lands at EOF only after repair); healthy graph = zero behavior change; /version reports 6.8.0; migrations/run.py --status/--apply idempotent.
• 1946 non-Neo4j tests pass; the full tests/neo4j/ suite passes against real Neo4j (incl. the new tag-script, idx_node_universal-drop, and undo-log tests).

---

Issues addressed — problem for the user, the fix, and the evidence

Iteration metrics corruption (I5/I3 + I5b)

• User problem: per-run/per-session metrics — tokens, cost, cache, message counts — were wrong for any multi-run session. Iterations from different runs collapsed onto one graph node (up to 15 runs on one node), last-write-wins clobbering the numbers; and the first fix silently regressed to the broken shape on any server restart or 5-day stale-session reap.
• Fix: run-scoped Iteration.node_id so runs never collide (+ additive iteration_scope); a durable handler cursor persisted atomically with the queue offset and restored on every worker rebuild (crash-restart AND stale-reap); a phantom-cursor guard for dead-lettered events. (Review remediation additionally extends that guard to the common retry-then-succeed path.)
• Evidence: end-to-end test against real Neo4j — a mid-session server restart and a stale-worker reap each produce zero duplicate Iteration nodes, continuous run-scoped numbering, correct run edges, iteration_scope=run. Full suite green.

working_dir not queryable (I1)

• User problem: you could not reliably query sessions by project — working_dir was ~0% queryable in-graph.
• Fix: working_dir lifted onto the root Session node, populate-if-missing, idempotent, forward-only. (Review remediation adds a blank validator + a DB-level non-overwrite guarantee.)
• Evidence: covered by the server test suite (non-Neo4j + Neo4j).

IncompleteSession mislabeling

• User problem: ~52.8% of sessions were flagged IncompleteSession, but ~99% were false positives (a forked-sub-session ordering race stamped the label and never cleared it) — poisoning session-health analytics and giving a false "we are losing events" signal.
• Fix: heal-forward — classify() strips the stale marker on every start/fork (single normalizer, so a future handler reorder cannot skip it); the end branch is unchanged (the genuine ~0.5% signal). Plus a one-off, out-of-band, idempotent backfill (scripts/relabel_incomplete_sessions.py) that clears historical false positives, gated behind a read-only reconciliation diagnostic. Downstream graph-query skill guidance corrected (bundle). (Review remediation: undo log now written before the mutation; tag-script coverage added.)
• Evidence: real-Neo4j test drives an out-of-order end→fork/start through the real handler and asserts the label is physically removed; backfill tests cover preview/apply parity, idempotence, the diagnostic gate, restore, and the undo-log-before-mutation ordering.

blob store unbounded growth

• User problem: the blob store had no delete or GC — disk under blob_path grew forever with no safe reclaim, heading toward out-of-disk failures on real deployments.
• Fix: new admin endpoint POST /admin/blobs/reclaim — preview-first (dry_run default true), deletes only blobs unreferenced by any node in a global all-workspace scan; durable undrained-queue in-flight gate (survives kill -9) + live-worker check + mtime floor; structural $blob_ref extraction (not a regex that would silently truncate and delete referenced blobs); explicit max_delete cap and per-delete audit; idempotent. (Review remediation: max_delete is now a real cap via Field(ge=1); a fail-closed carrier-allowlist tripwire added.)
• Evidence: tests/neo4j/test_blob_reclaim.py — 12 tests green against real Neo4j, incl. cross-workspace safety, special-character extraction, crash/undrained-queue and live-worker protection, dry-run/apply parity, idempotence, max-delete-required.

Maintenance mode + out-of-band upgrade path

• The passive SCHEMA_VERSION (on GET /version) + create-if-absent :SchemaMeta singleton remain. This PR lands the maintenance-mode + explicitly-triggered in-place-fix mechanism (POST /admin/maintenance and the degraded-graph gate), the out-of-band upgrade runner + manifest (migrations/run.py + migrations/manifest.yaml), and a detectable schema-version drift signal on /status (advisory). Full drift-detection/comparison + enforcement logic remains out of scope.

Deliberately NOT in this PR (tracked separately)

• Storage/leak follow-ups — sentinel _no_session__* logs, stale-reap .log/.offset leak, dead-letter auto-expiry, Neo4j retention — remain for later work; this PR resolves only the blob-store reclaim (Finding chore(deps): bump pygments from 2.19.2 to 2.20.0 #1).
• cost_usd — handled on the read side by the merged bundle graph-query skill (parse + toFloat + pin to :LlmResponseEvent); no server change or data transformation needed.
• pending_tool_block_ids cap — measure-first; commit-time size instrumentation is in place, cap deferred until real data warrants it.

Operational — ACTION REQUIRED on upgrade if legacy data is present

Migrations never run at server startup. Out-of-band, preview-first steps after deploy (scripts ship in the image):

• Degraded schema (missing :Node constraint / duplicates): the server boots into maintenance mode and gates ingest+query. Rectify with python migrations/run.py --status then --apply (local/VM/direct-Neo4j) or POST /admin/maintenance (network-reachable) — the server self-clears with no restart.
• IncompleteSession reconciliation: docker exec <container> python3 scripts/relabel_incomplete_sessions.py --dry-run then --apply.
• Blob reclaim is on-demand via POST /admin/blobs/reclaim. See CHANGELOG.md, migrations/manifest.yaml, and docs/maintenance-mode.md.

Validation summary

• 1946 non-Neo4j tests pass; full tests/neo4j/ suite passes against real Neo4j, incl. the end-to-end proofs above.

[Diego Colombo (colombod)]

Closing in favor of the stacked PR #79.

PR #78 (harden-durable-ingestion) landed a deep fix for issue #473 (silent event loss on non-atomic/cloud-FS append) and reworked the queue/cursor storage this PR was originally built on. Rather than rebase #70 across that overlap, its improvements were transplanted onto #78 as a stacked PR: #79.

Everything of value here is carried forward in #79 (durable cursor + retry-dedup + run-id tiebreaker, IncompleteSession heal-forward, schema-version subsystem, maintenance mode + lease-armed auto-repair on boot, out-of-band migrations CLI, working_dir non-overwrite lift), re-homed onto #78's storage-correct primitives (fail_on_data_conflict=True and the schema_ready gate preserved; the deploy-safe-boot behavior deliberately not brought), at server version 6.7.3.

Review continues on #79, which is stacked on #78 — merge #78 first, then #79.