diff --git a/AGENTS.md b/AGENTS.md index a6edbc3c..d69690c9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,13 +13,37 @@ See [README.md](README.md) for full setup instructions. --- -## Current Work: Documentation & Setup Cleanup - -We are cleaning up this repo's **documentation and setup instructions** so the -server + Neo4j run locally **without Docker Compose**, with API keys primed for -local runs. Scope is intentionally narrow — do this and nothing else. - -### KEEP (do NOT remove) +## Current Work: Phase-2 review remediation (server data-quality + deploy safety) + +**Active engagement.** We are addressing the issues from the Phase-2 PR review: +a blob-reclaim cap-inversion, duplicate `Iteration` nodes on the retry path, and — +the big one — making the server **safe (not merely survivable) on un-migrated / +degraded graph state**, via a first-class **maintenance mode** (gate ingest + query +when the `:Node` uniqueness constraint is absent, structured `503` + `Retry-After`, +`/status` advertising, live re-probe that self-clears without restart) plus an +explicitly-triggered **`/admin/maintenance`** execution channel and an +out-of-band upgrade path. Full plan + council verdicts live in the **workspace-root +`docs/`** (one level up): `docs/plans/2026-08-13-review-remediation-plan.md`, +`docs/plans/2026-08-13-ws3-implementation-spec.md`, `docs/council/2026-08-13-*`. + +**Engagement guardrails** (see the workspace-root `AGENTS.md` for the authoritative +version): issue-driven only; **minimal, surgical, no-regression** diffs; every fix +**evidence-backed** and **DTU-validated** (real Neo4j, real restart) before "done"; +migrations/rectification run **OUT-OF-BAND**, never in the server startup/critical +path. "Out-of-band" now has **two sanctioned execution channels**, both explicitly +triggered (never at startup) and sharing one mechanism (`neo4j_store.run_repair`): +(1) the standalone `migrations/run.py --apply` (local/VM/direct-Neo4j), and (2) the +network-reachable, admin-authenticated `POST /admin/maintenance` (the cloud channel). +The server itself performs zero migration work automatically — it only assesses, +advertises (`/status`, `/version`), and gates while degraded. + +> **Superseded:** the earlier "Documentation & Setup Cleanup" engagement (remove +> Docker Compose / `start.sh`, add a local Neo4j script) is **complete/historical**. +> Its `docs/`-boundary rule and the container base-image policy below are **standing +> constraints** and still apply. The old "do not refactor server code" boundary does +> **not** apply to this engagement — server code changes are the point of it. + +### Container base image policy (S360 / SCA) — STANDING CONSTRAINT (do NOT remove) - **`Dockerfile` (the server image).** This is the shipping-product container and it is **S360-compliant** via PR #50 (`payneio` — "adopt Azure Linux base + @@ -141,8 +165,9 @@ docker run -d --name neo4j-ci \ cp server-config.example.yaml server-config.yaml # Edit server-config.yaml with your Neo4j connection details -# 3. Start -uvicorn context_intelligence_server.main:app --reload +# 3. Start (use main:asgi_app — the middleware-wrapped app that enforces bearer auth; +# main:app is the bare app with NO auth middleware — dev/testing only) +uvicorn context_intelligence_server.main:asgi_app --reload ``` Or use Docker Compose to run everything together: diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..ecd9751f --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,282 @@ +# Changelog + +All notable changes to the Context Intelligence Server are recorded here, +newest first. This is a human-readable log. A `SCHEMA_VERSION` baseline data +point now exists (see below), but the *handling* machinery that would consume +it -- a versioned migration manifest and an automated upgrade/rollback runner +-- is deliberately deferred to a separate design. + +## 6.8.0 -- Phase-2 review remediation: maintenance mode, retry-loop dedup fix, blob-reclaim/working_dir integrity + +**Not schema-affecting.** No stored node/edge shape changes; `SCHEMA_VERSION` +(`status.py`) stays `1`. See `migrations/manifest.yaml` for the one +structural-rectification entry this release adds, and the README +["Upgrading"](README.md#upgrading) section for the operator recovery path. + +> ⚠️ **A deployment with pre-existing un-migrated/duplicate `:Node` data will +> now boot into MAINTENANCE MODE** (ingest + query return `503`) instead of +> the "degraded but still writing" state 6.7.1 shipped. This is the fix, not +> a regression -- 6.7.1's degraded mode let concurrent writes manufacture +> *new* duplicates while un-migrated. If `GET /status` reports +> `mode=maintenance` or `mode=degraded` after upgrading, rectify **once**, +> out-of-band, via either: +> - `python migrations/run.py --apply` (local / VM / direct Neo4j access), or +> - `POST /admin/maintenance` (cloud / ACA, where the private Neo4j is not +> directly reachable) -- poll `GET /admin/maintenance` to completion. +> +> The server then self-clears to healthy with **no restart**. A +> healthy/already-migrated graph boots under 6.8.0 with zero behavior +> change. Use `context-intelligence-upload` to backfill any events that +> could not be ingested during a maintenance window. + +- **Maintenance mode -- gate ingest + query on degraded schema, never + latched (B-1).** The `6.7.1` deploy-safe-boot signal computed + `schema_health` once at boot and gated nothing; all write paths (boot + recovery, `POST /events`, dead-letter replay) wrote unconditionally even + while the `:Node` uniqueness constraint was absent, silently manufacturing + *new* duplicates. The gate now lives at the single drain-loop chokepoint + (before every batch's `read_batch`/commit), reads a **live, TTL-cached + re-probe** of constraint presence (never latched -- a graph repaired + out-of-band self-clears without a restart), and returns a structured + `503` (`Retry-After` + JSON reason) for both ingest and the query/`cypher` + surface. `/status` and `/version` stay up throughout and advertise the + mode (`healthy` / `maintenance` / `degraded` / `unknown`, + `maintenance_started_at`, `maintenance_elapsed_seconds`). Because the gate + sits upstream of the queue-offset commit, a refused write never advances + its offset -- kill-9 mid-refuse -> restart -> repair -> exactly-once + redelivery, by construction. Entry/exit is logged + (`maintenance_entered` / `maintenance_completed`). + (`context_intelligence_server/maintenance.py`, `registry.py`, + `routers/admin.py`, `main.py`) +- **`POST` / `GET /admin/maintenance` -- the cloud unblocker.** Assess + + advertise + gate alone deadlocks on ACA: an un-migrated graph behind a + private Neo4j has no operator who can run a script. `POST + /admin/maintenance` triggers the rectification (same `run_repair` the + standalone script calls -- one shared logic home, `maintenance_ops.py`) + over the network, with atomic (CAS) single-flight, a promptly-returning + `202` (never blocks for the op's duration), and no write-bypass (the op + never consults the gate -- it just isn't gated in the first place, using + the admin driver directly). `GET /admin/maintenance` reports + `state`/`run_id`/`started_at`/`records_affected`/error for polling to + completion. Both routes are on the maintenance-gate allow-list (with + `/status`/`/version`) so they cannot 503 themselves out of existence. +- **`max_delete` cap-inversion fixed (B-3).** `POST + /admin/blob-reclaim`'s `max_delete` had no floor validator; a + fat-fingered `0` or negative value inverted the cap into an + effectively-unbounded delete. Now `Field(ge=1)` rejects `<1` at the + schema boundary with a `422`; the existing apply-mode `422` on `None` is + unchanged. (`routers/admin.py`) +- **Retry-loop duplicate `Iteration` nodes fixed on the common path + (B-2).** The `I5` run-scoped `Iteration.node_id` fix stopped duplicates + on the normal path, but the flush **retry** branch mutated + `iteration_count` before the write and never restored it between + attempts -- a transient-fail-then-succeed retry could commit under two + different counter values (`::iteration::1` and `::iteration::2` both + landing). The existing `snapshot_cursor`/`restore_cursor` guard (already + used by the post-budget path) now also wraps the common retry branch: + snapshot once per batch, restore to pre-batch state before every replay + attempt. (`registry.py`) +- **`working_dir` integrity.** Blank/whitespace-only `working_dir` is now + rejected by the same validator style `workspace` already uses + (`models.py`). The "never overwritten" guarantee is now enforced at the + Cypher level (`ON CREATE SET` / `coalesce`), closing a cross-replica + same-session race that could previously last-write-wins clobber a good + value from Python-only discipline. (`neo4j_store.py`) +- **Blob-carrier allowlist tripwire (W-5).** The hardcoded + `_BLOB_REF_CARRIER_PROPERTIES` allowlist used by blob-reclaim's + reference scan is now a single source of truth shared with the mint + site (`blob_processor.py`), with a fail-closed runtime tripwire so a + future blob-ref-carrying property added elsewhere and forgotten in the + allowlist cannot let reclaim silently misclassify a live blob as an + orphan. +- **Docs auth-fold.** `README.md` / `docs/local-development.md` previously + documented running the server via `main:app` (the bare FastAPI app, no + bearer-auth middleware). Corrected to `main:asgi_app` (the + middleware-wrapped entrypoint that enforces auth on `/admin/*` and data + routes), with a tripwire test. +- Version bump: `6.7.1` -> `6.8.0` (minor -- new `/admin/maintenance` + endpoint and gate behavior change on degraded graphs; no schema change). + +See `docs/plans/2026-08-13-review-remediation-plan.md` and +`docs/plans/2026-08-13-ws3-implementation-spec.md` (workspace root, not +shipped) for the full remediation writeup and council verdicts. + +## 6.7.1 -- Deploy-safe boot: server never crash-loops on un-migrated/unreachable/degraded graph state + +**Incident:** deploying a restart crash-looped the server against a graph with +a single legacy node lacking the `:Node` label -- `RuntimeError: ... Cold +start refuses to boot ... Run: doctor --fix`, `systemd Restart=always` -> +hard outage, unrecoverable on Azure Container Apps (the private graph is +unreachable from outside and `doctor --fix` cannot be run to break the loop). + +- **Boot never raises on graph/data state (B1).** The entire lifespan + startup sequence (schema DDL, the untagged-node probe, the SchemaMeta + baseline, and queue crash-recovery/reconcile) is now wrapped in ONE + try/except boundary. Whichever of the ~11 startup raise sites fails + (unreachable Neo4j, a `TransientError` during the ACA cold-start race, + credential rotation, a corrupt per-session `.offset`/dead-letter, a + genuine `:Node` constraint data conflict, ...) is logged LOUDLY + (`startup_degraded`) and boot proceeds to serve requests. Cold start now + calls `ensure_neo4j_schema(..., fail_on_data_conflict=False)` -- the same + default the mid-flight flush path already used -- and the untagged-node + probe no longer raises on a positive count. Only `run_repair`/ + `doctor --fix` still fails closed on a genuine post-repair conflict. +- **Crash recovery is defensive per-session (B6).** A corrupt queue for one + session is quarantined (logged, skipped) instead of aborting the whole + respawn loop; the B1 boundary is the backstop for anything this doesn't + catch. +- **Degraded mode never loses the write-path index (B2).** The `:Node` + uniqueness constraint is now attempted BEFORE any `DROP INDEX + idx_node_universal` (previously unconditional and first, which could + leave the graph with no index at all if the constraint then failed on + duplicates -- regressing the 25-30s `AllNodesScan` stall PR #67 removed). + The standalone index is dropped ONLY after the constraint succeeds. If + the constraint cannot be created, a fallback `idx_node_universal` index + is (re-)created so writes keep a `NodeIndexSeek` -- degraded mode now + costs atomicity only, never the seek. A one-time drop-and-retry recovers + the constraint automatically once the underlying conflict is fixed (so a + prior degraded boot's own fallback index can never permanently lock the + graph out of the constraint). Healthy graphs (constraint already + present): zero behavior change. +- **Tri-state migration health on `GET /status` (B3/B7).** New fields + `schema_health` (`"healthy"` / `"degraded"` / `"unknown"`), + `untagged_nodes` (int|null), `schema_checked_at` (ISO-8601, computed once + at boot), and `degraded_reason` (string|null). A probe failure reports + `"unknown"`, never coerced to `"healthy"`. **This signal reflects + data-migration state, not process liveness, and MUST NOT be wired to a + Kubernetes/ACA liveness or readiness probe** -- doing so would recreate + the exact crash-loop this fix removes, one layer up. `GET /version` is + unchanged except for the version bump below. +- **Accepted tradeoff:** while degraded (constraint absent), concurrent + writes to the same `(node_id, workspace)` can create a NEW duplicate -- + this is a ratchet, not bounded, so degraded mode is loud (ERROR-logged) + rather than silent. Remediation is out-of-band `doctor --fix` (reachable + deployments) or the in-place-fix capability tracked separately for ACA. +- Version bump: `6.7.0` -> `6.7.1`. + +See `docs/plans/2026-08-12-deploy-safe-boot-spec.md` (workspace root, not +shipped) for the full incident writeup and council amendment. + +## Unreleased -- data-quality fixes (I5, I5b, I1, IncompleteSession), schema_version baseline + maintenance scripts + +> ⚠️ **ACTION REQUIRED ON UPGRADE IF LEGACY DATA IS PRESENT.** +> This release changes `IncompleteSession` labeling. New events self-heal, but +> **historical graphs carry ~52.8% stale `IncompleteSession` markers (~99% false +> positives) that are NOT corrected automatically.** A one-off, out-of-band +> reconciliation must be run **once** after this server is deployed and verified: +> +> 1. Deploy this release; confirm the server is up (Part 1 heal-forward is live). +> 2. Read-only check + preview (safe, writes nothing): +> `python3 scripts/relabel_incomplete_sessions.py --dry-run` +> 3. If the built-in reconciliation diagnostic is clean, apply once: +> `python3 scripts/relabel_incomplete_sessions.py --apply` +> (`--apply` self-refuses if the diagnostic finds unexpected data; idempotent — +> safe to re-run; writes an undo-log for `--restore`.) +> +> The maintenance scripts now ship **inside the Docker image** under `/app/scripts/`, +> so on a VM/ACI deployment run them via `docker exec python3 +> scripts/relabel_incomplete_sessions.py --dry-run`. Migrations are **never** run at +> server startup. Fresh/empty graphs need no action. + +- **IncompleteSession mislabeling -- heal-forward + one-off backfill.** + `IncompleteSession` was a Neo4j label written once at `session:end` when the + Session node had no type label yet, and never revised. Forked sub-sessions drain + in independent per-session queues with no cross-session ordering, so a child's + `session:end` is routinely processed before its `session:fork`/`session:start` -> + `classify()` saw no type -> stamped `IncompleteSession`; the later `fork`/`start` + added the real terminal but never cleared the stale marker (live bisect: ~52.8% + of sessions labeled, ~99.4% false positives -- the node carries its own linked + start/fork event; genuine loss ~0.5%). + - **Heal-forward (code):** `classify()` now strips `IncompleteSession` on every + `start`/`fork` transition via a single `_heal_forward()` normalizer; the `end` + branch is unchanged (still the real signal for the genuine ~0.5%). Reuses the + existing `set_labels` remove path -- no store/Cypher change. Order-independent. + - **Backfill (data rectification, run once):** `scripts/relabel_incomplete_sessions.py` + -- standalone, out-of-band, idempotent. Clears `IncompleteSession` only from + provably-false-positive nodes (real terminal type OR a linked + `SessionStartEvent`/`SessionForkEvent`, following the + `(Session)-[:SOURCED_FROM]->(Event)` direction), leaving the genuine ~0.5% + untouched. `--apply` is hard-gated behind a read-only reconciliation diagnostic + (refuses if any linked-but-untyped nodes exist), batched via + `CALL {} IN TRANSACTIONS`, with a touched-id undo-log + `--restore` and a + before/after population summary. **SCHEMA_VERSION unchanged** (no bump; handling + deferred). See the upgrade notice above. + +- **Maintenance/migration scripts now ship in the Docker image.** `scripts/` is + `COPY`'d into the runtime image (`/app/scripts/`) so out-of-band data-rectification + tools (`relabel_incomplete_sessions.py`, `tag_legacy_pooled_iterations.py`, + `repair_dual_labels.py`, ...) can be run against a live cloud deployment via + `docker exec ... python3 scripts/.py`. Previously these existed only in the + source tree and were unreachable from a running container. Invoke with `python3` + (the runtime image has no `python` alias). + +- **I5b -- durable handler cursor (fixes the I5 regression on worker rebuild).** + The run cursor I5's `Iteration.node_id` depends on (`execution_start_ts`, + `iteration_count`, and the rest of `DataLayer2State` / `DataLayer3State`) was + in-memory only, so any worker rebuild mid-session -- a process restart with an + undrained tail, or a stale-session reap -- reset it and regressed node_ids to + the pre-I5 bare shape, re-pooling Iteration nodes and dropping run edges. The + cursor is now persisted **atomically with the queue offset** (folded into the + `.offset` record as a single JSON object written via `os.replace`) and + restored on every worker (re)creation in `drain_worker`, covering both the + crash-recovery and stale-reap paths. Legacy bare-integer `.offset` files are + read transparently. A dead-lettered line's in-memory mutations are rolled back + so they never enter the committed cursor (no phantom cursor pointing at a + discarded node). Persisting the full DL2/DL3 cursor also preserves the E09 + (tool-call), E14/E15 (prompt-flow) and E10/E11 (recipe) edges across a rebuild. + (`queue_manager.py`, `services.py`, `registry.py`) + +- **`Iteration.iteration_scope` additive property (`"run" | "unscoped"`).** + Every Iteration node now carries `iteration_scope`, stamped at all three + upsert sites (`provider:request`, `llm:request`, `llm:response`), so a + run-scoped node and a legitimately unscoped one (e.g. a `loop-basic` session + with no `execution:start`) are distinguishable in-graph rather than only by + node_id shape. Additive and forward-only: historical nodes are `null`; the + node_id shape is unchanged. (`handlers/data_layer_2/iteration.py`) + +- **`SCHEMA_VERSION` baseline data points (integer `1`).** The server now + declares the graph schema version it expects (`SCHEMA_VERSION` alongside + `SERVER_VERSION`, surfaced on `/version`) and records the version the data + store was initialised at as a singleton `(:SchemaMeta {id:'singleton'})` node + written **create-if-absent only** (`ON CREATE SET schema_version, last_updated`; + never overwritten) inside `ensure_neo4j_schema` as an O(1) MERGE. This lands + only the *data points* for a future upgrade design -- there is deliberately + **no** comparison, migration, or reconciliation logic in this change. + (`status.py`, `routers/version.py`, `neo4j_store.py`) + +- **I5 -- run-scoped `Iteration.node_id` (also fixes I3).** + `Iteration.node_id` is now composed with the active orchestrator-run + identifier: `{session_id}::orch_run::{ts}::iteration::{N}`, using the same + run disambiguator `OrchestratorRun` already tracks. Previously the id was + the bare `{session_id}::iteration::{N}`, with `N` a per-*session* (not + per-*run*) counter -- so a counter reset (e.g. drainer restart/replay) + could reproduce a prior run's `N` under a new run and MERGE two runs' + Iteration nodes onto the same node, clobbering `usage_input` / + `usage_output` / `usage_cache_write` / `message_count` (last-write-wins). + This also fixes I3 (`usage_cache_write` junk) as a direct consequence. + **Forward-only**: already-merged historical Iteration nodes are not + retroactively split by this fix; see the tagging script below for the + non-destructive way to mark the confirmed-corrupt subset. + (`context_intelligence_server/handlers/data_layer_2/iteration.py`) + +- **I1 -- `working_dir` lifted to a root `Session` property + (populate-if-missing).** `EventRequest` gains an optional top-level + `working_dir`; `post_events` lifts it into the event data so it rides the + existing pipeline into `ensure_session_node`, which sets + `Session.working_dir`. Fills the property from any event that carries a + non-empty `working_dir` -- including re-imports of pre-existing sessions -- + and never clobbers an already-set value (idempotent). **Forward-only**: + sessions with no `working_dir`-bearing event stay `null`. + (`context_intelligence_server/main.py`, `models.py`, `services.py`) + +- **`scripts/tag_legacy_pooled_iterations.py` (new maintenance script).** + Non-destructive tagging tool for historical (pre-I5) bare-id `Iteration` + nodes. A bare `node_id` alone does NOT mean a node is corrupt -- only + nodes MERGEd across **two or more** distinct `OrchestratorRun`s (via + `HAS_PART`) are confirmed corrupt; live-data verification found this is a + small minority (~4%) of bare-id nodes. The script tags only that + confirmed-corrupt subset with `data_quality = 'legacy_pooled_pre_fix'` + (idempotent, batched via `CALL { ... } IN TRANSACTIONS OF N ROWS`) and + explicitly leaves single-run and no-run-edge bare-id nodes untouched. The + destructive cleanup (splitting/deleting pooled nodes) is a **separate, + gated follow-up** -- not part of this change. diff --git a/Dockerfile b/Dockerfile index 03f9b2c9..8625e1da 100644 --- a/Dockerfile +++ b/Dockerfile @@ -23,6 +23,11 @@ COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/ COPY pyproject.toml . COPY context_intelligence_server/ context_intelligence_server/ +# Ship the standalone, out-of-band maintenance/migration scripts in the image so +# they can be run against a live deployment (e.g. `docker exec ... python +# scripts/.py`). These are data-rectification tools (never run at startup); +# see CHANGELOG.md. Required for cloud VMs/ACI where there is no repo checkout. +COPY scripts/ scripts/ COPY docker-entrypoint.sh . RUN chmod +x docker-entrypoint.sh diff --git a/README.md b/README.md index 01b95ffb..257a1a2a 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,49 @@ for the full ingest/drain flow. --- +## Upgrading + +**Healthy / already-migrated deployments upgrade to `6.8.0` with zero +action and zero behavior change.** `SCHEMA_VERSION` is unchanged (`1`) -- +this release adds no stored node/edge shape, only server behavior. See +[CHANGELOG.md](CHANGELOG.md) for the full `6.8.0` entry and +[migrations/manifest.yaml](migrations/manifest.yaml) for the +machine-readable upgrade-mechanism entry (`from -> to`, whether it's +schema-affecting, which script to run, and how to verify). + +> ⚠️ **A deployment carrying pre-existing un-migrated / duplicate `:Node` +> data now boots into MAINTENANCE MODE under `6.8.0`** -- `POST /events` +> and the query/`cypher` surface return a structured `503` (`Retry-After` + +> reason) instead of `6.7.1`'s "degraded but still writing" state, which +> could silently manufacture new duplicates while un-migrated. `GET +> /status` and `GET /version` stay up throughout and advertise the mode +> (`healthy` / `maintenance` / `degraded` / `unknown`, +> `maintenance_started_at`, `maintenance_elapsed_seconds`) -- see +> [docs/maintenance-mode.md](docs/maintenance-mode.md) for the full +> contract. +> +> **Rectify once, out-of-band -- pick whichever channel you can reach:** +> +> ```bash +> # Local / VM / direct Neo4j access: +> python migrations/run.py --status # read-only report (safe, writes nothing) +> python migrations/run.py --apply # rectify: dedup + :Node backfill + constraint create +> ``` +> +> ```bash +> # Cloud / ACA, where the private Neo4j is not directly reachable: +> curl -X POST -H "Authorization: Bearer $ADMIN_KEY" https:///admin/maintenance +> curl -H "Authorization: Bearer $ADMIN_KEY" https:///admin/maintenance # poll to completion +> ``` +> +> The server **self-clears to healthy with no restart** once rectified (the +> gate re-probes live; it never latches). Use +> `context-intelligence-upload` to backfill any events that could not be +> ingested during the maintenance window. A fresh/empty graph has nothing +> to rectify and boots normally. + +--- + ## Neo4j Plugins (APOC + GDS) The server needs Neo4j 5.x reachable over Bolt with the **APOC** procedures @@ -149,7 +192,7 @@ python scripts/prime-local-config.py --neo4j-password '' ```bash export AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_CONFIG_FILE="$(pwd)/server-config.yaml" uv sync -uv run uvicorn context_intelligence_server.main:app --host 127.0.0.1 --port 8000 +uv run uvicorn context_intelligence_server.main:asgi_app --host 127.0.0.1 --port 8000 ``` Open [http://localhost:8000](http://localhost:8000) to confirm the server is diff --git a/context_intelligence_server/blob_processor.py b/context_intelligence_server/blob_processor.py index bd7d8c25..85bc266e 100644 --- a/context_intelligence_server/blob_processor.py +++ b/context_intelligence_server/blob_processor.py @@ -10,6 +10,7 @@ from __future__ import annotations import logging +import re from typing import TYPE_CHECKING, Any if TYPE_CHECKING: @@ -27,6 +28,91 @@ ) +# --------------------------------------------------------------------------- +# Blob-ref carrier allowlist -- single source of truth (WS-5 runtime tripwire) +# --------------------------------------------------------------------------- +# +# Every ``ci-blob://`` URI minted below (``process_event_data``) is written +# into ``data``, which ``DefaultHandler`` always persists wholesale as the +# JSON-serialized ``data`` property on the Event node +# (handlers/data_layer_1/default.py). The blob-reclaim reference scan +# (``routers.admin._scan_referenced_uris``) enumerates every ``ci-blob://`` +# reference anywhere in the graph by walking a FIXED allowlist of node +# properties -- never an all-property/all-node scan (this codebase has scar +# tissue from a 1.3M-node AllNodesScan stall). A blob whose reference lives +# on a node property the scan doesn't know about is invisible to it and can +# be deleted as a false orphan. +# +# BLOB_REF_CARRIER_PROPERTIES is THE single source of truth for that +# allowlist, imported by ``routers.admin`` to build the scan's Cypher +# directly from this tuple (so the query text can never drift from it) and +# checked here, at the mint site, via :func:`assert_carrier_registered`. +# +# Adding a new carrier (a future field-lifter/enricher that promotes a +# blob-ref-shaped value onto a new node property) means adding its name +# here. Forgetting to is now a fail-closed error, not a silent GC hole. +BLOB_REF_CARRIER_PROPERTIES: tuple[str, ...] = ( + "data", + "tool_input", + "prompt", + "response", +) + +# Defensive validation, run once at import time: every carrier name must be +# a legal Cypher property identifier, because routers.admin interpolates +# these names directly into a Cypher query string. Guards against a future +# careless addition (e.g. containing a space or backtick) turning into a +# broken or injectable query rather than a loud, immediate import error. +_VALID_CARRIER_NAME_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") + + +def _validate_carrier_names(names: tuple[str, ...]) -> None: + for name in names: + if not _VALID_CARRIER_NAME_RE.match(name): + raise ValueError( + f"BLOB_REF_CARRIER_PROPERTIES entry {name!r} is not a valid " + "Cypher property identifier -- refusing to load (this tuple " + "is interpolated directly into a Cypher query by " + "routers.admin._scan_referenced_uris)" + ) + + +_validate_carrier_names(BLOB_REF_CARRIER_PROPERTIES) + + +class UnregisteredBlobCarrierError(RuntimeError): + """A ``ci-blob://`` reference is destined for a node property that is + not in :data:`BLOB_REF_CARRIER_PROPERTIES`. + + This is the WS-5 runtime tripwire: it converts a silent reclaim-GC hole + (a live blob deleted as an orphan because its carrier property was never + added to the allowlist) into a loud, immediate failure at the point the + omission is introduced -- not after a live blob is gone. + """ + + +def assert_carrier_registered(property_name: str) -> None: + """Fail loud if *property_name* is not a registered blob-ref carrier. + + Cheap (single tuple-membership check) and safe to call on every + ``process_event_data`` invocation. Raises + :class:`UnregisteredBlobCarrierError` -- deliberately NOT caught by the + per-field ``except Exception`` below, so it propagates out of + ``process_event_data``, through ``pipeline.process_event``'s outer + handler (which logs and re-raises), and the event is dead-lettered + instead of silently minting an unprotected blob reference. + """ + if property_name not in BLOB_REF_CARRIER_PROPERTIES: + raise UnregisteredBlobCarrierError( + f"ci-blob:// reference destined for node property {property_name!r} " + f"is not in BLOB_REF_CARRIER_PROPERTIES {BLOB_REF_CARRIER_PROPERTIES!r} " + "-- the blob-reclaim scan (context_intelligence_server.routers.admin) " + "will not see refs stored there and could delete this blob as an " + f"orphan. Add {property_name!r} to BLOB_REF_CARRIER_PROPERTIES " + "(context_intelligence_server/blob_processor.py) before shipping." + ) + + # --------------------------------------------------------------------------- # Internal helpers # --------------------------------------------------------------------------- @@ -88,6 +174,16 @@ async def process_event_data( """ _lift_raw_fields(data) + # WS-5 runtime tripwire: every ci-blob:// URI minted below lands in + # `data`, which DefaultHandler always persists wholesale onto the Event + # node's "data" property. Fail loud, BEFORE any blob is written, if that + # destination is ever missing from the allowlist the reclaim scan reads + # (see BLOB_REF_CARRIER_PROPERTIES above). Deliberately outside the + # per-field try/except below so it is never downgraded to a swallowed + # $blob_error -- it propagates out of process_event_data and dead-letters + # the event instead of silently minting an unprotected blob reference. + assert_carrier_registered("data") + for field_name in BLOB_FIELDS: value = data.get(field_name) if value is None: diff --git a/context_intelligence_server/config.py b/context_intelligence_server/config.py index 48b20e97..c95392b7 100644 --- a/context_intelligence_server/config.py +++ b/context_intelligence_server/config.py @@ -902,6 +902,17 @@ def _validate_crash_recovery_sweep_interval(cls, v: int) -> int: status_inactive_timeout: float = 1800.0 # 30 min — /status visibility stale_session_timeout: float = 432000.0 # 5 days — worker reap + # ------------------------------------------------------------------------- + # Maintenance mode (WS-3a: live gate + /status; WS-3c wires + # POST/GET /admin/maintenance on top of this same coordinator/seam) + # ------------------------------------------------------------------------- + maintenance_probe_ttl_seconds: float = 5.0 # :Node constraint probe cache TTL + maintenance_retry_after_seconds: int = 30 # Retry-After on the maintenance 503 + # WS-3c: bounded pre-op quiesce before run_repair (spec sec 5.3) -- covers + # ordinary in-flight flushes (_DRAIN_POLL_INTERVAL is 0.05s); a flush that + # outlives this is a residual, DETECTED risk (constraint create fails loud). + maintenance_quiesce_seconds: float = 2.0 + @classmethod def settings_customise_sources( cls, diff --git a/context_intelligence_server/handlers/data_layer_2/content_block.py b/context_intelligence_server/handlers/data_layer_2/content_block.py index c0238f79..2309725b 100644 --- a/context_intelligence_server/handlers/data_layer_2/content_block.py +++ b/context_intelligence_server/handlers/data_layer_2/content_block.py @@ -50,8 +50,11 @@ async def __call__(self, event: str, data: dict[str, Any]) -> HookResult: block_index = data.get("block_index") iteration_id = self.services.data_layer_2.active_iteration_id - # ID format is "{session_id}::iteration::{n}"; [-1] extracts the iteration number. - # If the cursor format ever changes, this extraction must be updated to match. + # active_iteration_id's trailing "::"-segment is always the plain iteration + # number, whether the cursor is the bare "{session_id}::iteration::{n}" shape + # or the run-scoped "{session_id}::orch_run::{ts}::iteration::{n}" shape + # (P2.1 fix, see IterationHandler) -- [-1] extracts it either way. If the + # cursor format ever changes this extraction must be updated to match. iteration_n = iteration_id.split("::")[-1] if iteration_id else "0" block_node_id = f"{session_id}::block::{iteration_n}::{block_index}" diff --git a/context_intelligence_server/handlers/data_layer_2/iteration.py b/context_intelligence_server/handlers/data_layer_2/iteration.py index 4c10d1e8..74aab632 100644 --- a/context_intelligence_server/handlers/data_layer_2/iteration.py +++ b/context_intelligence_server/handlers/data_layer_2/iteration.py @@ -36,6 +36,18 @@ class IterationHandler: def __init__(self, services: HookStateService) -> None: self.services = services + def _current_iteration_scope(self) -> str: + """The additive 'run' | 'unscoped' discriminator (D6), sourced from the + SAME cursor field (``execution_start_ts``) used to decide the node_id + shape. A single source of truth so all three upsert_node call sites + (provider:request, llm:request, llm:response) always agree. + """ + return ( + "run" + if self.services.data_layer_2.execution_start_ts is not None + else "unscoped" + ) + async def __call__(self, event: str, data: dict[str, Any]) -> HookResult: """Dispatch to the appropriate sub-handler. @@ -64,22 +76,56 @@ async def _handle_provider_request( ) -> None: """Create Iteration node and set active_iteration_id cursor. - - Computes iteration_id as '{session_id}::iteration::{iteration_number}' + - Computes iteration_id as run-scoped: + '{session_id}::orch_run::{execution_start_ts}::iteration::{iteration_number}' + when an orchestrator run is active (execution_start_ts cursor set), using the + SAME disambiguator OrchestratorRun uses for its own node_id (P2.1 / I5 fix). + Falls back to the bare '{session_id}::iteration::{iteration_number}' shape when + no run is active -- this bare shape is also what historical (pre-fix) Iteration + nodes look like, making them detectable as suspect. See CHANGELOG.md. - Sets active_iteration_id cursor on DataLayer2State - Creates Iteration:SST_EVENT node with session_id, iteration_number, started_at - Conditionally creates E06: OrchestratorRun -[:HAS_PART {sst_semantic: 'CONTAINS'}]-> Iteration when execution_start_ts cursor is set + + Without run-scoping, iteration_number alone (a counter scoped to the whole + session, not the run) can repeat across orchestrator runs -- e.g. after a + drainer restart/replay resets the in-memory counter -- causing distinct runs' + Iteration nodes to MERGE onto the same node_id (I5) and their usage figures to + clobber each other (I3, usage_cache_write in particular). """ # Increment counter to get the next iteration number self.services.data_layer_2.iteration_count += 1 iteration_number = self.services.data_layer_2.iteration_count timestamp: str = data.get("timestamp", "") - iteration_id = f"{session_id}::iteration::{iteration_number}" + + execution_start_ts = self.services.data_layer_2.execution_start_ts + orch_run_id: str | None = None + if execution_start_ts is not None: + orch_run_id = f"{session_id}::orch_run::{execution_start_ts}" + iteration_id = f"{orch_run_id}::iteration::{iteration_number}" + else: + iteration_id = f"{session_id}::iteration::{iteration_number}" # Set cursor so llm:request and llm:response can find this iteration self.services.data_layer_2.active_iteration_id = iteration_id + # D6: additive, queryable discriminator between a run-scoped iteration + # and a legitimate loop-basic session with no active orchestrator run. + # Bare shape != state-lost; see the module/method docstrings. + iteration_scope = self._current_iteration_scope() + if iteration_scope == "unscoped": + # INFO (not WARNING, spec §10.4): a loop-basic session with no + # execution:start is a normal case, not an alert-worthy anomaly. + logger.info( + "unscoped_iteration_emitted session=%s iteration_number=%d iteration_id=%s", + session_id, + iteration_number, + iteration_id, + extra={"session_id": session_id}, + ) + # Create the Iteration node await self.services.graph.upsert_node( iteration_id, @@ -88,13 +134,12 @@ async def _handle_provider_request( "session_id": session_id, "iteration_number": iteration_number, "started_at": timestamp, + "iteration_scope": iteration_scope, }, ) # E06 (conditional): OrchestratorRun -[:HAS_PART {sst_semantic: 'CONTAINS'}]-> Iteration - execution_start_ts = self.services.data_layer_2.execution_start_ts - if execution_start_ts is not None: - orch_run_id = f"{session_id}::orch_run::{execution_start_ts}" + if orch_run_id is not None: await self.services.graph.upsert_edge( orch_run_id, iteration_id, @@ -127,6 +172,12 @@ async def _handle_llm_request(self, data: dict[str, Any]) -> None: "model": data.get("model"), "message_count": data.get("message_count"), "has_system": data.get("has_system"), + # BLOCKER-2: stamp independently of provider:request's own + # write -- an Iteration node must never be created/updated + # without a scope value, even if this write is the first one + # to ever reach the node (e.g. a dead-lettered provider:request + # whose cursor mutation nonetheless survived). + "iteration_scope": self._current_iteration_scope(), }, ) @@ -157,6 +208,9 @@ async def _handle_llm_response(self, data: dict[str, Any]) -> None: "usage_input": usage.get("input_tokens"), "usage_output": usage.get("output_tokens"), "usage_cache_write": usage.get("cache_creation_input_tokens"), + # BLOCKER-2: see _handle_llm_request -- same completeness + # rationale applies to this, the third of the three sites. + "iteration_scope": self._current_iteration_scope(), }, ) diff --git a/context_intelligence_server/handlers/data_layer_2/session.py b/context_intelligence_server/handlers/data_layer_2/session.py index a39118f1..b4843980 100644 --- a/context_intelligence_server/handlers/data_layer_2/session.py +++ b/context_intelligence_server/handlers/data_layer_2/session.py @@ -61,6 +61,32 @@ class LabelTransition: remove: list[str] = field(default_factory=list) +def _heal_forward(transition: LabelTransition) -> LabelTransition: + """Strip a stale IncompleteSession marker on every start/fork transition. + + IncompleteSession is only ever a correct label at session:end, when + genuinely no type is known yet. It can never legitimately coexist with a + real terminal type (RootSession/SubSession/ForkedSession) or with a + start/fork event that reconfirms one — co-occurrence is always the + out-of-order / false-positive case: a forked sub-session's session:end + drained before its session:fork/session:start, stamping the marker before + the real terminal arrived (see docs/issues/incomplete-session-mislabeling.md). + + Applied once, uniformly, to the *result* of every start/fork branch — + including the no-op branches, where a node may already carry a stale + marker from an earlier out-of-order end — rather than hand-editing each + branch's `remove` list. This makes the invariant impossible to miss if a + branch is added later. REMOVE of an absent label is a documented no-op + (see GraphState.set_labels / Neo4jGraphStore.set_labels), so healing is + always safe even when no stale marker exists. + """ + if "IncompleteSession" in transition.remove: + return transition + return LabelTransition( + add=transition.add, remove=[*transition.remove, "IncompleteSession"] + ) + + class SessionLabelStateMachine: """State machine for session type label transitions. @@ -80,40 +106,17 @@ def classify( # clears StubSession, so genuine late enrichment removes the marker. Removing # a label that isn't present is a silent no-op (see GraphState.set_labels / # Neo4jGraphStore.set_labels), so it is always safe to include in `remove`. + # + # NOTE on "IncompleteSession" healing: every start/fork transition also + # strips IncompleteSession via _heal_forward() below — see that function's + # docstring for the invariant. It is applied to the branch's return value + # as a single normalization step, not hand-edited per branch, so it + # cannot be missed by a future branch. if event == "start": - if current_type in ("ForkedSession", "SubSession"): - return LabelTransition() - if current_type == "RootSession": - if has_parent: - return LabelTransition( - add=["SubSession", "SST_EVENT"], - remove=["RootSession", "StubSession"], - ) - return LabelTransition() - # bare session (current_type is None) - if has_parent: - return LabelTransition( - add=["Session", "SubSession", "SST_EVENT"], - remove=["StubSession"], - ) - return LabelTransition( - add=["RootSession", "Session", "SST_EVENT"], - remove=["StubSession"], - ) + return _heal_forward(self._classify_start(current_type, has_parent)) if event == "fork": - if current_type == "ForkedSession": - return LabelTransition() - if current_type in ("RootSession", "SubSession"): - return LabelTransition( - add=["ForkedSession", "SST_EVENT"], - remove=[current_type, "StubSession"], - ) - # bare session (current_type is None) - return LabelTransition( - add=["Session", "ForkedSession", "SST_EVENT"], - remove=["StubSession"], - ) + return _heal_forward(self._classify_fork(current_type, has_parent)) if event == "end": if current_type is not None: @@ -126,15 +129,57 @@ def classify( # so StubSession is cleared here too. # # NOTE: if a real start/fork ever arrives AFTER this end (out-of-order, - # vanishingly rare), _handle_start/_handle_fork will classify normally - # and add the real terminal. IncompleteSession may then coexist as an - # audit trail — that is acceptable; no special stripping is needed. + # which routinely happens for forked sub-sessions whose independent + # queue drains before the parent's), _handle_start/_handle_fork now heal + # forward — classify() strips IncompleteSession via _heal_forward() the + # moment the real start/fork is processed, so the stale marker no + # longer coexists with the real terminal. See + # docs/issues/incomplete-session-mislabeling.md. return LabelTransition( add=["IncompleteSession", "SST_EVENT"], remove=["StubSession"] ) raise ValueError(f"classify() received unknown event: {event!r}") + @staticmethod + def _classify_start(current_type: str | None, has_parent: bool) -> LabelTransition: + """Compute the start-event transition, before IncompleteSession healing.""" + if current_type in ("ForkedSession", "SubSession"): + return LabelTransition() + if current_type == "RootSession": + if has_parent: + return LabelTransition( + add=["SubSession", "SST_EVENT"], + remove=["RootSession", "StubSession"], + ) + return LabelTransition() + # bare session (current_type is None) + if has_parent: + return LabelTransition( + add=["Session", "SubSession", "SST_EVENT"], + remove=["StubSession"], + ) + return LabelTransition( + add=["RootSession", "Session", "SST_EVENT"], + remove=["StubSession"], + ) + + @staticmethod + def _classify_fork(current_type: str | None, has_parent: bool) -> LabelTransition: + """Compute the fork-event transition, before IncompleteSession healing.""" + if current_type == "ForkedSession": + return LabelTransition() + if current_type in ("RootSession", "SubSession"): + return LabelTransition( + add=["ForkedSession", "SST_EVENT"], + remove=[current_type, "StubSession"], + ) + # bare session (current_type is None) + return LabelTransition( + add=["Session", "ForkedSession", "SST_EVENT"], + remove=["StubSession"], + ) + class SessionHandler: """Handles session lifecycle events. diff --git a/context_intelligence_server/handlers/data_layer_2/state.py b/context_intelligence_server/handlers/data_layer_2/state.py index 47b363c2..dac5cc38 100644 --- a/context_intelligence_server/handlers/data_layer_2/state.py +++ b/context_intelligence_server/handlers/data_layer_2/state.py @@ -28,6 +28,12 @@ class DataLayer2State: # E15 OrchestratorRun→Prompt turn-flow cursor last_completed_orch_run_id: str | None = None - # Iteration counter — incremented on each provider:request; used to compute - # iteration_id as '{session_id}::iteration::{iteration_count}' + # Iteration counter — incremented on each provider:request; combined with + # execution_start_ts (via IterationHandler) to compute the run-scoped + # iteration_id '{session_id}::orch_run::{execution_start_ts}::iteration::{iteration_count}' + # (falls back to the bare '{session_id}::iteration::{iteration_count}' shape when no + # orchestrator run is active). Scoped per-session, not reset per run: uniqueness across + # runs comes from the orch_run_id prefix, not from this counter (P2.1 fix for I5/I3) — + # resetting it would collide with ContentBlockHandler's block_node_id derivation, which + # keys solely off this counter's value, not the run. iteration_count: int = 0 diff --git a/context_intelligence_server/handlers/data_layer_3/delegation.py b/context_intelligence_server/handlers/data_layer_3/delegation.py index efeda5c8..3358b1fb 100644 --- a/context_intelligence_server/handlers/data_layer_3/delegation.py +++ b/context_intelligence_server/handlers/data_layer_3/delegation.py @@ -52,12 +52,17 @@ def _discriminate_root_vs_unresolved(parent_session: dict[str, Any] | None) -> s -> "unresolved" (fails loud; monitored; never mistaken for a real answer) CRITICAL: branches on the TERMINAL label ONLY, never on - ``IncompleteSession``. Live graph data shows ``IncompleteSession`` - co-labels a terminal label ~41% of the time (a session can reach - session:end with session:start/fork permanently missed, out of order -- - see SessionHandler._handle_end). Treating ``IncompleteSession`` as a - discriminator would mis-flag hundreds of genuine root/forked sessions as - unresolved. + ``IncompleteSession``. Historically ``IncompleteSession`` co-labeled a + terminal label ~41% of the time (a session's session:end drained before + its session:start/fork, out of order -- see SessionHandler._handle_end). + As of the heal-forward fix, SessionLabelStateMachine.classify() now + strips ``IncompleteSession`` the moment a real start/fork is processed + (see docs/issues/incomplete-session-mislabeling.md), so co-occurrence is + no longer expected for newly-processed events -- only pending a one-off + backfill for historical nodes written before the fix. This function + still ignores ``IncompleteSession`` as a discriminator regardless: it was + never a reliable signal and using it would mis-flag genuine root/forked + sessions as unresolved. """ labels: list[str] = (parent_session or {}).get("labels", []) if "RootSession" in labels: diff --git a/context_intelligence_server/logging_config.py b/context_intelligence_server/logging_config.py index 588c93e6..b9633458 100644 --- a/context_intelligence_server/logging_config.py +++ b/context_intelligence_server/logging_config.py @@ -115,6 +115,18 @@ def setup_logging() -> None: Idempotent: if the root logger already has handlers attached (e.g. because the application lifespan is exercised multiple times in tests) this function returns immediately without adding duplicate handlers. + + Deploy-safe boot (council amendment, 2026-08-12): this function MUST NEVER + raise. The stdout/stderr StreamHandler is configured FIRST and + unconditionally, so console logging always exists to report a degraded + state. The RotatingFileHandler (and its parent-directory mkdir) is + best-effort and wrapped in its own try/except: on failure (e.g. the + `/data` volume not yet mounted, or a read-only path on Azure Container + Apps) it falls back to console-only and logs a loud WARNING, rather than + crash-looping the worker. This is the exact real-process failure a + live boot test caught: `setup_logging()` used to `mkdir('/data')` before + the lifespan try/except boundary and a `PermissionError` there sank the + whole worker with no Neo4j involvement at all. """ settings = get_settings() log_path = Path(settings.log_path) @@ -123,41 +135,52 @@ def setup_logging() -> None: log_path = log_path / "server.jsonl" log_level = settings.log_level - # Ensure parent directory exists - log_path.parent.mkdir(parents=True, exist_ok=True) - formatter = JsonFormatter() root_logger = logging.getLogger() root_logger.setLevel(log_level) - # Guard: skip handler registration if our RotatingFileHandler is already present. - # Checking for a RotatingFileHandler (rather than any handler) avoids false - # positives from pytest's log-capture handler which is always present during tests. - if any( - isinstance(h, logging.handlers.RotatingFileHandler) - for h in root_logger.handlers - ): + # Guard: skip handler registration if our StreamHandler is already present. + # Checking for our own console StreamHandler (rather than any handler) + # avoids false positives from pytest's log-capture handler which is always + # present during tests. We key off the console handler (not the file + # handler) because the file handler may legitimately be ABSENT in a + # degraded boot -- if we keyed off the file handler, a degraded first call + # would re-run on the next call and stack duplicate console handlers. + if any(getattr(h, "_ci_console_handler", False) for h in root_logger.handlers): return - # stdout stream handler + # stdout stream handler -- configured FIRST and unconditionally, so console + # logging ALWAYS exists (deploy-safe boot: we must be able to report a + # degraded state even when file logging is unavailable). stream_handler = logging.StreamHandler(sys.stdout) stream_handler.setFormatter(formatter) + stream_handler.setLevel(log_level) + # Tag so the idempotency guard above can recognise our own console handler. + stream_handler._ci_console_handler = True # type: ignore[attr-defined] root_logger.addHandler(stream_handler) - # rotating file handler - file_handler = logging.handlers.RotatingFileHandler( - filename=str(log_path), - maxBytes=_MAX_BYTES, - backupCount=_BACKUP_COUNT, - ) - file_handler.setFormatter(formatter) - root_logger.addHandler(file_handler) - - # Gate the handlers at the configured level so the DEBUG-demoted access logs - # (below) are hidden at INFO and surface only when the level is DEBUG. - stream_handler.setLevel(log_level) - file_handler.setLevel(log_level) + # rotating file handler -- best-effort. A failure here (unwritable /data, + # volume not yet mounted, wrong perms) MUST NOT crash boot; fall back to + # console-only. Both the parent-dir mkdir and the handler construction can + # raise OSError/PermissionError, so both live inside this guard. + try: + log_path.parent.mkdir(parents=True, exist_ok=True) + file_handler = logging.handlers.RotatingFileHandler( + filename=str(log_path), + maxBytes=_MAX_BYTES, + backupCount=_BACKUP_COUNT, + ) + file_handler.setFormatter(formatter) + file_handler.setLevel(log_level) + root_logger.addHandler(file_handler) + except OSError as exc: + # Console logging is already wired above, so this warning is delivered. + root_logger.warning( + "file logging unavailable at %s, using console only: %s", + log_path, + exc, + ) # Route uvicorn/gunicorn loggers up to the root JsonFormatter so every line # from those frameworks is one-line JSON (not plain text) for Azure Log diff --git a/context_intelligence_server/main.py b/context_intelligence_server/main.py index 0bf0709f..73473513 100644 --- a/context_intelligence_server/main.py +++ b/context_intelligence_server/main.py @@ -10,7 +10,7 @@ import time from collections.abc import AsyncGenerator from contextlib import asynccontextmanager, suppress -from datetime import datetime +from datetime import UTC, datetime from pathlib import Path from typing import Any @@ -35,6 +35,11 @@ from context_intelligence_server.idempotency import EventIdempotencyCache from context_intelligence_server.identity_store import IdentityStore from context_intelligence_server.logging_config import setup_logging +from context_intelligence_server.maintenance import ( + MAINTENANCE_ALLOW_LIST, + coordinator, + maintenance_gate_middleware, +) from context_intelligence_server.models import ( CypherRequest, EventRequest, @@ -43,12 +48,14 @@ from context_intelligence_server.neo4j_store import ( count_untagged_nodes, ensure_neo4j_schema, + ensure_schema_version_baseline, + read_graph_schema_version, ) from context_intelligence_server.registry import SessionRegistry from context_intelligence_server.routers.admin import router as admin_router from context_intelligence_server.routers.queues import router as queues_router from context_intelligence_server.routers.version import router as version_router -from context_intelligence_server.status import build_status_response +from context_intelligence_server.status import SCHEMA_VERSION, build_status_response _settings = get_settings() @@ -204,173 +211,368 @@ async def _crash_recovery_sweep_loop(interval: int, respawn_limit: int) -> None: logger.warning("crash_recovery_sweep: tick failed, will retry: %s", exc) +def _record_schema_health( + app: FastAPI, + constraint_established: bool, + untagged: int | None, +) -> None: + """Compute and stash the tri-state schema-health signal on app.state. + + Council amendment B3/B7 (deploy-safe boot, 2026-08-12): health is a + tri-state enum, never coerced to a false "healthy": + + - ``"healthy"`` -- the :Node uniqueness constraint is established AND + the untagged-node probe ran and found 0. + - ``"degraded"`` -- the constraint is absent (a data conflict, logged + loudly by ``ensure_neo4j_schema``/B4) OR the probe found untagged + nodes. Bounded/repairable, but never silent. + - ``"unknown"`` -- *untagged* is None, meaning the probe itself could + not run (Neo4j unreachable, credential rejection, etc). A probe that + cannot answer must say so, not report green (B3). + + Written prohibition (B7): this is a **data-migration** signal, computed + once at boot. It MUST NOT be wired to a Kubernetes/ACA liveness or + readiness probe -- doing so would recreate the exact crash-loop this fix + removes, one layer up. See docs/azure-deployment.md. + """ + app.state.schema_untagged_nodes = untagged + app.state.schema_checked_at = datetime.now(UTC).isoformat() + + if untagged is None: + app.state.schema_health = "unknown" + app.state.schema_degraded_reason = ( + "untagged-node probe failed -- graph may be unreachable or " + "credentials rejected; schema state could not be determined" + ) + return + + reasons: list[str] = [] + if not constraint_established: + reasons.append(":Node uniqueness constraint absent (data conflict)") + if untagged > 0: + reasons.append(f"{untagged} node(s) lacking the :Node label") + + if reasons: + app.state.schema_health = "degraded" + app.state.schema_degraded_reason = "; ".join(reasons) + logger.error("schema_degraded: %s", app.state.schema_degraded_reason) + else: + app.state.schema_health = "healthy" + app.state.schema_degraded_reason = None + + @asynccontextmanager async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: - """Manage application lifespan: configure logging and create shared Neo4j driver.""" - setup_logging() - _admin = _settings.resolve_neo4j_admin() - _query = _settings.resolve_neo4j_query() - logger.info( - "lifespan_startup: creating Neo4j drivers admin_url=%s query_url=%s query_access_mode=%s", - _admin.url, - _query.url, - _query.access_mode, - ) - # Admin (read/write): schema init + all mutation paths. Keep the existing - # app.state.neo4j_driver NAME so nothing that reads it silently breaks. - # build_neo4j_driver() is the SAME helper doctor.run_doctor() uses, so the - # server and the doctor CLI can never construct this connection differently. - app.state.neo4j_driver = build_neo4j_driver(_admin) - # Cypher-query (read-intent): /cypher + dashboard reads. - app.state.neo4j_query_driver = AsyncGraphDatabase.driver( - _query.url, auth=_query.auth - ) - # Stash the resolved query access_mode so /cypher opens READ sessions without - # re-resolving settings on every request. - app.state.neo4j_query_access_mode = _query.access_mode - # Initialize schema (indexes + uniqueness constraints) BEFORE the server starts - # accepting requests. This ensures the Session uniqueness constraint is active - # before any concurrent flush() transactions execute MERGE, which prevents the - # duplicate-Session-node race condition observed under concurrent upload load. - logger.info( - "lifespan_startup: initializing Neo4j schema (indexes + uniqueness constraints)" - ) - # Cold start FAILS LOUD on schema/data corruption that requires - # `doctor --fix` -- an un-migrated graph (duplicate legacy nodes OR - # nodes lacking the universal :Node label). Nothing has been written yet - # at cold start, so refusing to boot loses no data: this is the safest - # possible moment to surface an impossible state as an un-missable - # signal rather than a log line someone greps for later. Contrast with - # the flush path (Neo4jGraphStore._ensure_schema), which must keep - # self-healing and never raise (Salil's blocker -- raising there would - # dead-letter real in-flight activity records). fail_on_data_conflict=True - # here mirrors run_repair's contract: a :Node constraint data conflict - # raises a RuntimeError naming `doctor --fix` instead of being logged - # and swallowed. - await ensure_neo4j_schema(app.state.neo4j_driver, fail_on_data_conflict=True) - logger.info("lifespan_startup: Neo4j schema initialized") - # Fail-loud migration-health guard: duplicate nodes are already caught - # above by the :Node constraint (fail_on_data_conflict=True); this catches - # the OTHER un-migrated shape the constraint can't see on its own -- - # nodes that simply lack the :Node label altogether, which violate no - # constraint and so raise nothing by themselves. O(1) via the counts - # store (see count_untagged_nodes) -- this must never regress into the - # AllNodesScan stall PR #67 removed from the write path. + """Manage application lifespan: configure logging and create shared Neo4j driver. + + Deploy-safe boot (council amendment, 2026-08-12): the server MUST boot + and serve regardless of graph migration/reachability state -- a deploy + (restart) must never crash-loop. See + docs/plans/2026-08-12-deploy-safe-boot-spec.md for the full incident and + rationale. Concretely: + + - Schema DDL no longer fails closed on graph *data* state + (``ensure_neo4j_schema(..., fail_on_data_conflict=False)``); a genuine + :Node constraint data conflict is logged loudly and degrades + ``schema_health`` instead of raising. + - The untagged-node probe no longer raises on a positive count; it feeds + the same tri-state health signal (see ``_record_schema_health``). + - B1: the ENTIRE startup body -- from the FIRST statement + (setup_logging), through driver construction and the app.state + assignments, to schema DDL, the untagged probe, the SchemaMeta + baseline, and queue/recovery/reconcile -- is wrapped in ONE + try/except boundary. This is a structural invariant, not a per-site + patch list: whichever startup step fails (an unwritable /data log + dir before the volume is mounted, Neo4j unreachable, a TransientError + during the ACA cold-start race, credential rotation, a corrupt queue + file, ...), the exception is logged LOUDLY and boot proceeds. Nothing + may sit before the boundary and prevent the ASGI app from reaching + `yield` and serving requests. (A live real-process boot test caught + the earlier version where setup_logging + driver construction sat + BEFORE the try and a PermissionError on `/data` crash-looped the + worker with no Neo4j involvement at all.) + - B6: crash-recovery iterates sessions defensively -- a corrupt + per-session offset/dead-letter quarantines THAT session (logged, + skipped) rather than sinking the whole boot; the B1 boundary is the + backstop for anything the per-session guard doesn't catch. + """ + # These MUST be set BEFORE the try so they exist on app.state even if the + # very first statement inside the boundary raises. The shutdown `finally` + # and the /status handler both read them via getattr with defaults, but + # seeding them here keeps the tri-state honest from the first instant. # - # A connectivity/probe failure here is NOT the same as "confirmed - # un-migrated" -- it means graph state could not be determined, not that - # it was determined to be bad -- so it is logged at DEBUG and swallowed - # rather than treated as a corruption finding; the flush path's - # self-heal still covers a genuinely dirty graph once it becomes - # reachable. + # B3/B7: tri-state schema-health defaults -- "unknown" until the startup + # sequence below proves otherwise. A boundary failure leaves these at + # "unknown" (never coerced to healthy) -- see _record_schema_health. + app.state.schema_health = "unknown" + app.state.schema_untagged_nodes = None + app.state.schema_checked_at = None + app.state.schema_degraded_reason = None + # W-2: queue-recovery health is a SEPARATE signal from schema_health -- + # a queue fault is not a schema fault (see the inner try/except around + # the crash-recovery block below). Defaults "healthy"; only the + # queue-recovery block itself may downgrade it to "degraded". + app.state.queue_health = "healthy" + # Driver slots default to None so the shutdown finally can close them + # safely even if construction below never ran (B1: construction is now + # INSIDE the boundary, so it can fail without these ever being assigned). + app.state.neo4j_driver = None + app.state.neo4j_query_driver = None + app.state.neo4j_query_access_mode = None + # W-2: queue-health defaults to "ok"; the inner queue-recovery try/except + # below degrades it independently of schema_health. + app.state.queue_health = "healthy" + # #73: the periodic crash-recovery sweep task is created INSIDE the B1 + # boundary only under a finite ceiling, so seed it to None here so the + # shutdown `finally` can cancel it safely even if startup never got there. + _sweep_task: asyncio.Task[None] | None = None + + # B1: ONE loud try/except boundary around the ENTIRE startup body -- + # setup_logging FIRST, then driver construction, then schema/probe/ + # recovery. Boot NEVER raises regardless of which step fails. try: - untagged = await count_untagged_nodes(app.state.neo4j_driver) - except Exception as exc: # noqa: BLE001 - connectivity probe, not a confirmed bad state - _LOG_MSG = "startup migration-health probe skipped (graph unreachable?): %s" - logger.debug(_LOG_MSG, exc) - untagged = 0 - if untagged: - raise RuntimeError( - f"Neo4j graph has {untagged} node(s) lacking the :Node label " - "(un-migrated). Cold start refuses to boot to avoid duplicating " - "them on write. Run: context-intelligence-server doctor --fix" + # setup_logging() is itself resilient (never raises -- console + # logging is always configured, file logging is best-effort; see + # logging_config.setup_logging), but it lives INSIDE the boundary + # anyway so the invariant holds structurally rather than depending on + # that guarantee. The except handler below uses the module-level + # `logger`, which works regardless of whether setup_logging attached + # any handlers (Python's lastResort emits to stderr as a floor). + setup_logging() + + _admin = _settings.resolve_neo4j_admin() + _query = _settings.resolve_neo4j_query() + logger.info( + "lifespan_startup: creating Neo4j drivers admin_url=%s query_url=%s " + "query_access_mode=%s", + _admin.url, + _query.url, + _query.access_mode, ) - # Crash recovery (decisions #5/#6): on startup, respawn one drainer per - # session that still has an undrained, complete line. The workspace is - # parsed from that session's FIRST log line so the respawned worker is - # bound to the same workspace it was originally created with. - # - # Conservation-counter recovery runs FIRST, and its two steps are - # order-load-bearing: reconcile MUST precede seed. recovery_reconcile_dead - # advances committed offsets past already-dead pending lines so the - # dead-letter counts are settled; only then does recovery_seed_counts read - # disk to reconstruct the accepted/written baseline. Seeding before - # reconciling would leave a residual==1 false DEGRADED. Both run before the - # respawn loop so the respawned drainers start from a conserved baseline. - await registry.queue_manager.recovery_reconcile_dead() - _accepted_seed, _written_seed = await registry.queue_manager.recovery_seed_counts() - registry.seed_counters(_accepted_seed, _written_seed) - recovered = await registry.queue_manager.recover() - # Bound how many drainers this boot respawns (incident: an unbounded - # backlog respawned 94/94 drainers before the server could serve a - # single request, driving a ~4 minute boot and 43.9 GB RSS that tripped - # the OOM killer -- which then never let the backlog shrink because - # every restart repeated the same unbounded respawn). None (the default) - # preserves today's behaviour exactly: every recovered session is - # processed on this boot, unbounded. `recovered` is already sorted - # (QueueManager.recover()), so which sessions are processed this boot - # vs. deferred is deterministic across restarts of the same backlog. - # - # Deferred sessions are NOT touched in any way here -- no read, no - # write, no drainer -- so they remain exactly as durable and - # recoverable as they were before this boot: a later boot's recover() - # call reports them again, and a new event for that session arriving - # via POST /events spawns its drainer immediately via get_or_create(), - # independent of this startup loop. - respawn_limit = _settings.crash_recovery_respawn_limit - if respawn_limit is not None and len(recovered) > respawn_limit: - to_process = recovered[:respawn_limit] - deferred_count = len(recovered) - respawn_limit - else: - to_process = recovered - deferred_count = 0 - respawned = 0 - for sid in to_process: - batch = await registry.queue_manager.read_batch(sid, max_items=1) - if not batch.lines: - continue - if _recover_one_session(sid, batch.lines[0], registry.get_or_create): - respawned += 1 - if deferred_count: - # Loud on purpose (WARNING, not INFO): a deferred backlog must never - # be a silent, un-discoverable fact -- that silence is exactly what - # let the 38 GB spool go unnoticed for two days in the incident this - # guards against. Names the exact counts and the setting to raise. - logger.warning( - "lifespan_startup: crash-recovery respawn cap reached " - "(crash_recovery_respawn_limit=%d): %d/%d respawned this boot, " - "%d session(s) deferred to a later boot (untouched on disk, " - "still fully recoverable). Raise crash_recovery_respawn_limit " - "to respawn more per boot.", - respawn_limit, - respawned, - len(to_process), - deferred_count, - ) - logger.info( - "lifespan_startup: crash recovery respawned %d/%d drainers", - respawned, - len(recovered), - ) - # Periodic deferred-backlog sweep: only meaningful under a FINITE ceiling - # (a deferred tail can exist). With the default unbounded ceiling - # (respawn_limit is None) there is no deferred tail, so NO background task - # is started -- existing deployments are completely unaffected. When a - # finite ceiling IS set, this drains the deferred tail over time instead of - # stranding it until a restart or a new event (see _crash_recovery_sweep_loop - # and config.crash_recovery_sweep_interval_seconds). - _sweep_task: asyncio.Task[None] | None = None - _sweep_interval = _settings.crash_recovery_sweep_interval_seconds - if respawn_limit is not None and _sweep_interval > 0: - _sweep_task = asyncio.create_task( - _crash_recovery_sweep_loop(_sweep_interval, respawn_limit) + # Admin (read/write): schema init + all mutation paths. Keep the + # existing app.state.neo4j_driver NAME so nothing that reads it + # silently breaks. build_neo4j_driver() is the SAME helper + # doctor.run_doctor() uses, so the server and the doctor CLI can + # never construct this connection differently. Driver construction is + # a non-blocking local object build (no network call) -- but it lives + # inside the boundary anyway so a misconfigured URL/auth (e.g. a + # malformed bolt scheme) can never crash-loop boot. + app.state.neo4j_driver = build_neo4j_driver(_admin) + # Cypher-query (read-intent): /cypher + dashboard reads. + app.state.neo4j_query_driver = AsyncGraphDatabase.driver( + _query.url, auth=_query.auth ) + # Stash the resolved query access_mode so /cypher opens READ sessions + # without re-resolving settings on every request. + app.state.neo4j_query_access_mode = _query.access_mode + + # Initialize schema (indexes + uniqueness constraints) BEFORE the + # server starts accepting requests. This ensures the Session + # uniqueness constraint is active before any concurrent flush() + # transactions execute MERGE, which prevents the duplicate-Session- + # node race condition observed under concurrent upload load. logger.info( - "crash_recovery_sweep: enabled (interval=%ds, ceiling=%d) -- " - "deferred backlog will drain progressively, not just on restart", - _sweep_interval, - respawn_limit, + "lifespan_startup: initializing Neo4j schema (indexes + uniqueness constraints)" + ) + # fail_on_data_conflict=False (deploy-safe boot): a genuine :Node + # constraint data conflict is logged loudly by ensure_neo4j_schema + # (B2/B4: it also establishes a fallback idx_node_universal so the + # write path keeps a NodeIndexSeek) and reported via the return + # value instead of raising -- boot must never fail closed on graph + # *data* state. Only run_repair/`doctor --fix` still opts into + # fail_on_data_conflict=True (see ensure_neo4j_schema's docstring). + constraint_established = await ensure_neo4j_schema( + app.state.neo4j_driver, fail_on_data_conflict=False + ) + logger.info("lifespan_startup: Neo4j schema initialized") + + # Migration-health probe: duplicate nodes are already caught above by + # the :Node constraint; this catches the OTHER un-migrated shape the + # constraint can't see on its own -- nodes that simply lack the + # :Node label altogether, which violate no constraint and so raise + # nothing by themselves. O(1) via the counts store (see + # count_untagged_nodes) -- this must never regress into the + # AllNodesScan stall PR #67 removed from the write path. + # + # B3: a probe failure is NOT the same as "confirmed clean" -- it + # means graph state could not be determined. untagged stays None so + # _record_schema_health reports "unknown", never "healthy". + try: + untagged: int | None = await count_untagged_nodes(app.state.neo4j_driver) + except Exception as exc: # noqa: BLE001 - connectivity probe, not confirmed bad state + logger.warning( + "startup migration-health probe failed (graph unreachable? " + "credentials rejected?): %s", + exc, + ) + untagged = None + + _record_schema_health(app, constraint_established, untagged) + + # SchemaMeta baseline singleton (section 10.2 of the cursor-durability + # spec). Deliberately called ONLY here -- startup, single-writer -- NOT + # from ensure_neo4j_schema (which also runs on every Neo4jGraphStore's + # first flush and from doctor --fix; see ensure_schema_version_baseline's + # docstring for why that would be redundant/concurrent instead of + # single-writer). Must run AFTER ensure_neo4j_schema above so the rest of + # the schema (indexes/constraints) is already established. Non-fatal by + # design (see docstring): never raises, so it cannot block server boot. + await ensure_schema_version_baseline(app.state.neo4j_driver) + + # Crash recovery (decisions #5/#6): on startup, respawn one drainer per + # session that still has an undrained, complete line. The workspace is + # parsed from that session's FIRST log line so the respawned worker is + # bound to the same workspace it was originally created with. + # + # Conservation-counter recovery runs FIRST, and its two steps are + # order-load-bearing: reconcile MUST precede seed. recovery_reconcile_dead + # advances committed offsets past already-dead pending lines so the + # dead-letter counts are settled; only then does recovery_seed_counts read + # disk to reconstruct the accepted/written baseline. Seeding before + # reconciling would leave a residual==1 false DEGRADED. Both run before the + # respawn loop so the respawned drainers start from a conserved baseline. + respawn_limit = _settings.crash_recovery_respawn_limit + # W-2: this recovery block is wrapped in its OWN inner try/except, + # separate from the outer B1 boundary. A failure here is a QUEUE + # fault, not a schema fault -- conflating the two (the pre-W-2 + # behavior) made a queue-recovery exception masquerade as + # schema_health="unknown" with reason "startup sequence failed", + # which is operator-misleading. Re-raises nothing: the outer B1 + # boundary is still the backstop for anything this doesn't catch. + try: + await registry.queue_manager.recovery_reconcile_dead() + _accepted_seed, _written_seed = ( + await registry.queue_manager.recovery_seed_counts() + ) + registry.seed_counters(_accepted_seed, _written_seed) + recovered = await registry.queue_manager.recover() + # Bound how many drainers this boot respawns (incident: an unbounded + # backlog respawned 94/94 drainers before the server could serve a + # single request, driving a ~4 minute boot and 43.9 GB RSS that tripped + # the OOM killer -- which then never let the backlog shrink because + # every restart repeated the same unbounded respawn). None (the default) + # preserves today's behaviour exactly: every recovered session is + # processed on this boot, unbounded. `recovered` is already sorted + # (QueueManager.recover()), so which sessions are processed this boot + # vs. deferred is deterministic across restarts of the same backlog. + # + # Deferred sessions are NOT touched in any way here -- no read, no + # write, no drainer -- so they remain exactly as durable and + # recoverable as they were before this boot: a later boot's recover() + # call reports them again, and a new event for that session arriving + # via POST /events spawns its drainer immediately via get_or_create(), + # independent of this startup loop. + if respawn_limit is not None and len(recovered) > respawn_limit: + to_process = recovered[:respawn_limit] + deferred_count = len(recovered) - respawn_limit + else: + to_process = recovered + deferred_count = 0 + respawned = 0 + for sid in to_process: + # B6: iterate defensively -- a corrupt per-session .offset/ + # dead-letter quarantines THAT session (logged, skipped) rather + # than sinking the whole boot via an unhandled exception here. + try: + batch = await registry.queue_manager.read_batch(sid, max_items=1) + if not batch.lines: + continue + if _recover_one_session(sid, batch.lines[0], registry.get_or_create): + respawned += 1 + except Exception: # quarantine this session, don't sink boot + logger.exception("recovery_session_quarantined session=%s", sid) + if deferred_count: + # Loud on purpose (WARNING, not INFO): a deferred backlog must never + # be a silent, un-discoverable fact -- that silence is exactly what + # let the 38 GB spool go unnoticed for two days in the incident this + # guards against. Names the exact counts and the setting to raise. + logger.warning( + "lifespan_startup: crash-recovery respawn cap reached " + "(crash_recovery_respawn_limit=%d): %d/%d respawned this boot, " + "%d session(s) deferred to a later boot (untouched on disk, " + "still fully recoverable). Raise crash_recovery_respawn_limit " + "to respawn more per boot.", + respawn_limit, + respawned, + len(to_process), + deferred_count, + ) + logger.info( + "lifespan_startup: crash recovery respawned %d/%d drainers", + respawned, + len(recovered), + ) + except Exception: # W-2: a queue fault, not a schema fault + logger.exception( + "queue_recovery_degraded: queue-recovery sequence failed but " + "boot continues (deploy-safe boot invariant) -- this is a " + "queue-health signal, distinct from schema_health" + ) + app.state.queue_health = "degraded" + + # WS-3a: wire the live admin driver into the maintenance coordinator + # so the gate/status probe can run. Placed after schema-health is + # recorded (not inside the W-2 block above) so a queue-recovery + # failure never prevents the coordinator from being bound -- the + # maintenance gate must reflect graph/schema state, not queue state. + coordinator.bind_driver( + app.state.neo4j_driver, + untagged=untagged, + probe_ttl_seconds=_settings.maintenance_probe_ttl_seconds, + ) + # Periodic deferred-backlog sweep: only meaningful under a FINITE ceiling + # (a deferred tail can exist). With the default unbounded ceiling + # (respawn_limit is None) there is no deferred tail, so NO background task + # is started -- existing deployments are completely unaffected. When a + # finite ceiling IS set, this drains the deferred tail over time instead + # of stranding it until a restart or a new event (see + # _crash_recovery_sweep_loop and config.crash_recovery_sweep_interval_seconds). + _sweep_interval = _settings.crash_recovery_sweep_interval_seconds + if respawn_limit is not None and _sweep_interval > 0: + _sweep_task = asyncio.create_task( + _crash_recovery_sweep_loop(_sweep_interval, respawn_limit) + ) + logger.info( + "crash_recovery_sweep: enabled (interval=%ds, ceiling=%d) -- " + "deferred backlog will drain progressively, not just on restart", + _sweep_interval, + respawn_limit, + ) + except Exception as exc: # B1: deploy-safe boot -- NEVER crash-loop + logger.exception( + "startup_degraded: lifespan startup sequence failed but boot " + "continues (deploy-safe boot invariant -- the server must never " + "crash-loop on graph/data state)" ) + app.state.schema_health = "unknown" + app.state.schema_degraded_reason = f"startup sequence failed: {exc}" + app.state.schema_checked_at = datetime.now(UTC).isoformat() + try: yield finally: + # Cancel the periodic crash-recovery sweep (if one was started) before + # closing drivers. It is created INSIDE the B1 boundary, so a startup + # failure may leave it as None -- guard for that. CancelledError is the + # expected result of cancel() and is suppressed. if _sweep_task is not None: _sweep_task.cancel() with suppress(asyncio.CancelledError): await _sweep_task + # Defensive shutdown: driver construction is INSIDE the B1 boundary + # now, so either driver may be None (construction failed or never + # ran). Close only what exists, and never let a close() error escape + # shutdown -- it is not a boot concern and must not mask the reason + # the app is shutting down. logger.info("lifespan_shutdown: closing Neo4j drivers") - await app.state.neo4j_driver.close() - await app.state.neo4j_query_driver.close() + for _attr in ("neo4j_driver", "neo4j_query_driver"): + _driver = getattr(app.state, _attr, None) + if _driver is None: + continue + try: + await _driver.close() + except Exception: # shutdown best-effort -- never raise here + logger.exception("lifespan_shutdown: error closing %s", _attr) app = FastAPI( @@ -387,6 +589,12 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: app.include_router(admin_router) app.include_router(version_router) app.include_router(queues_router) +# WS-3a: registered on `app` itself, NOT on the auth-wrapped `asgi_app` -- +# so it cannot be bypassed by the bare `main:app` entrypoint (the same class +# of bug the auth-fold fix below addresses one layer over). BearerTokenMiddleware +# wraps `app`, so auth still runs FIRST; an unauthenticated request 401s +# before it ever reaches this gate. +app.middleware("http")(maintenance_gate_middleware) _start_time = time.time() registry = SessionRegistry() # Expose the registry singleton on app.state so routers can read it via @@ -462,6 +670,26 @@ def _assert_admin_not_exempt() -> None: ) +def _assert_maintenance_endpoint_allow_listed() -> None: + """Startup assertion (WS-3a MUST-FIX #1): the maintenance-mode allow-list + must always contain ``/admin/maintenance``, ``/status``, and ``/version``. + + Called by ``create_asgi_app`` before constructing the middleware, mirroring + ``_assert_admin_not_exempt`` above. Without this, ``/admin/maintenance`` + could accidentally be gated by its own allow-list -- 503ing at precisely + the moment it exists to unblock, recreating the ACA deadlock this + endpoint was built to close. + """ + required = {"/admin/maintenance", "/status", "/version"} + missing = required - MAINTENANCE_ALLOW_LIST + if missing: + raise RuntimeError( + f"Availability invariant violated: {sorted(missing)!r} missing from " + f"maintenance.MAINTENANCE_ALLOW_LIST -- these paths must never be " + f"gated by maintenance mode." + ) + + def _assert_neo4j_clients_explicit(settings: Settings) -> None: """Startup assertion (doc 11 gap #12): the deployed profile MUST declare the structured neo4j.admin / neo4j.cypher_query clients explicitly. @@ -521,6 +749,9 @@ def create_asgi_app( # This runs before any middleware construction so the failure is loud and # immediate — no request ever reaches an unauthenticated /admin endpoint. _assert_admin_not_exempt() + # WS-3a MUST-FIX #1: /admin/maintenance (and /status, /version) must never + # be gated by maintenance mode -- same defence-in-depth pattern as above. + _assert_maintenance_endpoint_allow_listed() s = settings if settings is not None else _settings _assert_neo4j_clients_explicit(s) @@ -851,6 +1082,75 @@ async def get_status(request: Request) -> dict[str, Any]: else {} ), } + # WS-3a: mode/schema_health are now DE-LATCHED -- sourced from the live, + # TTL-cached MaintenanceCoordinator probe instead of the boot-only + # snapshot. This self-clears after an out-of-band repair (`doctor --fix` + # or `POST /admin/maintenance`) within the probe's TTL, with NO restart + # required -- the exact latch this replaces. + # + # *** schema_health/mode MUST NOT be wired to a Kubernetes/ACA *** + # *** liveness or readiness probe. Doing so would recreate the exact *** + # *** crash-loop the deploy-safe-boot fix removes, one layer up *** + # *** (see docs/azure-deployment.md). *** + _maint = await coordinator.status() + response["mode"] = _maint.mode + response["maintenance_started_at"] = _maint.started_at + response["maintenance_elapsed_seconds"] = _maint.elapsed_seconds + if _maint.constraint_present is None: + response["schema_health"] = "unknown" + elif _maint.constraint_present is False: + response["schema_health"] = "degraded" + elif (_maint.untagged_nodes or 0) > 0: + response["schema_health"] = "degraded" + else: + response["schema_health"] = "healthy" + # untagged_nodes is now the LIVE (TTL-cached) count from the same + # coordinator probe that drives mode/schema_health -- NOT the boot-time + # app.state snapshot. This is the second half of the de-latch: after an + # out-of-band repair (POST /admin/maintenance or `doctor --fix`) the count + # self-clears within one probe TTL, with NO restart. (Previously this + # stayed pinned to the boot value, so schema_health/untagged_nodes kept + # reporting `degraded` until the process restarted.) + response["untagged_nodes"] = _maint.untagged_nodes + # Live probe timestamp (bounded staleness <= the probe TTL), replacing + # the old boot-only snapshot timestamp. + response["schema_checked_at"] = datetime.now(UTC).isoformat() + # degraded_reason is sourced from the SAME live coordinator probe that + # drives mode/schema_health above (_maint.reason), NOT the boot-time + # app.state snapshot. The snapshot version goes stale after an + # out-of-band repair (POST /admin/maintenance or `doctor --fix`): mode + # correctly de-latches to "healthy" but the boot-time reason string kept + # asserting a constraint-absent condition that was no longer true. This + # is the same reason MaintenanceCoordinator.status() already produces + # for the 503 body (maintenance_response), so /status and the 503 stay + # consistent -- and it naturally clears to None once the live probe + # confirms the constraint is present and no maintenance op is running. + response["degraded_reason"] = _maint.reason + # W-2: queue-recovery health is reported SEPARATELY from schema_health -- + # a queue-recovery fault at boot is not a schema fault (see lifespan). + response["queue_health"] = getattr(request.app.state, "queue_health", "healthy") + # W-4: ADVISORY drift signal ONLY, NOT a guard. Surface the STORED + # :SchemaMeta.schema_version (read fresh from the graph) next to the + # server's compiled-in SCHEMA_VERSION so a server/graph mismatch is + # DETECTABLE by automation -- no gating, no migration, no behavior + # change of any kind results from this. `read_graph_schema_version` is a + # separate read-only helper (neo4j_store.py); it is intentionally NOT + # wired into `ensure_schema_version_baseline`'s write path or into + # `GET /version` (both stay exactly as they were -- see the read/write + # separation documented on `ensure_schema_version_baseline`). Full + # mismatch handling/migration is deferred (tracked separately). + _schema_driver = getattr(request.app.state, "neo4j_driver", None) + _graph_schema_version = ( + await read_graph_schema_version(_schema_driver) + if _schema_driver is not None + else None + ) + response["graph_schema_version"] = _graph_schema_version + response["schema_version_current"] = ( + None + if _graph_schema_version is None + else _graph_schema_version == SCHEMA_VERSION + ) return response @@ -913,6 +1213,12 @@ async def post_events( body = await http_request.body() body_obj = json.loads(body) body_obj["created_by"] = contributor_id # overwrite, never setdefault + # I1: lift the optional top-level working_dir envelope field into body_obj["data"] + # so it rides the existing data pipeline (registry's _parse_line extracts "data" + # wholesale) and reaches ensure_session_node. Forward-only: only set when the client + # supplied it; absent/empty leaves Session.working_dir null. + if request.working_dir and isinstance(body_obj.get("data"), dict): + body_obj["data"]["working_dir"] = request.working_dir body = json.dumps(body_obj, separators=(",", ":")).encode() await registry.queue_manager.append(worker_key, body) registry.record_accepted() # count the durably-accepted event diff --git a/context_intelligence_server/maintenance.py b/context_intelligence_server/maintenance.py new file mode 100644 index 00000000..4c2cba37 --- /dev/null +++ b/context_intelligence_server/maintenance.py @@ -0,0 +1,527 @@ +"""Maintenance-mode coordinator -- the single seam for gate + status + op state. + +This module owns ALL maintenance-mode state: the tri-state live constraint +probe, the current-mode derivation, the single-flight maintenance-operation +record, and the structured 503 response. It is deliberately the ONLY place +this state lives so that the HTTP gate (``maintenance_gate_middleware``), the +drain-loop gate (``registry.drain_worker``), and the ``/status`` / +``GET /admin/maintenance`` surfaces can never drift from one another. + +Constraints (do not violate): +- This module MUST NOT import ``registry`` -- ``registry`` imports this + module, never the reverse. +- This module MUST NOT contain graph-mutation logic. Migration/repair logic + lives in ``neo4j_store.run_repair``; this module only reads (the + constraint-presence probe) and tracks in-process op state. + +Swappability (council D2, multi-replica deferred): every call site touches +only ``gate_closed`` / ``status`` / ``try_begin_op`` / ``finish_op`` / +``current_op``. Replacing the in-process op record with a graph-store-backed +lock node later means reimplementing the body of those five methods -- +zero call-site changes. + +No driver bound -- the load-bearing no-regression property: tests that never +run ``lifespan`` never call ``bind_driver``, so the probe always returns +``None`` (unknown) and the gate is never closed. This keeps the ENTIRE +existing unit-test suite's behavior unchanged. +""" + +from __future__ import annotations + +import asyncio +import logging +import time +import uuid +from collections.abc import Awaitable, Callable +from dataclasses import dataclass +from datetime import UTC, datetime +from typing import Any, Literal + +from fastapi import Request +from fastapi.responses import JSONResponse, Response + +from context_intelligence_server.config import get_settings +from context_intelligence_server.neo4j_store import count_untagged_nodes + +logger = logging.getLogger("context_intelligence_server") + +# --------------------------------------------------------------------------- +# Types +# --------------------------------------------------------------------------- + +MaintenanceMode = Literal["healthy", "maintenance", "degraded", "unknown"] +OpState = Literal["unknown", "running", "succeeded", "failed"] + + +@dataclass(frozen=True) +class OpRecord: + """Snapshot of the (per-process) maintenance operation state. + + ``state`` initializes to "unknown" (council D4) so "never ran" is + distinguishable from "ran, record lost to a crash" (which would also + read as a false "succeeded" if initialized there instead). + """ + + state: OpState + run_id: str | None # uuid4 hex; freshness marker AND future fencing token + started_at: str | None # ISO-8601 UTC + completed_at: str | None # ISO-8601 UTC; set ONLY on the genuine-execution path + records_affected: int | None + error: str | None # human-readable; persists across "failed" + + +@dataclass(frozen=True) +class MaintenanceStatus: + """Snapshot of the current maintenance mode, returned by ``status()``.""" + + mode: MaintenanceMode + constraint_present: bool | None # None == probe could not answer + reason: str | None # human-readable cause of the current mode + started_at: str | None # when the CURRENT maintenance window opened + elapsed_seconds: float | None # None when not in maintenance + op: OpRecord + # Live (TTL-cached) untagged-node count backing the `degraded` term. + # Defaulted so existing MaintenanceStatus(...) constructions in tests keep + # working; None == the probe could not answer (or no driver bound). + untagged_nodes: int | None = None + + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- + +_CONSTRAINT_NAME = "node_node_id_workspace_unique" +_PROBE_CYPHER = ( + "SHOW CONSTRAINTS YIELD name " + f"WHERE name = '{_CONSTRAINT_NAME}' RETURN count(*) AS c" +) +_PROBE_TTL_SECONDS = 5.0 # default; overridable per-instance via bind_driver() +_RETRY_AFTER_DEFAULT = ( + 30 # fallback; live value is settings.maintenance_retry_after_seconds +) + +# Allow-list, not deny-list: a deny-list is scatter-and-miss (spec sec 3a-1). +# Any route added later to the app is blocked-by-default -- the safe direction. +MAINTENANCE_ALLOW_LIST: frozenset[str] = frozenset( + {"/status", "/version", "/admin/maintenance", "/docs", "/openapi.json"} +) + + +def _now_iso() -> str: + return datetime.now(UTC).isoformat() + + +# --------------------------------------------------------------------------- +# The seam +# --------------------------------------------------------------------------- + + +class MaintenanceCoordinator: + """The seam. See module docstring for the swappability contract.""" + + def __init__(self) -> None: + self._driver: Any = None + self._boot_untagged: int | None = None + self._probe_ttl_seconds: float = _PROBE_TTL_SECONDS + + # TTL-cached, single-flight probe state (constraint presence). + self._probe_lock = asyncio.Lock() + self._cache_populated: bool = False + self._cached_present: bool | None = None + self._cache_expires_at: float = 0.0 # monotonic clock + + # TTL-cached, single-flight probe state for the untagged-node count -- + # a SEPARATE cache from the constraint probe above (independent fate: + # a count failure must not poison the constraint signal, and vice + # versa). This is what de-latches the degraded/untagged half of the + # health signal: it self-clears within one TTL after an out-of-band + # repair, exactly like the constraint probe, instead of staying pinned + # to the boot-time snapshot until a restart. count_untagged_nodes is + # O(1) via Neo4j's counts store (see its docstring), so probing it on + # the cached path is as cheap as the constraint catalog read. + self._untagged_lock = asyncio.Lock() + self._untagged_cache_populated: bool = False + self._cached_untagged: int | None = None + self._untagged_cache_expires_at: float = 0.0 # monotonic clock + + # Op state -- init "unknown" (council D4: never-run != crash-lost). + self._op = OpRecord( + state="unknown", + run_id=None, + started_at=None, + completed_at=None, + records_affected=None, + error=None, + ) + + # Maintenance-window bookkeeping (for /status + transition logging). + self._window_started_at: str | None = None + self._window_started_monotonic: float | None = None + + # WS-3c: strong references to in-flight maintenance-op background + # tasks. asyncio only holds a WEAK reference to a task created via + # ``asyncio.create_task`` -- without an external strong ref the task + # can be garbage-collected mid-run. ``retain_task`` holds it here and + # a completion callback discards it so this set never grows unbounded. + self._background_tasks: set[asyncio.Task[Any]] = set() + + # -- binding -------------------------------------------------------- + + def bind_driver( + self, + driver: Any, + *, + untagged: int | None = None, + probe_ttl_seconds: float | None = None, + ) -> None: + """Wire the live admin Neo4j driver (called from ``lifespan``). + + ``untagged`` is the boot-time untagged-node count already computed by + ``main._record_schema_health`` -- injected here so this module never + imports ``main`` (would be circular). ``probe_ttl_seconds`` overrides + the default TTL (normally sourced from + ``settings.maintenance_probe_ttl_seconds`` by the caller). + + Tests that never call this leave the driver unbound: the probe then + always returns ``None`` (unknown) and the gate stays OPEN -- the + load-bearing no-regression property for the existing suite. + """ + self._driver = driver + self._boot_untagged = untagged + if probe_ttl_seconds is not None: + self._probe_ttl_seconds = probe_ttl_seconds + # Rebinding invalidates any cached constraint probe result. + self._cache_populated = False + self._cached_present = None + self._cache_expires_at = 0.0 + # Seed the untagged cache with the boot-time count so the FIRST + # /status (before any live re-probe) reports the same value the old + # boot snapshot did -- then let it expire after one TTL so the live + # probe takes over and de-latches it. Seeding as populated (not cold) + # keeps the existing TTL-cache tests' hit-counts unchanged for the + # first in-window call. + self._cached_untagged = untagged + self._untagged_cache_populated = True + self._untagged_cache_expires_at = time.monotonic() + self._probe_ttl_seconds + + # -- the constraint probe -------------------------------------------- + + async def _run_probe(self) -> bool | None: + """One live catalog read. Tri-state: True / False / None (unknown). + + ``SHOW CONSTRAINTS`` is a catalog read -- no data scan, no dependence + on graph size. Unlike ``count_untagged_nodes``/``count_duplicate_nodes`` + (neo4j_store.py), this MUST be cheap enough to run on the request + path; it must never become an ``AllNodesScan``. + """ + if self._driver is None: + return None + try: + async with self._driver.session() as session: + result = await session.run(_PROBE_CYPHER) + count = 0 + async for record in result: + count = record["c"] + return count > 0 + except Exception as exc: # noqa: BLE001 -- connectivity probe, not confirmed bad state + logger.warning("maintenance_probe_failed error=%s", exc) + return None + + async def _probe_constraint_present(self) -> bool | None: + """TTL-cached, single-flight wrapper around ``_run_probe``. + + Double-checked locking: the fast path (warm cache) never touches the + lock, so N concurrent callers with a warm cache never contend on it. + Only callers that observe an expired/empty cache take the lock, and + the re-check immediately after acquiring collapses concurrent + expiry-time callers into exactly one live probe. + """ + now = time.monotonic() + if self._cache_populated and now < self._cache_expires_at: + return self._cached_present + async with self._probe_lock: + now = time.monotonic() + if self._cache_populated and now < self._cache_expires_at: + return self._cached_present + result = await self._run_probe() + self._cached_present = result + self._cache_populated = True + self._cache_expires_at = time.monotonic() + self._probe_ttl_seconds + return result + + # -- the untagged-node probe (de-latches the degraded half) ---------- + + async def _run_untagged_probe(self) -> int | None: + """One live untagged-node count. Tri-state: int / None (unknown). + + ``count_untagged_nodes`` is O(1) via Neo4j's counts store (total minus + :Node count), NOT the ``WHERE NOT n:Node`` AllNodesScan -- safe on the + cached request path. A probe failure returns None (unknown, no + ``degraded`` signal fabricated) exactly as the constraint probe does; + it is caught here so it can never poison the independent constraint + signal. + """ + if self._driver is None: + return None + try: + return await count_untagged_nodes(self._driver) + except Exception as exc: # noqa: BLE001 -- connectivity probe, not confirmed bad state + logger.warning("maintenance_untagged_probe_failed error=%s", exc) + return None + + async def _probe_untagged(self) -> int | None: + """TTL-cached, single-flight wrapper around ``_run_untagged_probe``. + + Same double-checked-locking shape as ``_probe_constraint_present`` but + with its OWN cache/lock so the two probes never share fate. Seeded at + ``bind_driver`` with the boot count, then live-refreshed each TTL -- + this is what lets an out-of-band repair clear ``degraded`` without a + restart (the latch this fixes). + """ + now = time.monotonic() + if self._untagged_cache_populated and now < self._untagged_cache_expires_at: + return self._cached_untagged + async with self._untagged_lock: + now = time.monotonic() + if self._untagged_cache_populated and now < self._untagged_cache_expires_at: + return self._cached_untagged + result = await self._run_untagged_probe() + self._cached_untagged = result + self._untagged_cache_populated = True + self._untagged_cache_expires_at = time.monotonic() + self._probe_ttl_seconds + return result + + # -- mode derivation (single source of truth for /status + the gate) -- + + def _handle_transition( + self, mode: MaintenanceMode, reason: str | None, run_id: str | None + ) -> None: + """Detect + log open<->closed transitions exactly once each. + + No ``await`` anywhere in this method: it cannot be preempted + mid-execution by another coroutine, so two concurrent callers + observing the same transition can never both log it (whichever runs + first flips ``_window_started_at``; the second then sees the + already-updated state and no-ops). + """ + is_maintenance = mode == "maintenance" + was_maintenance = self._window_started_at is not None + if is_maintenance and not was_maintenance: + self._window_started_at = _now_iso() + self._window_started_monotonic = time.monotonic() + logger.info( + "maintenance_entered", + extra={ + "reason": reason, + "run_id": run_id, + "trigger": "op" if self._op.state == "running" else "constraint", + }, + ) + elif not is_maintenance and was_maintenance: + duration = ( + time.monotonic() - self._window_started_monotonic + if self._window_started_monotonic is not None + else None + ) + logger.info( + "maintenance_completed", + extra={ + "reason": reason, + "run_id": run_id, + "duration_seconds": duration, + }, + ) + self._window_started_at = None + self._window_started_monotonic = None + + async def _derive_mode( + self, + ) -> tuple[MaintenanceMode, str | None, bool | None, int | None]: + """The ONE place mode is computed. Both ``gate_closed`` and + ``status`` call this so a transition is caught no matter which + surface is being polled (spec sec 2.4).""" + op = self._op + constraint_present = await self._probe_constraint_present() + # untagged is only load-bearing for the `degraded` term, which only + # applies when the constraint IS present. Probing it only in that + # branch keeps the absent/unknown/op-running paths (and their existing + # TTL-cache hit-count tests) untouched, and avoids a needless count on + # a graph we already know is in maintenance. + untagged: int | None = None + + if op.state == "running": + mode: MaintenanceMode = "maintenance" + reason = "maintenance operation in progress" + elif constraint_present is False: + mode = "maintenance" + reason = ":Node uniqueness constraint absent -- migration required" + elif constraint_present is None: + mode = "unknown" + reason = "constraint probe could not determine graph state" + else: + # constraint present: consult the LIVE (TTL-cached) untagged count + # so an out-of-band repair de-latches degraded->healthy with no + # restart. None (probe could not answer) is NOT coerced to + # degraded -- no evidence, so healthy stands. + untagged = await self._probe_untagged() + if untagged is not None and untagged > 0: + mode = "degraded" + reason = f"{untagged} node(s) lacking the :Node label" + else: + mode = "healthy" + reason = None + + self._handle_transition(mode, reason, op.run_id) + return mode, reason, constraint_present, untagged + + # -- public seam ------------------------------------------------------ + + async def gate_closed(self) -> bool: + """True iff ingest/query must be refused right now. + + ``gate_closed() == op_running (live) OR constraint_absent (TTL-cached)``. + ``degraded`` and ``unknown`` do NOT close the gate -- only + ``mode == "maintenance"`` does (D-C, D-E). + """ + mode, _reason, _constraint_present, _untagged = await self._derive_mode() + return mode == "maintenance" + + async def status(self) -> MaintenanceStatus: + """Full snapshot for ``/status`` and ``GET /admin/maintenance``.""" + mode, reason, constraint_present, untagged = await self._derive_mode() + elapsed = ( + time.monotonic() - self._window_started_monotonic + if self._window_started_monotonic is not None + else None + ) + return MaintenanceStatus( + mode=mode, + constraint_present=constraint_present, + reason=reason, + started_at=self._window_started_at, + elapsed_seconds=elapsed, + op=self._op, + untagged_nodes=untagged, + ) + + def try_begin_op(self) -> str | None: + """Synchronous single-flight CAS -- begin an op iff none is running. + + No ``await`` between the check and the set: in asyncio this makes + the check-and-set atomic (D-G / MUST-FIX #3). Per-process only; + ``run_id`` is the future multi-replica fencing token. + """ + if self._op.state == "running": + return None + run_id = uuid.uuid4().hex + self._op = OpRecord( + state="running", + run_id=run_id, + started_at=_now_iso(), + completed_at=None, + records_affected=None, + error=None, + ) + return run_id + + def finish_op( + self, run_id: str, *, records_affected: int | None, error: str | None + ) -> None: + """Record the outcome of the op started by ``try_begin_op``. + + Sets ``completed_at`` -- the ONLY place it is written, which is what + makes ``completed_at`` a genuine freshness marker (D-H). A + run_id mismatch (stale/foreign completion signal) is logged and + ignored rather than corrupting the current op record. + """ + if self._op.run_id != run_id: + logger.warning( + "maintenance_finish_op_run_id_mismatch expected=%s got=%s", + self._op.run_id, + run_id, + ) + return + self._op = OpRecord( + state="failed" if error else "succeeded", + run_id=run_id, + started_at=self._op.started_at, + completed_at=_now_iso(), + records_affected=records_affected, + error=error, + ) + + def current_op(self) -> OpRecord: + return self._op + + def retain_task(self, task: asyncio.Task[Any]) -> None: + """Hold a strong reference to an in-flight maintenance-op task. + + ``asyncio.create_task`` only returns a task the event loop tracks + weakly; with no other strong reference, the task object can be + garbage-collected mid-run (a well-known asyncio footgun -- see the + "Important" note in the stdlib ``asyncio.create_task`` docs). The + caller (``routers/admin.py``) MUST call this immediately after + creating the task. The completion callback discards the reference + once the task finishes, so this set never grows unbounded. + """ + self._background_tasks.add(task) + task.add_done_callback(self._background_tasks.discard) + + +# Module singleton -- the ONE coordinator instance shared by the HTTP gate, +# the drain-loop gate, and /status. +coordinator: MaintenanceCoordinator = MaintenanceCoordinator() + + +# --------------------------------------------------------------------------- +# The structured 503 (one producer) +# --------------------------------------------------------------------------- + + +def maintenance_response(status: MaintenanceStatus, retry_after: int) -> JSONResponse: + """The ONE producer of the maintenance 503. + + Deliberately NOT ``HTTPException(503, detail=...)`` -- that renders + ``{"detail": ...}``, the wrong contract for this response. + """ + schema_health = "unknown" if status.constraint_present is None else "degraded" + return JSONResponse( + status_code=503, + content={ + "status": "maintenance", + "reason": status.reason, + "retry_after": retry_after, + "schema_health": schema_health, + "maintenance_started_at": status.started_at, + }, + headers={"Retry-After": str(retry_after)}, + ) + + +# --------------------------------------------------------------------------- +# HTTP gate middleware +# --------------------------------------------------------------------------- + + +async def maintenance_gate_middleware( + request: Request, call_next: Callable[[Request], Awaitable[Response]] +) -> Response: + """Allow-list middleware: refuse every non-allow-listed path while the + coordinator reports ``mode == "maintenance"``. + + Registered on ``app`` itself (not the auth-wrapped ASGI app) so it cannot + be bypassed by the bare ``app`` entrypoint -- see WS-3a spec sec 3a-1. + Blocks (non-exhaustive; see docs/maintenance-mode.md once shipped): + ``POST /events``, ``POST /cypher``, ``GET /blobs/*``, ``GET/POST + /queues/*`` (including dead-letter replay), and all ``/admin/*`` except + ``/admin/maintenance``. + """ + if request.url.path in MAINTENANCE_ALLOW_LIST: + return await call_next(request) + st = await coordinator.status() + if st.mode == "maintenance": + retry_after = getattr( + get_settings(), "maintenance_retry_after_seconds", _RETRY_AFTER_DEFAULT + ) + return maintenance_response(st, retry_after) + return await call_next(request) diff --git a/context_intelligence_server/maintenance_ops.py b/context_intelligence_server/maintenance_ops.py new file mode 100644 index 00000000..5c8007a3 --- /dev/null +++ b/context_intelligence_server/maintenance_ops.py @@ -0,0 +1,79 @@ +"""Maintenance operation execution -- the ONE shared logic home for running +the maintenance repair operation, used by both ``POST /admin/maintenance`` +(WS-3c) and, later, the standalone out-of-band migration script (spec +sec 5.4, not built in this change). + +This module writes NO new dedup/repair algorithm (D-F, WS-3 spec sec 0/5.1): +it wraps the existing ``neo4j_store.run_repair`` with exactly the two pieces +of bookkeeping an *in-process, gated* caller needs that a standalone script +would not: + +- a bounded pre-op quiesce sleep (spec sec 5.3) to let ordinary in-flight + flushes land before the schema DDL runs, and +- recording the outcome via the ``MaintenanceCoordinator`` seam + (``finish_op``), so ``GET /admin/maintenance`` and ``/status`` observe it. + +Kept standalone-friendly on purpose: ``run_maintenance_operation`` takes the +driver and coordinator explicitly (no FastAPI ``Request``, no import of +``main`` or ``routers.admin``), so a future standalone script can reuse it +directly if it ever wants coordinator-aware bookkeeping. Per D-D/5.2, no +bypass mechanism exists here or anywhere else -- this function reaches Neo4j +only through the driver it is given (the admin driver, ``app.state.neo4j_driver`` +at the HTTP call site) and never consults ``coordinator.gate_closed()``. +""" + +from __future__ import annotations + +import asyncio +import logging +from typing import Any + +from context_intelligence_server.maintenance import MaintenanceCoordinator, coordinator +from context_intelligence_server.neo4j_store import run_repair + +logger = logging.getLogger("context_intelligence_server") + + +async def run_maintenance_operation( + driver: Any, + run_id: str, + *, + quiesce_seconds: float, + database: str = "neo4j", + coord: MaintenanceCoordinator = coordinator, +) -> None: + """Run one maintenance operation to completion and record its outcome. + + Args: + driver: The Neo4j admin driver (``app.state.neo4j_driver`` at the + HTTP call site). Never the query-only driver. + run_id: The run id returned by ``coord.try_begin_op()`` -- the + caller MUST have already won the CAS before scheduling this. + quiesce_seconds: Seconds to sleep before calling ``run_repair`` (spec + sec 5.3). Pass 0 to skip (e.g. direct unit testing). + database: Neo4j database name, forwarded to ``run_repair``. + coord: The coordinator instance to report back to. Defaults to the + process-wide singleton; overridable for tests. + + On success, calls ``coord.finish_op(run_id, records_affected=n, error=None)`` + with ``n = duplicates_removed + nodes_tagged`` (spec sec 6.1). On any + exception, calls ``coord.finish_op(run_id, records_affected=None, + error=str(exc))`` -- the op is recorded ``failed``, never silently lost, + and the gate stays closed (``op_running`` only clears via ``finish_op``) + until an operator retries. + """ + if quiesce_seconds > 0: + logger.info("maintenance_quiesce run_id=%s seconds=%s", run_id, quiesce_seconds) + await asyncio.sleep(quiesce_seconds) + try: + result = await run_repair(driver, database=database) + records_affected = result["duplicates_removed"] + result["nodes_tagged"] + coord.finish_op(run_id, records_affected=records_affected, error=None) + logger.info( + "maintenance_op_succeeded run_id=%s records_affected=%s", + run_id, + records_affected, + ) + except Exception as exc: + logger.exception("maintenance_op_failed run_id=%s", run_id) + coord.finish_op(run_id, records_affected=None, error=str(exc)) diff --git a/context_intelligence_server/models.py b/context_intelligence_server/models.py index 03cf0793..95d5bed8 100644 --- a/context_intelligence_server/models.py +++ b/context_intelligence_server/models.py @@ -14,10 +14,19 @@ class EventRequest(BaseModel): The Amplifier client must always supply workspace on every event. Events without workspace (e.g. an incorrectly configured hook) are rejected at the endpoint with HTTP 422. + + working_dir is OPTIONAL (I1) — the bundle hook emits it + as a top-level envelope field alongside workspace, but older clients/events + won't have it. Absent/empty is fine and leaves the Session node's + working_dir property null. Populate-if-missing: the Session node's + working_dir is filled in by the first subsequent event (including a + re-import via the upload CLI) that carries a non-empty value, but an + already-populated value is never overwritten. """ event: str workspace: str + working_dir: str | None = None idempotency_key: str | None = None data: dict[str, Any] @@ -29,6 +38,20 @@ def workspace_must_not_be_empty(cls, v: str) -> str: raise ValueError("workspace must not be empty") return v + @field_validator("working_dir") + @classmethod + def working_dir_must_not_be_blank(cls, v: str | None) -> str | None: + """Allow ``None`` (working_dir is optional, unlike workspace) but reject + blank/whitespace-only strings. + + Mirrors ``workspace_must_not_be_empty``'s normalize-or-reject stance: a + whitespace-only value (e.g. ``" "``) is never a legitimate path and must + not write through to the Session node verbatim. + """ + if v is not None and not v.strip(): + raise ValueError("working_dir must not be blank") + return v + class EventResponse(BaseModel): """Response returned after an event is accepted.""" diff --git a/context_intelligence_server/neo4j_store.py b/context_intelligence_server/neo4j_store.py index 52ddc7d9..00c84cda 100644 --- a/context_intelligence_server/neo4j_store.py +++ b/context_intelligence_server/neo4j_store.py @@ -15,13 +15,15 @@ import logging import re from collections.abc import Generator -from datetime import datetime +from datetime import UTC, datetime from typing import Any, LiteralString, cast from neo4j import AsyncGraphDatabase from neo4j import unit_of_work as _unit_of_work from neo4j.exceptions import DriverError, Neo4jError +from context_intelligence_server.status import SCHEMA_VERSION + _LOG = logging.getLogger(__name__) # --------------------------------------------------------------------------- @@ -596,21 +598,35 @@ async def ensure_neo4j_schema( Because the dedup/backfill no longer run here, the ``:Node`` uniqueness constraint (Step 6) can fail if the graph still has untagged/duplicate legacy nodes. Whether that failure raises or is swallowed depends on - *fail_on_data_conflict* (see below) -- this function has TWO callers with - OPPOSITE correctness requirements: - - - **Cold start** (the lifespan handler, ``main.py``): must fail CLOSED. - Refusing to boot against an un-migrated graph is correct -- pass - ``fail_on_data_conflict=True``. - - **Mid-flight flush** (``Neo4jGraphStore._ensure_schema``, called from - inside ``_flush_body``'s try block on every flush until the schema - latches): must NOT raise. A ``RuntimeError`` escaping here propagates - through ``_flush_body``'s ``except ... raise`` and gets counted as a - flush failure, which dead-letters the entire in-flight batch of real - activity records -- data loss, not merely a refused boot. Leave the - default (``False``) at this call site so a data conflict is logged and - reported via the return value instead, letting the batch survive and - the next flush retry once the graph has been repaired. + *fail_on_data_conflict* (see below). + + Deploy-safe boot (council amendment, 2026-08-12): cold start (the + lifespan handler, ``main.py``) no longer fails closed on graph *data* + state -- a deploy must never crash-loop against an un-migrated, + unreachable, or degraded graph. Cold start now calls this with the + DEFAULT ``fail_on_data_conflict=False``, same as the mid-flight flush + path, and instead surfaces the result as a tri-state ``schema_health`` + signal on ``GET /status`` (see ``main.py``'s lifespan and + ``_record_schema_health``). Only ``run_repair`` / ``doctor --fix`` + still opts into ``fail_on_data_conflict=True`` -- a lingering conflict + there, AFTER dedup+backfill, is a genuine repair failure worth failing + loud on. + + - **Cold start / mid-flight flush** (``main.py``'s ``lifespan()``; + ``Neo4jGraphStore._ensure_schema``, called from inside + ``_flush_body``'s try block on every flush until the schema latches): + must NOT raise due to graph *data* state. A ``RuntimeError`` escaping + the flush path propagates through ``_flush_body``'s + ``except ... raise`` and gets counted as a flush failure, which + dead-letters the entire in-flight batch of real activity records -- + data loss, not merely a refused boot; escaping the lifespan would + crash-loop the deploy. Leave the default (``False``) at both call + sites so a data conflict is logged and reported via the return value + instead. + - **``run_repair`` / ``doctor --fix``**: passes + ``fail_on_data_conflict=True``. Nothing has been written that could be + lost, dedup+backfill have already run, and a genuine post-repair + conflict is a real repair failure the operator needs to see. Args: driver: An ``AsyncDriver`` instance created via @@ -619,12 +635,12 @@ async def ensure_neo4j_schema( workspace: Reserved for future workspace-scoped schema; currently unused. fail_on_data_conflict: When True, a genuine data conflict on the ``:Node`` uniqueness constraint (Step 6) raises - ``RuntimeError`` (fail-closed; correct for cold start). + ``RuntimeError`` (fail-closed; correct for + ``run_repair``/``doctor --fix`` only -- see above). When False (default), the same conflict is logged as a - WARNING pointing at ``doctor --fix`` and this function - returns ``False`` instead of raising (fail-open; correct - for the flush path, where raising would dead-letter real - events -- see caller-specific guidance above). + WARNING/ERROR pointing at ``doctor --fix`` and this + function returns ``False`` instead of raising (fail-open; + correct for cold start and the flush path). Returns: ``True`` iff the schema is **fully established** — every index and @@ -632,10 +648,13 @@ async def ensure_neo4j_schema( ``False`` if any index/constraint could not be created -- e.g. Neo4j was unreachable and the connectivity error was swallowed, or (when *fail_on_data_conflict* is False) the ``:Node`` constraint hit a data - conflict -- to avoid dead-lettering real events. Callers use this to - decide whether to retry schema init on a later flush rather than - latching a half-built schema and leaving the uniqueness constraint - permanently absent. + conflict even after a drop-and-retry (see Step 6 comments below) -- + in the latter case a fallback ``idx_node_universal`` index is created + so the write path keeps a ``NodeIndexSeek`` instead of an + ``AllNodesScan`` (atomicity only is lost, never the seek). Callers + use this to decide whether to retry schema init on a later flush + rather than latching a half-built schema and leaving the uniqueness + constraint permanently absent. Raises: RuntimeError: if *fail_on_data_conflict* is True and the ``:Node`` @@ -874,35 +893,279 @@ async def _create_constraint( # see the function docstring) — see _is_constraint_data_conflict and # _create_constraint. # - # A uniqueness constraint carries its OWN backing range index, and Neo4j - # refuses to create it while a standalone index on the same (label, - # properties) exists ("a constraint cannot be created until the index has - # been dropped"). #19 shipped a plain `idx_node_universal` index on - # :Node(node_id, workspace); drop it first (IF EXISTS, idempotent) so the - # constraint can take over the seek role. + # Council amendment B2 (deploy-safe boot, 2026-08-12): a uniqueness + # constraint carries its OWN backing range index, and Neo4j refuses to + # create it while a standalone index on the same (label, properties) + # exists ("a constraint cannot be created until the index has been + # dropped" -- confirmed live as Neo.ClientError.Schema.IndexAlreadyExists). + # The PRE-amendment code unconditionally dropped `idx_node_universal` + # BEFORE attempting the constraint every boot -- if the constraint then + # failed on a genuine data conflict, boot proceeded with NO index at + # all, regressing the AllNodesScan stall PR #67 removed from the write + # path. Fixed ordering: attempt the constraint FIRST; only drop the + # standalone index after a successful create (zero risk, zero added + # cost on the common healthy-graph path, where no such index exists to + # begin with -- CREATE CONSTRAINT IF NOT EXISTS is then a no-op). If + # the FIRST attempt fails, retry once after dropping + # `idx_node_universal` -- this handles the case where the failure is + # merely a leftover/fallback standalone index blocking the constraint + # (IndexAlreadyExists) rather than a genuine data conflict; without the + # retry, a degraded boot's own remediation (the fallback index created + # below) would permanently lock the graph out of ever re-establishing + # the constraint, even after `doctor --fix` cleans the underlying + # data. If the constraint is still not established after the retry, + # (re-)create `idx_node_universal` as an explicit fallback so degraded + # mode costs atomicity only, never the index seek (B4: loud -- + # ERROR-logged -- never a silent degradation). # ------------------------------------------------------------------ - try: - await session.run("DROP INDEX idx_node_universal IF EXISTS") - except (Neo4jError, DriverError) as exc: # pragma: no cover - tolerate - _LOG.debug( - "ensure_neo4j_schema: DROP INDEX idx_node_universal skipped " - "(benign): %s", - exc, - ) - fully_established = ( - await _create_constraint( + _NODE_CONSTRAINT_CYPHER = ( + "CREATE CONSTRAINT node_node_id_workspace_unique IF NOT EXISTS " + "FOR (n:Node) REQUIRE (n.node_id, n.workspace) IS UNIQUE" + ) + + async def _try_node_constraint() -> bool: + return await _create_constraint( session, "Node", - "CREATE CONSTRAINT node_node_id_workspace_unique IF NOT EXISTS " - "FOR (n:Node) REQUIRE (n.node_id, n.workspace) IS UNIQUE", + _NODE_CONSTRAINT_CYPHER, fail_on_data_conflict=fail_on_data_conflict, ) - and fully_established - ) + + node_constraint_established = await _try_node_constraint() + + if not node_constraint_established: + # Only retry when a standalone idx_node_universal is CONFIRMED + # present (a cheap, read-only check) -- that specific index is + # the one condition that guarantees a genuine-data-conflict-free + # retry could succeed (Neo4j refuses CREATE CONSTRAINT while a + # standalone index covers the same properties, independent of + # whether the underlying data is otherwise clean). If no such + # index exists, the failure can only be a genuine data conflict + # (or a connectivity/benign race already handled by + # _create_constraint above) -- retrying would just repeat the + # same failure, so skip it and go straight to the fallback below. + try: + _idx_present = await _run_single_count( + session, + "SHOW INDEXES YIELD name " + "WHERE name = 'idx_node_universal' " + "RETURN count(*) AS c", + ) + except (Neo4jError, DriverError) as exc: # pragma: no cover + _LOG.debug( + "ensure_neo4j_schema: idx_node_universal existence check " + "skipped (benign): %s", + exc, + ) + _idx_present = 0 + + if _idx_present: + try: + await session.run("DROP INDEX idx_node_universal IF EXISTS") + except (Neo4jError, DriverError) as exc: # pragma: no cover + _LOG.debug( + "ensure_neo4j_schema: pre-retry DROP INDEX " + "idx_node_universal skipped (benign): %s", + exc, + ) + else: + node_constraint_established = await _try_node_constraint() + + if node_constraint_established: + # Constraint carries its own backing index; a standalone + # idx_node_universal (legacy, or a prior degraded boot's + # fallback) is now redundant. Drop it ONLY after the constraint + # is confirmed established -- never before -- so a degraded boot + # is never left with NEITHER the constraint NOR a fallback index. + try: + await session.run("DROP INDEX idx_node_universal IF EXISTS") + except (Neo4jError, DriverError) as exc: # pragma: no cover + _LOG.debug( + "ensure_neo4j_schema: post-success DROP INDEX " + "idx_node_universal skipped (benign): %s", + exc, + ) + else: + # Still degraded: genuine data conflict (or connectivity issue) + # survived the retry. Loud, not silent (B4) -- create/keep the + # fallback index so the write-path MERGE still gets a + # NodeIndexSeek; only atomicity is lost, never the seek. + _LOG.error( + "ensure_neo4j_schema: :Node uniqueness constraint NOT " + "established -- creating fallback idx_node_universal so " + "writes keep an index seek (atomicity only is lost; " + "concurrent-duplicate risk is a RATCHET, not bounded, while " + "degraded). Run doctor --fix once the graph is reachable." + ) + await _create_index( + "CREATE INDEX idx_node_universal IF NOT EXISTS " + "FOR (n:Node) ON (n.node_id, n.workspace)" + ) + + fully_established = node_constraint_established and fully_established + + # NOTE: the :SchemaMeta baseline singleton (§10.2 of the cursor- + # durability spec) is deliberately NOT created here. This function + # runs on THREE paths with very different frequency/concurrency + # characteristics: the lifespan startup handler (once), EVERY + # Neo4jGraphStore's first flush via ``_ensure_schema`` (one store per + # SessionWorker -- i.e. once per worker, concurrently, on every cold + # start), and ``run_repair``/``doctor --fix``. A SchemaMeta write + # belongs on the startup-only, single-writer path -- see + # ``ensure_schema_version_baseline`` below, called exactly once from + # ``main.py``'s lifespan handler AFTER this function establishes the + # rest of the schema. return fully_established +async def ensure_schema_version_baseline( + driver: Any, + *, + database: str = "neo4j", +) -> None: + """Create the :SchemaMeta uniqueness constraint and baseline singleton. + + BASELINE ONLY (§10.2 of the cursor-durability spec) — create-if-absent, + no comparison/migration. Call this **exactly once, from the lifespan + startup handler**, AFTER ``ensure_neo4j_schema`` has established the rest + of the schema (indexes + uniqueness constraints). + + Why this is its own function, not part of ``ensure_neo4j_schema``: + ``ensure_neo4j_schema`` also runs on every ``Neo4jGraphStore``'s first + flush via ``_ensure_schema`` (one store per SessionWorker — so once per + worker, concurrently, on every cold start) and from + ``run_repair``/``doctor --fix``. A SchemaMeta baseline write is a + single-writer, startup-only concern; living inside ``ensure_neo4j_schema`` + would have it fire redundantly (and concurrently) on every worker's first + flush instead. + + Ordering matters: the uniqueness constraint on ``(:SchemaMeta).id`` is + created FIRST, then the singleton MERGE. This is what makes the + create-if-absent MERGE race-free even under concurrent callers — without + the constraint, two concurrent MERGEs on a fresh database can each pass + the existence check and both create a ``{id: 'singleton'}`` node. + + ``ON CREATE SET`` ONLY: if the node already exists it is left untouched. + Reconciling an existing node's ``schema_version`` against the running + server's value is deliberately deferred "handling" logic for a later + phase — do NOT add an ``ON MATCH SET`` here, and do NOT add a helper + that both reads ``SCHEMA_VERSION`` and mutates this node (that coupling + is exactly what would let comparison/upgrade logic sneak in unreviewed). + The read path (``GET /version``) and this write path stay structurally + separate. + + O(1): a single constraint DDL statement plus a MERGE on a fixed singleton + key. Never a graph scan. + + Non-fatal by design: any ``Neo4jError``/``DriverError`` here (including a + connectivity blip) is logged as a WARNING and swallowed rather than + raised, mirroring the tolerant try/except pattern used throughout + ``ensure_neo4j_schema`` — a transient failure on this passive data point + must never crash server boot. + + Args: + driver: An ``AsyncDriver`` instance created via + ``AsyncGraphDatabase.driver(...)``. + database: Target Neo4j database name (default: ``"neo4j"``). + """ + try: + async with driver.session(database=database) as session: + try: + await session.run( + "CREATE CONSTRAINT schemameta_id_unique IF NOT EXISTS " + "FOR (m:SchemaMeta) REQUIRE m.id IS UNIQUE" + ) + except (Neo4jError, DriverError) as exc: + if isinstance(exc, Neo4jError) and exc.code in _BENIGN_SCHEMA_CODES: + _LOG.debug( + "ensure_schema_version_baseline: SchemaMeta " + "uniqueness constraint already present (benign " + "concurrent-schema race, code=%s)", + exc.code, + ) + else: + _LOG.warning( + "ensure_schema_version_baseline: could not create " + "SchemaMeta uniqueness constraint; continuing " + "without it: %s", + exc, + ) + + await session.run( + "MERGE (m:SchemaMeta {id: 'singleton'}) " + "ON CREATE SET m.schema_version = $schema_version, " + "m.last_updated = $now", + schema_version=SCHEMA_VERSION, + now=datetime.now(UTC).isoformat(), + ) + except (Neo4jError, DriverError) as exc: + # Mirror the tolerance pattern in ensure_neo4j_schema: never let a + # connectivity blip here escape and crash server startup over a + # passive baseline data point. + _LOG.warning( + "ensure_schema_version_baseline: could not write SchemaMeta " + "baseline singleton (connectivity error); continuing without " + "it: %s", + exc, + ) + + +async def read_graph_schema_version( + driver: Any, + *, + database: str = "neo4j", +) -> int | None: + """Read-only: the STORED ``:SchemaMeta{id:'singleton'}.schema_version``. + + W-4 -- ADVISORY DRIFT SIGNAL ONLY, NOT A GUARD. This is a separate, + read-only companion to ``ensure_schema_version_baseline`` above, kept + structurally apart from that write path on purpose (see its docstring's + "do NOT add a helper that both reads SCHEMA_VERSION and mutates this + node" warning). This function does not import or compare against + ``SCHEMA_VERSION`` at all -- it only reads back whatever is stored. + Callers (``GET /status``) may compare the returned value against + ``status.SCHEMA_VERSION`` themselves for advisory telemetry so a + server/graph mismatch is *detectable*; this function performs no + comparison, gating, or migration of any kind. Full mismatch + handling/migration is deliberately deferred (tracked separately). + + Returns ``None`` when the singleton is absent (bootstrap: startup has + never completed ``ensure_schema_version_baseline`` against this graph) + or when the read itself fails for any reason (treated as "unknown", not + as an error) -- mirrors the never-raise, never-500-``/status`` contract + of ``_check_driver_connected`` in ``main.py``. + + O(1): a point lookup by the unique ``id`` key, never a graph scan. Reads + via ``async for`` (rather than ``.single()``), the same reason + ``_run_single_count`` above does: this works against both the real async + driver and the test-suite's mock session, which only implements async + iteration. + + Args: + driver: An ``AsyncDriver`` instance created via + ``AsyncGraphDatabase.driver(...)``. + database: Target Neo4j database name (default: ``"neo4j"``). + """ + try: + async with driver.session(database=database) as session: + result = await session.run( + "MATCH (m:SchemaMeta {id: 'singleton'}) " + "RETURN m.schema_version AS schema_version" + ) + async for record in result: + value = record["schema_version"] + return int(value) if value is not None else None + return None + except Exception as exc: # noqa: BLE001 - defensive: /status must never 500 + _LOG.warning( + "read_graph_schema_version: could not read SchemaMeta singleton " + "(connectivity error?); returning None: %s", + exc, + ) + return None + + def _serialized_row_size(value: Any) -> int: """Return a cheap conservative proxy for the serialized byte size of *value*. @@ -985,12 +1248,24 @@ def _chunk_list( def _build_node_props(data: dict[str, Any], workspace: str) -> dict[str, Any]: """Assemble the sanitized props dict for a single node row. - ``labels`` and ``created_by`` are excluded from the returned dict: + ``labels``, ``created_by``, and ``working_dir`` are excluded from the returned dict: - ``labels`` are applied separately via ``SET n:Label`` statements (not stored as props). - ``created_by`` travels ONLY as the ``$created_by`` query param (never a node property) so it cannot affect node identity or clobber the ``ON CREATE SET n.created_by`` stamp. + - ``working_dir`` (W-3 defect 2) is deliberately excluded from the blind + ``SET n += row.props`` merge -- it travels instead as the row's separate + top-level ``working_dir`` key (see ``_write_batch``'s Session-node write), so a + DB-level ``coalesce(n.working_dir, row.working_dir)`` can be applied instead of a + last-write-wins overwrite. An already-set working_dir must never be clobbered by + a cross-writer/replica race; this was previously enforced ONLY in the Python + layer (services.py's populate-if-missing check), which does not protect against + concurrent writers racing the same node. """ - raw = {k: v for k, v in data.items() if k not in ("labels", "created_by")} + raw = { + k: v + for k, v in data.items() + if k not in ("labels", "created_by", "working_dir") + } _convert_temporal_props(raw) # ISO str -> datetime, in place props = Neo4jGraphStore._sanitize_properties(raw) props["workspace"] = workspace @@ -1032,6 +1307,16 @@ async def _write_batch( row: dict[str, Any] = {"node_id": node_id, "props": props} if "Session" in labels: + # W-3 defect 2: working_dir is carried as a separate top-level row key + # (never merged via the blind `+=`) so the Session write below can apply + # a DB-level `coalesce(n.working_dir, row.working_dir)` instead of an + # overwrite. working_dir is a Session-only property (services.py is the + # sole writer); non-Session rows never carry it, so the generic + # non-Session write path (_NODE_MERGE_CYPHER, other_rows below) is + # deliberately left untouched by this change. + working_dir_value = data.get("working_dir") + if working_dir_value: + row["working_dir"] = working_dir_value session_rows.append(row) else: other_rows.append(row) @@ -1056,12 +1341,38 @@ async def _write_batch( # purely (node_id, workspace) on :Node — every node carries :Node, and the # :Node uniqueness constraint (ensure_neo4j_schema) makes concurrent MERGEs # atomic, the role the :Session constraint used to play for this MERGE. + # + # W-3 defect 2 (DB-level working_dir non-overwrite guarantee): a second, + # separate SET clause is appended AFTER the main `SET n += row.props, n:Session` + # — `SET n.working_dir = coalesce(n.working_dir, row.working_dir)`. This is a + # minimal, targeted addition, not a rewrite: + # - The MERGE identity clause (the node lookup) is byte-for-byte unchanged, + # so the query plan for finding/creating `n` is unaffected. + # - `row.working_dir` is a plain top-level UNWIND row key (not part of + # row.props), so it never participates in the `+=` merge and cannot be + # blindly overwritten by it. Rows without a working_dir simply omit the + # key, and Cypher map access on a missing key returns null, so + # `coalesce(n.working_dir, null)` is a no-op for every non-working_dir row. + # - When `n.working_dir` is already set, coalesce keeps the EXISTING value + # (the guarantee: an already-populated working_dir is never clobbered by a + # concurrent/replica writer). When `n.working_dir` is null and the row + # supplies one, coalesce fills the gap — the same populate-if-missing rule + # services.py already enforces in Python, now also guaranteed at the DB + # level (defense in depth against races the Python-layer check cannot see). + # - Sequential SET clauses within one statement guarantee ordering: the first + # SET fully applies (including any legitimate working_dir-adjacent props) + # before the second SET reads n.working_dir, so there is no read-before- + # write ambiguity within the single MERGE lock hold. + # This is scoped to Session nodes only (working_dir is a Session-only property; + # services.py is its sole writer) — the generic non-Session write path + # (_NODE_MERGE_CYPHER below) is deliberately left untouched. res = await tx.run( "UNWIND $rows AS row " f"MERGE (n:{_UNIVERSAL_NODE_LABEL} " "{node_id: row.node_id, workspace: row.props.workspace}) " "ON CREATE SET n.created_by = $created_by " - "SET n += row.props, n:Session", + "SET n += row.props, n:Session " + "SET n.working_dir = coalesce(n.working_dir, row.working_dir)", rows=session_rows, created_by=created_by, ) diff --git a/context_intelligence_server/queue_manager.py b/context_intelligence_server/queue_manager.py index 6fba186e..c1a2c64e 100644 --- a/context_intelligence_server/queue_manager.py +++ b/context_intelligence_server/queue_manager.py @@ -4,8 +4,20 @@ - ``.log`` — append-only, newline-terminated, opaque ``bytes``. Each line is one enqueued record. The log is never rewritten in place. -- ``.offset`` — a single integer: the byte position in the log - that has been durably processed (committed). A missing offset file means 0. +- ``.offset`` — a single JSON record, written atomically: + ``{"v": 1, "offset": , "cursor": }``. ``offset`` is the + byte position in the log that has been durably processed (committed); + ``cursor`` is an opaque snapshot of cross-handler session state + (``HookStateService.snapshot_cursor()``), persisted so a worker rebuild + (crash restart or stale-session reap) restores its in-memory cursor + instead of resetting it. The cursor is written INSIDE the same atomic + ``os.replace`` as the offset — never as a separate file — so the two can + never skew relative to each other: a crash mid-write loses both together, + and a successful write always carries a cursor consistent with its offset. + Legacy files whose content is a bare integer (pre-upgrade shape) are still + accepted on read and yield ``cursor = None``; the next commit rewrites the + file in the new JSON form. A missing offset file means offset 0, cursor + ``None``. - ``.dead.jsonl`` — append-only dead-letter records for batches that could not be processed after exhausting retries. @@ -13,7 +25,8 @@ Appends use a plain durable ``write()``. This gives PROCESS-crash durability (the bytes are handed to the OS page cache and survive a process crash). POWER-LOSS durability via ``fsync`` is deliberately - deferred to Phase B3 (fsync group-commit). + deferred to Phase B3 (fsync group-commit). This applies equally to the + offset+cursor record: no ``fsync`` is issued on that write either. session_id contract: Every public method validates ``session_id`` and raises ``ValueError`` if @@ -88,13 +101,66 @@ def _offset_path(self, session_id: str) -> Path: def _dead_path(self, session_id: str) -> Path: return self._dir / f"{session_id}.dead.jsonl" - def _read_committed_offset(self, session_id: str) -> int: + def _write_offset_record( + self, key: str, offset: int, cursor: dict[str, Any] | None + ) -> None: + """Sole writer of the ``.offset`` file — a single atomic JSON record. + + Writes ``{"v": 1, "offset": offset, "cursor": cursor}`` to a temp file + and ``os.replace``s it into place, so a reader never observes a torn + or partial record. Folding the cursor into the same record as the + offset (rather than a sidecar file) is what guarantees the two can + never skew: a crash loses both together, never one without the other. + No ``fsync`` — same process-crash-durable, not-power-durable contract + as the rest of this module. + """ + final = self._offset_path(key) + tmp = self._dir / f"{key}.offset.tmp" + record = {"v": 1, "offset": offset, "cursor": cursor} + tmp.write_text(json.dumps(record, separators=(",", ":")), encoding="utf-8") + os.replace(tmp, final) + + def _read_offset_record(self, key: str) -> tuple[int, dict[str, Any] | None]: + """Read the ``.offset`` file, returning ``(offset, cursor)``. + + Accepts both the current JSON record shape and the legacy bare-integer + shape (pre-upgrade). A missing or empty file yields ``(0, None)``. + + Cursor unreadability degrades to ``None`` (D5: never crash boot over a + corrupt/unknown cursor) — an unknown ``v`` or non-dict ``cursor`` + silently discards the cursor while still honoring the offset. Offset + unreadability is NOT degraded: a malformed record (bad JSON, or an + ``offset`` that isn't an int) raises ``ValueError`` loudly, exactly as + the legacy ``int(text)`` parse did — degrading a corrupt offset to 0 + would replay the entire log and manufacture duplicate nodes, which is + a worse and quieter failure than a loud boot error. + """ try: - text = self._offset_path(session_id).read_text("utf-8") + text = self._offset_path(key).read_text("utf-8") except FileNotFoundError: - return 0 + return 0, None text = text.strip() - return int(text) if text else 0 + if not text: + return 0, None + if text.startswith("{"): + try: + rec = json.loads(text) + except json.JSONDecodeError as exc: + raise ValueError(f"Malformed offset record for {key!r}: {exc}") from exc + offset = rec.get("offset") + if not isinstance(offset, int): + raise ValueError( + f"Malformed offset record for {key!r}: offset is not an int" + ) + cursor = rec.get("cursor") + if rec.get("v") == 1 and isinstance(cursor, dict): + return offset, cursor + return offset, None + # Legacy bare-integer shape. + return int(text), None + + def _read_committed_offset(self, session_id: str) -> int: + return self._read_offset_record(session_id)[0] def _complete_data_end(self, session_id: str) -> int: """Byte position after the last complete (newline-terminated) line. @@ -217,23 +283,29 @@ def _read() -> Batch: return await asyncio.to_thread(_read) - async def commit(self, session_id: str, new_offset: int) -> None: - """Atomically and durably persist ``new_offset`` (the ack). - - Writes the offset to a temp file and uses ``os.replace`` for an atomic - rename, so a reader never observes a torn or partial offset file. No - ``fsync`` is issued here: this gives process-crash durability, while - power-loss durability is deferred to Phase B3 (fsync group-commit). + async def commit( + self, + session_id: str, + new_offset: int, + cursor: dict[str, Any] | None, + ) -> None: + """Atomically and durably persist ``new_offset`` (the ack) + ``cursor``. + + ``cursor`` has NO default: every call site must spell it explicitly + (a deliberate footgun-avoidance choice — see spec §10.4) so it is + never accidentally omitted at a commit site. + + Writes the offset+cursor record to a temp file and uses ``os.replace`` + for an atomic rename, so a reader never observes a torn or partial + offset file. No ``fsync`` is issued here: this gives process-crash + durability, while power-loss durability is deferred to Phase B3 (fsync + group-commit). Folding the cursor into this same atomic write (D1) is + what makes offset and cursor always advance together — never skewed. """ self._validate_session_id(session_id) - final = self._offset_path(session_id) - tmp = self._dir / f"{session_id}.offset.tmp" - - def _commit() -> None: - tmp.write_text(str(new_offset), encoding="utf-8") - os.replace(tmp, final) - - await asyncio.to_thread(_commit) + await asyncio.to_thread( + self._write_offset_record, session_id, new_offset, cursor + ) async def dead_letter(self, session_id: str, raw: bytes, error: str) -> None: """Append one dead-letter record for an unprocessable batch line. @@ -281,6 +353,16 @@ def _delete() -> None: await asyncio.to_thread(_delete) + async def read_cursor(self, session_id: str) -> dict[str, Any] | None: + """Return the persisted cursor for ``session_id``, or ``None``. + + ``None`` covers: no offset file, a legacy bare-integer offset file, or + a JSON record whose cursor is absent/unreadable (D5 safe-degrade — + see ``_read_offset_record``). + """ + self._validate_session_id(session_id) + return await asyncio.to_thread(lambda: self._read_offset_record(session_id)[1]) + async def read_dead_letters(self, session_id: str) -> list[dict]: """Return all dead-letter records for ``session_id`` in append order. @@ -297,6 +379,30 @@ def _read() -> list[dict]: return await asyncio.to_thread(_read) + async def is_fully_drained(self, session_id: str) -> bool: + """Return True iff *session_id* has no undrained (unprocessed) log data. + + Durable-state predicate (B3 of the blob-reclaim design, `docs/plans/ + 2026-08-12-blob-reclaim-endpoint-spec.md`): mirrors the per-session + check inside :meth:`recover` (committed offset >= complete-data end + means fully drained). Derived purely from the ``.log``/``.offset`` + files on disk, so it is independent of in-memory worker liveness and + survives a `kill -9` + restart -- a crashed process's undrained queue + still reads as NOT drained here even though no worker is registered + for it. + + A session with no ``.log`` file at all reads as fully drained + (``0 >= 0``), which is correct: it never had any queued data. + """ + self._validate_session_id(session_id) + + def _check() -> bool: + return self._read_committed_offset(session_id) >= self._complete_data_end( + session_id + ) + + return await asyncio.to_thread(_check) + async def active_sessions(self) -> list[str]: """Return sorted session_ids with undrained data. @@ -707,7 +813,7 @@ def _reconcile() -> int: log_path = self._log_path(key) if not log_path.exists(): continue - committed = self._read_committed_offset(key) + committed, cursor = self._read_offset_record(key) complete_end = self._complete_data_end(key) pos = committed with open(log_path, "rb") as f: @@ -721,10 +827,12 @@ def _reconcile() -> int: pos += len(raw) total_skipped += 1 if pos > committed: - final = self._offset_path(key) - tmp = self._dir / f"{key}.offset.tmp" - tmp.write_text(str(pos), encoding="utf-8") - os.replace(tmp, final) + # Lines skipped here were dead-lettered and therefore never + # dispatched to handlers, so the cursor is unchanged and + # MUST be carried through unmodified (spec §5.1, R2): + # dropping it here would silently wipe cross-handler state + # on the recovery path. + self._write_offset_record(key, pos, cursor) self._stats_cache = None return total_skipped diff --git a/context_intelligence_server/registry.py b/context_intelligence_server/registry.py index ad078fee..febd7a0b 100644 --- a/context_intelligence_server/registry.py +++ b/context_intelligence_server/registry.py @@ -11,6 +11,7 @@ from context_intelligence_server.blob_store import AsyncDiskBlobStore from context_intelligence_server.config import get_settings +from context_intelligence_server.maintenance import coordinator from context_intelligence_server.status import EventRecord, ring_buffer from context_intelligence_server.neo4j_store import Neo4jGraphStore from context_intelligence_server.pipeline import process_event, setup_handlers @@ -21,6 +22,17 @@ _DRAIN_MAX_BATCH = 100 _DRAIN_POLL_INTERVAL = 0.05 # idle poll cadence; bounded by flush_timeout +# Poll cadence while the maintenance gate is closed. Slower than the normal +# drain cadence on purpose: nothing can be done while gated, so a 20x slower +# spin costs <=1s of resume latency and avoids N-session busy polling. +_GATED_POLL_INTERVAL = 1.0 + +# R3 (spec §10.4): pending_tool_block_ids is unbounded and now serialized on +# every commit (§3 D2). We measure, we do not cap -- capping at serialization +# would silently drop E09 edges. Only log once the dict is non-trivially +# sized, so a healthy session (a handful of in-flight tool blocks) never +# spams a commit-cadence (as frequent as every ~100 events) log line. +_PENDING_TOOL_BLOCK_LOG_THRESHOLD = 50 # A positive residual must PERSIST this long before it is called degraded. # Must exceed the worst-case transient-skew window: the derive_all_stats @@ -283,6 +295,23 @@ async def _process_one( ) ) + def _log_pending_tool_block_size(self, worker: SessionWorker) -> None: + """R3 measurement (spec §10.4): observe ``pending_tool_block_ids`` + growth at the commit path -- the dict is populated at + content_block.py and only drained by a matching tool_call:start pop, + so unmatched blocks accumulate for the life of the session and are + now serialized on every commit. Measure, do not cap: capping at + serialization would silently drop E09 CAUSED edges. + """ + size = len(worker.services.data_layer_2.pending_tool_block_ids) + if size > _PENDING_TOOL_BLOCK_LOG_THRESHOLD: + logger.info( + "pending_tool_block_ids_size session=%s size=%d", + worker.session_id, + size, + extra={"session_id": worker.session_id}, + ) + async def _flush_barrier(self, worker: SessionWorker) -> None: """The ONE Neo4j-write boundary: a semaphore-gated, awaited flush. @@ -320,12 +349,44 @@ async def drain_worker( handlers = setup_handlers(worker.services) qm = self.queue_manager session_id = worker.session_id + # I5b: restore the cross-handler cursor persisted alongside the + # committed offset. Covers BOTH worker-rebuild triggers through this + # single entry point: a crash-restart recovery (main.py) and a + # stale-session reap rebuild (this method, below) both spawn the new + # worker through get_or_create -> start_drain -> drain_worker. + restored_cursor = await qm.read_cursor(session_id) + worker.services.restore_cursor(restored_cursor) + if restored_cursor: + logger.info( + "cursor_restored session=%s", + session_id, + extra={"session_id": session_id}, + ) poll_interval = min(flush_timeout, _DRAIN_POLL_INTERVAL) idle_elapsed = 0.0 attempts = 0 + # Pre-batch cursor snapshot for the common (budget-not-exhausted) + # retry branch below -- see the snapshot/restore comment at the + # retry `continue` for why this is needed. + pre_batch_cursor: dict[str, Any] | None = None while True: try: + # Maintenance gate -- FIRST statement inside the loop, before + # any consuming step (read_batch/process/flush/commit). This + # placement (rather than gating spawn at get_or_create -> + # start_drain) is what makes the offset-ordering guarantee + # free: everything below is unreachable while gated, so + # qm.commit() never runs and the on-disk offset never + # advances. Kill-9 mid-maintenance -> restart -> repair -> + # replay from the last successfully-flushed offset, exactly + # once. Drainers still spawn during maintenance; they idle + # here instead of refusing to spawn (which would itself be a + # second latch -- see the WS-3a spec sec 3a-2). + if await coordinator.gate_closed(): + await asyncio.sleep(_GATED_POLL_INTERVAL) + continue + batch = await qm.read_batch(session_id, max_items=_DRAIN_MAX_BATCH) if not batch.lines: @@ -352,6 +413,15 @@ async def drain_worker( idle_elapsed = 0.0 # --- dispatch + durable write barrier, one error path --- + if attempts == 0: + # First attempt at this (start_offset-identified) batch: + # snapshot the cursor BEFORE any handler mutates it, so a + # failed attempt can restore to this exact pre-batch + # state before replay (see the retry-branch comment + # below). Not re-taken on subsequent attempts of the + # SAME batch (attempts > 0) so a 3rd attempt still rolls + # back to the ORIGINAL pre-batch state, not attempt 2's. + pre_batch_cursor = worker.services.snapshot_cursor() try: saw_terminal = await self._process_batch(worker, batch, handlers) await self._flush_barrier(worker) @@ -399,11 +469,28 @@ async def drain_worker( # idempotent MERGE makes the replay a no-op). The backoff # avoids a tight Neo4j-hammering retry loop on a transient # deadlock and keeps retries on the loop's poll cadence. + # + # Phantom-cursor guard (common-path sibling of the + # _handle_exhausted_batch guard above): handlers mutate + # cross-handler cursor state (e.g. IterationHandler's + # iteration_count / active_iteration_id) BEFORE their + # graph write. The MERGE being idempotent does NOT make + # replay a no-op when a handler's node_id is derived from + # a mutable cursor counter -- each replay would advance + # that counter again and mint a NEW node_id for the same + # never-committed batch. Restore the pre-batch snapshot + # before the replay so every attempt starts from + # identical cursor state; only the attempt that actually + # commits leaves its mutation in place. + worker.services.restore_cursor(pre_batch_cursor) await asyncio.sleep(poll_interval) continue attempts = 0 - await qm.commit(session_id, batch.end_offset) + self._log_pending_tool_block_size(worker) + await qm.commit( + session_id, batch.end_offset, worker.services.snapshot_cursor() + ) self.record_written(len(batch.lines)) logger.debug( "batch_committed events=%d offset=%d", @@ -453,6 +540,15 @@ async def _handle_exhausted_batch( good lines flush normally. Every line advances the offset past itself (commit), so the whole batch is accounted for. No silent loss, no binary shrink, no cross-line contamination. + + BLOCKER-1 (phantom-cursor guard, R6xD1): handlers mutate cross-handler + cursor state (e.g. IterationHandler sets active_iteration_id / + iteration_count) BEFORE their graph write. If that write is the one + that fails, discard_buffer() throws away the write but NOT the + already-applied in-memory mutation. A pre-line cursor snapshot is + taken and, when the line ends up dead-lettered, restored — so the + commit below can never persist a cursor pointing at a node that was + never written. A line that succeeds keeps its mutation untouched. """ qm = self.queue_manager session_id = worker.session_id @@ -464,6 +560,7 @@ async def _handle_exhausted_batch( offset = batch.start_offset for raw in batch.lines: line_end = offset + len(raw) + 1 # +1 for the newline read_batch strips + pre_line_cursor = worker.services.snapshot_cursor() try: event, _ws, data = self._parse_line(raw) await self._process_one(worker, event, data, handlers) @@ -481,7 +578,13 @@ async def _handle_exhausted_batch( # it cannot contaminate the NEXT line's flush. A successful flush # clears the buffer itself; only the failure path needs this. worker.services.graph.discard_buffer() - await qm.commit(session_id, line_end) + # BLOCKER-1: roll the cursor back to its pre-line state so this + # line's in-memory mutation (already applied by the handler + # before the write above failed) does not enter the committed + # snapshot below — its graph write was just discarded. + worker.services.restore_cursor(pre_line_cursor) + self._log_pending_tool_block_size(worker) + await qm.commit(session_id, line_end, worker.services.snapshot_cursor()) offset = line_end async def _finalize_session(self, worker: SessionWorker, handlers: Any) -> None: @@ -501,7 +604,10 @@ async def _finalize_session(self, worker: SessionWorker, handlers: Any) -> None: except Exception: logger.exception("finalize_tail_flush_failed session=%s", session_id) return # NOT finalized: keep worker alive, leave tail uncommitted - await qm.commit(session_id, tail.end_offset) + self._log_pending_tool_block_size(worker) + await qm.commit( + session_id, tail.end_offset, worker.services.snapshot_cursor() + ) self.record_written(len(tail.lines)) logger.debug( "batch_committed events=%d offset=%d", diff --git a/context_intelligence_server/routers/admin.py b/context_intelligence_server/routers/admin.py index 4606a303..d06ae101 100644 --- a/context_intelligence_server/routers/admin.py +++ b/context_intelligence_server/routers/admin.py @@ -33,14 +33,28 @@ from __future__ import annotations +import asyncio +import json import logging +import os import re +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Any from fastapi import APIRouter, Depends, HTTPException, Request -from pydantic import BaseModel, field_validator +from fastapi.responses import JSONResponse +from neo4j import READ_ACCESS, WRITE_ACCESS +from pydantic import BaseModel, Field, field_validator -from context_intelligence_server.config import _ALL_ZEROS_GUID, _GUID_RE +from context_intelligence_server.blob_processor import ( + BLOB_REF_CARRIER_PROPERTIES as _BLOB_REF_CARRIER_PROPERTIES, +) +from context_intelligence_server.config import _ALL_ZEROS_GUID, _GUID_RE, get_settings from context_intelligence_server.identity_store import IdentityStore +from context_intelligence_server.maintenance import coordinator +from context_intelligence_server.maintenance_ops import run_maintenance_operation # --------------------------------------------------------------------------- # Validation constants @@ -53,6 +67,21 @@ # (non-empty, non-whitespace, sane upper bound for an identifier string). _MAX_CONTRIBUTOR_LEN = 256 +# --------------------------------------------------------------------------- +# Blob-reclaim constants (see docs/plans/2026-08-12-blob-reclaim-endpoint-spec.md, +# "Council amendment -- AUTHORITATIVE" section for the governing design) +# --------------------------------------------------------------------------- + +# B3 (council amendment): hard mtime-floor safety net, defense-in-depth behind +# the durable undrained-queue gate. min_age_minutes below this is rejected +# (422) rather than silently raised -- a caller passing 0 must not be able to +# disable the age gate entirely. +_MIN_AGE_FLOOR_MINUTES = 15 + +# "sample" is bounded to keep the response small; totals (orphans_found, +# reclaimable_bytes) remain authoritative even when the sample is truncated. +_MAX_SAMPLE = 50 + # --------------------------------------------------------------------------- # Module-level audit logger # --------------------------------------------------------------------------- @@ -309,6 +338,360 @@ def _audit_delete(request: Request, *, target: str) -> None: ) +def _audit_blob_reclaim_delete(request: Request, *, uri: str) -> None: + """Emit one structured audit log line per successfully-deleted blob. + + NEVER logs blob contents -- only the ``ci-blob://`` URI is recorded. + """ + logger.info( + "admin.audit action=blob_reclaim target=%s who=%s", + uri, + _admin_who(request), + ) + + +# --------------------------------------------------------------------------- +# Blob reclaim -- orphaned-blob GC (design: docs/plans/2026-08-12-blob- +# reclaim-endpoint-spec.md, "Council amendment -- AUTHORITATIVE" section) +# --------------------------------------------------------------------------- + + +def _access_mode_const(mode: str) -> str: + """Map the configured query access-mode string ("READ"/"WRITE") to the + driver's access-mode constant. + + Deliberately duplicated (not imported) from ``main._neo4j_access_const``: + importing from ``main`` here would create a circular import (``main`` + already imports ``routers.admin`` at module load time). Two lines of + duplication is cheaper than that coupling. + """ + return READ_ACCESS if mode == "READ" else WRITE_ACCESS + + +@dataclass(frozen=True) +class _OnDiskBlob: + """One blob file discovered on disk during the reclaim scan.""" + + session_id: str + key: str + uri: str + path: Path + mtime: float + size: int + + +def _scan_disk_blobs(blob_root: Path) -> list[_OnDiskBlob]: + """Enumerate on-disk blobs under *blob_root* (step 1 of the design). + + Walks ``/*/blobs/*.json`` -- the exact layout + ``AsyncDiskBlobStore`` writes to (``blob_store.py``). ``*.tmp`` residue + from an in-progress write (``tempfile.mkstemp(..., suffix=".tmp")``) is + skipped defensively, though the ``*.json`` glob already excludes it. + A file that vanishes between the glob listing and ``stat()`` (e.g. a + concurrent delete) is silently skipped rather than raising. + """ + blobs: list[_OnDiskBlob] = [] + for p in blob_root.glob("*/blobs/*.json"): + if p.name.endswith(".tmp"): + continue + session_id = p.parent.parent.name + key = p.stem + try: + st = p.stat() + except FileNotFoundError: + continue + blobs.append( + _OnDiskBlob( + session_id=session_id, + key=key, + uri=f"ci-blob://{session_id}/{key}", + path=p, + mtime=st.st_mtime, + size=st.st_size, + ) + ) + return blobs + + +def _collect_blob_refs(obj: Any, out: set[str]) -> None: + """Recursively walk a decoded JSON value collecting ``$blob_ref`` URIs. + + B2 (council amendment, highest severity): this is STRUCTURAL extraction + over the parsed object, never a regex over the serialized string. A + regex anchored on ``ci-blob://`` truncates at the first unescaped + special character (e.g. a literal ``"`` in a session_id, which + ``queue_manager._validate_session_id`` explicitly permits -- it only + rejects ``/ \\ \\0``), silently misclassifying a genuinely-referenced + blob as orphan. ``json.loads`` has already resolved all escaping by the + time this function runs, so any character in a URI (quotes, non-ASCII) + is handled correctly -- there is no APOC path and no regex path. + """ + if isinstance(obj, dict): + ref = obj.get("$blob_ref") + if isinstance(ref, str): + out.add(ref) + for v in obj.values(): + _collect_blob_refs(v, out) + elif isinstance(obj, list): + for item in obj: + _collect_blob_refs(item, out) + + +# Reference-scan hardening (docs/plans/2026-08-12-blob-reclaim-reference-scan- +# hardening.md): the known node properties that can carry a "ci-blob://" +# token anywhere in the graph, keyed to the property name Cypher matches on. +# +# ASSUMPTION -- make it explicit + greppable: the reference scan is scoped to +# EXACTLY these properties, not an all-property-all-node walk, for +# performance (a per-property UNION lets Neo4j evaluate ONE property per row +# instead of toString()-ing every key of every node -- this codebase has +# scar tissue from a 1.3M-node AllNodesScan stall). Adding a NEW ci-blob +# carrier property in the future (a new field-lifter, a new enricher +# property, etc.) REQUIRES adding it to this tuple -- otherwise the reclaim +# GC will not see refs stored there and could misclassify a live blob as an +# orphan. Covers every carrier in the hardening doc's table: +# data -- *Event.data (all 14 event types) +# tool_input -- ToolPreEvent/ToolPostEvent.tool_input (L1), ToolCall.tool_input (L2) +# prompt -- PromptSubmitEvent/PromptCompleteEvent.prompt (L1), Prompt.prompt (L2) +# response -- OrchestratorRun.response (L2) +# NOTE: the canonical allowlist lives in blob_processor as BLOB_REF_CARRIER_PROPERTIES +# and is imported (aliased to _BLOB_REF_CARRIER_PROPERTIES) at the top of this module -- +# a single source of truth shared by the mint site (blob_processor) and this reclaim +# scan. Do NOT re-declare it here. + +# Fallback extraction for carrier values that are NOT valid JSON (a bare +# string property, e.g. a plain-string tool_input/prompt that itself +# contains a ci-blob:// URI rather than the {"$blob_ref": "..."} wrapper). +# Safe here -- unlike a regex over a JSON-*serialized* string (see the B2 +# docstring above) -- because these values are already fully-decoded Neo4j +# property strings with no JSON escaping left to trip over. +_BARE_BLOB_URI_RE = re.compile(r'ci-blob://[^"\s]+') + + +def _extract_blob_refs_from_value(val: str, out: set[str]) -> None: + """Extract every ``ci-blob://`` URI referenced by one carrier property value. + + Two extraction paths, unioned into *out*: + + 1. **Structural (preferred):** ``json.loads(val)`` then recurse with + :func:`_collect_blob_refs`. Handles the ``{"$blob_ref": "..."}`` + JSON-string carriers -- ``neo4j_store._sanitize_properties`` + JSON-serializes any dict/list property value on write, so + ``Event.data``, ``ToolCall.tool_input``, ``Prompt.prompt``, and + ``OrchestratorRun.response`` all round-trip through this path when + they hold a dict/list. + 2. **Regex fallback (only on JSON-parse failure):** a plain-string + carrier is written through VERBATIM (``_sanitize_properties`` only + JSON-serializes dict/list values -- a bare string is stored as-is), + so it is never valid JSON and always lands here. Extracts every bare + ``ci-blob://[^"\\s]+`` token directly from the decoded string -- + covers a lifted ``*.tool_input``/``*.prompt`` property that is a + plain string mentioning a blob URI. + + A value that is neither valid JSON nor contains a bare token contributes + nothing -- this can only ever fail to positively assert a reference, + never falsely assert one, matching the conservative-skip contract of + :func:`_scan_referenced_uris`. + """ + try: + obj = json.loads(val) + except (TypeError, ValueError): + out.update(_BARE_BLOB_URI_RE.findall(val)) + return + _collect_blob_refs(obj, out) + + +def _carrier_scan_clause(prop: str) -> str: + """Build one ``UNION ALL`` branch of the reclaim reference-scan query + for carrier property *prop*. + + ``data`` is special-cased: ``Event.data`` is always a JSON string + (``DefaultHandler`` writes ``json.dumps(data)`` -- see + ``handlers/data_layer_1/default.py``) and the scan for it is scoped to + ``:Event``, matching this carrier's scope in the pre-hardening scan. Every + other registered carrier is an unrestricted-label match with + ``toString()``, since the property may be lifted onto any node type as a + dict, list, or bare string. + """ + if prop == "data": + return "MATCH (n:Event) WHERE n.data CONTAINS 'ci-blob://' RETURN n.data AS val" + return ( + f"MATCH (n) WHERE n.{prop} IS NOT NULL " + f"AND toString(n.{prop}) CONTAINS 'ci-blob://' " + f"RETURN toString(n.{prop}) AS val" + ) + + +# Generated FROM _BLOB_REF_CARRIER_PROPERTIES (imported from +# blob_processor.BLOB_REF_CARRIER_PROPERTIES) -- not hand-duplicated -- so the +# query text can never drift from the allowlist. See +# tests/test_blob_processor.py for the regression test that locks this +# agreement structurally (extracts every `n.` reference out of the +# built query and diffs it against the allowlist tuple). +_BLOB_REF_SCAN_QUERY = " UNION ALL ".join( + _carrier_scan_clause(prop) for prop in _BLOB_REF_CARRIER_PROPERTIES +) + + +async def _scan_referenced_uris(request: Request) -> set[str]: + """Enumerate every ``ci-blob://`` URI referenced anywhere in the graph. + + Step 2 of the design -- deliberately GLOBAL, never workspace-filtered + (hazard #1): blobs are session_id-scoped while nodes are + (node_id, workspace)-scoped, so a per-workspace scan could delete another + workspace's live data. + + Reference-scan hardening (docs/plans/2026-08-12-blob-reclaim-reference- + scan-hardening.md): the referenced set is computed GRAPH-WIDE over the + known ``ci-blob://`` carrier properties (:data:`_BLOB_REF_CARRIER_PROPERTIES` + -- ``data``, ``tool_input``, ``prompt``, ``response``), not ``Event.data`` + only. This makes the scan correct BY CONSTRUCTION -- a strict superset of + the prior ``Event.data``-only scan -- rather than resting on the + (empirically true today, but unenforced) pipeline-ordering invariant that + ``DefaultHandler`` always persists every ref onto ``Event.data`` before any + field-lifter/enricher can strip or promote it elsewhere. Widening the scan + can only ever *protect* more blobs, never delete more: any URI the old + scan found is still found here (still scanned via the ``data`` branch), + plus any URI that lives ONLY on ``tool_input``/``prompt``/``response`` is + now ALSO found. See + ``tests/neo4j/test_blob_reclaim.py::test_b1_event_data_carries_every_blob_ref`` + for the regression test that continues to pin the pipeline-ordering + invariant (belt-and-suspenders now, not the sole safety net), and the + ``test_*_only_referenced_via_*`` tests alongside it that pin the new + per-property carriers directly. + + Query shape (performance-critical -- see :data:`_BLOB_REF_CARRIER_PROPERTIES`): + a ``UNION ALL`` of four single-property predicates, each touching exactly + ONE property per row (``data`` restricted to ``:Event``, matching the + prior scan's scope for that carrier; the other three unrestricted across + labels since ToolCall/Prompt/OrchestratorRun are ordinary nodes). This + avoids a pathological ``MATCH (n) ... [k IN keys(n) WHERE toString(n[k]) + ...]`` all-property-all-node walk, which would toString() every key of + every node in the graph -- this codebase has scar tissue from exactly + that shape of full scan (a 1.3M-node AllNodesScan stall). ``UNION ALL`` + (not plain ``UNION``) is deliberate: plain ``UNION``'s implicit DISTINCT + would force Neo4j to materialize and dedupe every row before returning + the first one, defeating streaming; the Python ``set`` below already + dedupes, so ``UNION ALL`` costs nothing and preserves the stream. + + B2 (council amendment, retained): no APOC. Each returned value is + extracted via :func:`_extract_blob_refs_from_value` (structural + ``json.loads`` + recursive walk, falling back to a bare-token regex only + on JSON-parse failure -- see that function's docstring for why the regex + path is safe here and was NOT safe for the original ``Event.data`` scan). + A malformed/unparseable value is skipped conservatively -- it can never + positively assert an orphan, only fail to positively assert a reference. + + Streams rows via the async driver iterator rather than materializing all + carrier-property strings at once (only the resulting, much smaller, URI + set is retained), per the design's "bound its own work" cost note. + """ + driver = request.app.state.neo4j_query_driver + access_mode = _access_mode_const(request.app.state.neo4j_query_access_mode) + referenced: set[str] = set() + async with driver.session(default_access_mode=access_mode) as session: + # Graph-wide, per-property UNION ALL -- see the docstring above for + # why this shape (not an all-property walk, not plain UNION). The + # query text is generated from _BLOB_REF_CARRIER_PROPERTIES + # (_BLOB_REF_SCAN_QUERY, module level) -- not hand-duplicated here -- + # so it can never drift from the allowlist. + result = await session.run(_BLOB_REF_SCAN_QUERY) + async for record in result: + val = record["val"] + if not isinstance(val, str): + # toString() on a non-null value is always a str; this guards + # conservatively against an unexpected driver type mapping. + continue + _extract_blob_refs_from_value(val, referenced) + return referenced + + +async def _select_orphans(request: Request, *, min_age_minutes: int) -> dict[str, Any]: + """The ONE selection path shared by dry-run and apply (I5b hard rule). + + Returns a dict with every response field EXCEPT ``dry_run``/``sample``/ + ``rescanned``/``deleted``/``deleted_bytes`` (the caller fills those in), + plus a ``candidates`` key (list[_OnDiskBlob], sorted by uri for + deterministic sampling/capping) that the caller pops before returning the + response and uses to actually delete in apply mode. + + Safety gates applied to every on-disk blob not in the referenced set + (step 3 of the design): + 1. Undrained-queue gate (primary, durable -- B3): skipped when the + session has a live worker (``registry.active_sessions()``) OR its + queue is not fully drained (``QueueManager.is_fully_drained``, + durable across restarts). Counted as ``skipped_pending_session``. + 2. mtime floor (defense-in-depth -- B3): skipped when younger than + ``min_age_minutes`` (already clamped >= ``_MIN_AGE_FLOOR_MINUTES`` by + the request body validator). Counted as ``skipped_recent``. + """ + settings = get_settings() + blob_root = Path(settings.blob_path) + disk_blobs = _scan_disk_blobs(blob_root) + + referenced = await _scan_referenced_uris(request) + + registry = request.app.state.registry + queue_manager = registry.queue_manager + live_workers = set(registry.active_sessions()) + + now = time.time() + age_cutoff_seconds = min_age_minutes * 60 + + candidates: list[_OnDiskBlob] = [] + skipped_recent = 0 + skipped_pending_session = 0 + reclaimable_bytes = 0 + + for blob in disk_blobs: + if blob.uri in referenced: + continue + if blob.session_id in live_workers or not await queue_manager.is_fully_drained( + blob.session_id + ): + skipped_pending_session += 1 + continue + if now - blob.mtime < age_cutoff_seconds: + skipped_recent += 1 + continue + candidates.append(blob) + reclaimable_bytes += blob.size + + candidates.sort(key=lambda b: b.uri) + + return { + "scanned_disk_blobs": len(disk_blobs), + "referenced_uris": len(referenced), + "orphans_found": len(candidates), + "reclaimable_bytes": reclaimable_bytes, + "skipped_recent": skipped_recent, + "skipped_pending_session": skipped_pending_session, + "candidates": candidates, + } + + +class BlobReclaimBody(BaseModel): + """Body for POST /admin/blobs/reclaim.""" + + dry_run: bool = True + min_age_minutes: int = 60 + max_delete: int | None = Field(default=None, ge=1) + + @field_validator("min_age_minutes") + @classmethod + def _min_age_at_least_floor(cls, v: int) -> int: + """B3 (council amendment): reject (422) below the hard safety floor + rather than silently raising -- a caller passing 0 must not be able + to disable the age gate. + """ + if v < _MIN_AGE_FLOOR_MINUTES: + raise ValueError( + f"min_age_minutes must be >= {_MIN_AGE_FLOOR_MINUTES} " + f"(hard safety floor); got {v}" + ) + return v + + # --------------------------------------------------------------------------- # Per-request store dependencies (via app.state — no circular import) # --------------------------------------------------------------------------- @@ -544,3 +927,174 @@ def list_keys( return { "keys": [{"hash": h, "id": record.get("id", "")} for h, record in store.items()] } + + +# --- Blob reclaim (orphaned-blob GC) ---------------------------------------- + + +@router.post("/blobs/reclaim", status_code=200) +async def reclaim_blobs(body: BlobReclaimBody, request: Request) -> dict[str, Any]: + """Preview (dry-run) or apply reclamation of orphaned blob files. + + An orphan is an on-disk blob (``//blobs/.json``) + whose ``ci-blob://`` URI is referenced by NO ``:Event.data`` anywhere in + the graph (scanned globally, across ALL workspaces), and whose session is + both fully drained (durable ``QueueManager`` state, not in-memory worker + liveness) and older than ``min_age_minutes``. See + ``docs/plans/2026-08-12-blob-reclaim-endpoint-spec.md`` for the full + design and its council-mandated safety amendments (B1-B3). + + ``dry_run=true`` (default) computes and reports the candidate set without + deleting anything. ``dry_run=false`` requires ``max_delete`` (422 + otherwise -- a conscious blast-radius opt-in for an irreversible, + cross-workspace delete) and performs its OWN fresh, authoritative + ``_select_orphans`` scan at delete time (``rescanned: true`` in the + response) -- it never deletes a URI that is referenced or pending at the + moment of deletion, independent of any earlier dry-run preview. + + Deletion is a direct ``os.unlink`` on the blob path (idempotent -- a file + already gone is a no-op) capped at ``max_delete``; ``orphans_found`` and + ``reclaimable_bytes`` always reflect the FULL candidate set even when + ``max_delete`` caps how many are actually removed. One structured audit + log line is emitted per successful delete; blob CONTENTS are never + logged, only the ``ci-blob://`` URI. + """ + if not body.dry_run and body.max_delete is None: + raise HTTPException( + status_code=422, + detail=( + "max_delete is required when dry_run=false -- a conscious " + "blast-radius opt-in for an irreversible, cross-workspace " + "delete. Omit dry_run (or set it true) to preview first." + ), + ) + + # I5b: the ONE selection path. In apply mode this call itself IS the + # "own authoritative fresh scan at delete time" the design requires -- + # there is no earlier cached scan in this request to grow stale. + selection = await _select_orphans(request, min_age_minutes=body.min_age_minutes) + candidates: list[_OnDiskBlob] = selection.pop("candidates") + + response: dict[str, Any] = { + "dry_run": body.dry_run, + **selection, + "sample": [b.uri for b in candidates[:_MAX_SAMPLE]], + "rescanned": not body.dry_run, + "deleted": 0, + "deleted_bytes": 0, + } + + if body.dry_run: + return response + + assert body.max_delete is not None # guaranteed by the 422 guard above + deleted = 0 + deleted_bytes = 0 + # TOCTOU note: low severity (an unlink of an already-gone file is a + # no-op below); this loop runs immediately after the fresh scan above, + # well within the age floor, so the window is negligible -- stated, not + # assumed (design "RISKS folded in"). + for blob in candidates[: body.max_delete]: + try: + os.unlink(blob.path) + except FileNotFoundError: + continue # already gone -- idempotent, not an error + deleted += 1 + deleted_bytes += blob.size + _audit_blob_reclaim_delete(request, uri=blob.uri) + + response["deleted"] = deleted + response["deleted_bytes"] = deleted_bytes + return response + + +# --- Maintenance operation (WS-3c; seam + gate built in WS-3a) -------------- +# +# See docs/plans/2026-08-13-ws3-implementation-spec.md sec 6 for the full +# design. This endpoint is on ``maintenance.MAINTENANCE_ALLOW_LIST`` (enforced +# structurally by ``main._assert_maintenance_endpoint_allow_listed`` at +# startup) so it stays reachable even while the gate is closed -- otherwise +# it would 503 at exactly the moment it exists to unblock. + + +@router.post("/maintenance", status_code=202) +async def post_maintenance(request: Request) -> JSONResponse: + """Trigger the maintenance repair operation (dedup -> :Node backfill -> + schema DDL), reusing the existing ``neo4j_store.run_repair`` -- see + ``maintenance_ops.run_maintenance_operation``. + + Single-flight (council MUST-FIX #3): ``coordinator.try_begin_op()`` is a + synchronous compare-and-swap with no ``await`` between check and set, so + two concurrent POSTs can never both start an op. A second POST while one + is running gets **409** with the in-progress ``run_id`` -- it never + starts a second op. + + Returns promptly (council MUST-FIX): the operation runs as a background + task; this handler returns **202** immediately rather than blocking for + the op's full duration (which includes the quiesce sleep + an + O(graph-size) dedup pass). + + Idempotent honest re-scan (D-H): POST against an already-clean graph + still performs a genuine ``run_repair`` call and returns a **fresh** + ``run_id``/``completed_at`` with ``records_affected: 0`` -- it never + short-circuits, which would make ``0`` indistinguishable from "did not + run" and defeat the freshness marker. + """ + run_id = coordinator.try_begin_op() + if run_id is None: + current = coordinator.current_op() + return JSONResponse( + status_code=409, + content={ + "detail": "maintenance operation already running", + "run_id": current.run_id, + "state": current.state, + }, + ) + + settings = get_settings() + # Defensive getattr (matches _check_driver_connected's convention above): + # lets this route degrade gracefully rather than 500 if ever hit before + # lifespan has bound a driver -- run_repair() will raise on a None + # driver, which finish_op() records as a normal `failed` outcome. + driver = getattr(request.app.state, "neo4j_driver", None) + task = asyncio.create_task( + run_maintenance_operation( + driver, + run_id, + quiesce_seconds=settings.maintenance_quiesce_seconds, + ) + ) + coordinator.retain_task( + task + ) # strong ref -- see MaintenanceCoordinator.retain_task + + op = coordinator.current_op() + return JSONResponse( + status_code=202, + content={"run_id": run_id, "state": op.state, "started_at": op.started_at}, + ) + + +@router.get("/maintenance", status_code=200) +async def get_maintenance() -> dict[str, Any]: + """Report maintenance-operation progress. + + Separate, admin-authenticated route -- council: required, and never + folded into ``/status`` (``/status`` is unauthenticated; folding these + fields in would be an auth-boundary leak). ``state`` initializes to + ``"unknown"`` on boot so "never ran" is distinguishable from "ran, record + lost to a crash" (council D4). + """ + st = await coordinator.status() + op = st.op + return { + "mode": st.mode, + "state": op.state, + "run_id": op.run_id, + "started_at": op.started_at, + "completed_at": op.completed_at, + "elapsed_seconds": st.elapsed_seconds, + "records_affected": op.records_affected, + "error": op.error, + } diff --git a/context_intelligence_server/routers/version.py b/context_intelligence_server/routers/version.py index 1273273f..97377136 100644 --- a/context_intelligence_server/routers/version.py +++ b/context_intelligence_server/routers/version.py @@ -4,19 +4,23 @@ from fastapi import APIRouter -from context_intelligence_server.status import SERVER_VERSION +from context_intelligence_server.status import SCHEMA_VERSION, SERVER_VERSION router = APIRouter() @router.get("/version") -async def get_version() -> dict[str, str]: - """Return the running server version. +async def get_version() -> dict[str, str | int]: + """Return the running server version and expected graph schema version. This endpoint is intentionally unauthenticated so clients can check server compatibility without credentials. + ``schema_version`` is a read-only baseline data point (no comparison or + upgrade logic lives here — see ``SCHEMA_VERSION`` in ``status.py``). + Returns: - JSON object with a single ``version`` key, e.g. ``{"version": "2.0.0"}``. + JSON object with ``version`` and ``schema_version`` keys, e.g. + ``{"version": "2.0.0", "schema_version": 1}``. """ - return {"version": SERVER_VERSION} + return {"version": SERVER_VERSION, "schema_version": SCHEMA_VERSION} diff --git a/context_intelligence_server/services.py b/context_intelligence_server/services.py index b5f0230b..f19f67ed 100644 --- a/context_intelligence_server/services.py +++ b/context_intelligence_server/services.py @@ -7,8 +7,10 @@ from __future__ import annotations +import dataclasses import fnmatch import logging +from dataclasses import asdict from datetime import datetime from typing import Any @@ -256,9 +258,19 @@ async def ensure_session_node(self, session_id: str, data: dict[str, Any]) -> No ``_seen_sessions`` cache, return immediately. 2. Graph query — call ``graph.get_node(session_id)``. If the node already exists (e.g. from a previous run), repopulate the cache and - return without overwriting any data. If the node is absent, create + return without overwriting any existing data — except ``working_dir`` + is populate-if-missing (see below). If the node is absent, create it with labels ``["Session"]`` and ``status = 'running'``. + ``working_dir`` is populate-if-missing on every call, not just node + creation: if the incoming event carries a non-empty ``working_dir`` and + the node does not already have one, the gap is filled. An + already-populated ``working_dir`` is never overwritten, and an empty/ + absent incoming value never clears an existing one. This is what lets + re-importing an existing local session (e.g. via the upload CLI) + backfill ``working_dir`` on a Session node that was created before the + field existed. + This method is a safety net that creates a minimal session node if it doesn't exist. ``SessionHandler`` is the sole authority on session type labels (``RootSession``, ``SubSession``, ``ForkedSession``). @@ -285,10 +297,23 @@ async def ensure_session_node(self, session_id: str, data: dict[str, Any]) -> No # 4. Worker B's flush later issues a fresh MERGE → duplicate node. # upsert_node uses union-merge for labels, so existing type labels # (e.g. "RootSession") are preserved — this call never strips labels. - await self.graph.upsert_node( - session_id, - {"labels": ["Session"], "status": "running", "session_id": session_id}, - ) + stub_data: dict[str, Any] = { + "labels": ["Session"], + "status": "running", + "session_id": session_id, + } + # I1: populate-if-missing. The node may already + # exist with working_dir null/absent (e.g. it predates this event, or was + # created via a delegation/fork reference before its own lifecycle events + # arrived). If this event carries a non-empty working_dir AND the existing + # node does not already have one, fill the gap. Never overwrite an + # already-populated value, and never write an empty incoming value — + # this is the fix that lets re-importing an existing local session (e.g. + # via the upload CLI) backfill working_dir instead of leaving it null + # forever because this branch used to return early without touching it. + if data.get("working_dir") and not existing.get("working_dir"): + stub_data["working_dir"] = data["working_dir"] + await self.graph.upsert_node(session_id, stub_data) self._seen_sessions.add(session_id) return @@ -316,6 +341,13 @@ async def ensure_session_node(self, session_id: str, data: dict[str, Any]) -> No node_data["started_at"] = _ts if "agent" in data: node_data["agent"] = data["agent"] + # I1: lift working_dir onto the Session node when the + # ingest layer supplied it (see main.py post_events). Populate-if-missing: the + # already-exists branch above fills the same gap for nodes created before this + # field existed, so working_dir is not stuck null forever -- it is backfilled + # the next time any event (including a re-import via the upload CLI) carries it. + if data.get("working_dir"): + node_data["working_dir"] = data["working_dir"] await self.graph.upsert_node(session_id, node_data) self._seen_sessions.add(session_id) # only cache after successful write @@ -367,3 +399,58 @@ async def touch_session(self, session_id: str, timestamp: str) -> None: timestamp, exc_info=True, ) + + # ------------------------------------------------------------------ + # Durable cursor (I5b: worker-rebuild cursor durability) + # ------------------------------------------------------------------ + + def snapshot_cursor(self) -> dict[str, Any]: + """Return a JSON-safe snapshot of cross-handler cursor state. + + Full snapshot of ``data_layer_2``/``data_layer_3`` (not a hand-picked + subset): every field on those dataclasses is JSON-native, and taking + the whole dataclass via ``asdict`` is both simpler and safer than a + hand-maintained allowlist that silently misses new fields. + """ + return { + "dl2": asdict(self.data_layer_2), + "dl3": asdict(self.data_layer_3), + } + + def restore_cursor(self, record: dict[str, Any] | None) -> None: + """Restore cross-handler cursor state from a persisted snapshot. + + No-op on ``record is None`` (nothing to restore \u2014 e.g. a brand-new + session, or a legacy ``.offset`` file with no cursor). Otherwise, + for each of ``data_layer_2``/``data_layer_3``, only the field NAMES + present on the current dataclass are assigned \u2014 unknown/renamed keys + in the record are dropped, and any field missing from the record + keeps its dataclass default. This field-name filtering (rather than + a fixed schema) is what lets the persisted format tolerate dataclass + evolution in both directions without a version bump. + + Safe-degrade (D5): any failure while restoring is caught, logged at + WARNING, and leaves the dataclasses at their defaults \u2014 a corrupt or + unexpected cursor must never crash boot. + """ + if record is None: + return + try: + for key, target in ( + ("dl2", self.data_layer_2), + ("dl3", self.data_layer_3), + ): + value = record.get(key) + if not isinstance(value, dict): + continue + valid_fields = {f.name for f in dataclasses.fields(type(target))} + for field_name, field_value in value.items(): + if field_name in valid_fields: + setattr(target, field_name, field_value) + logger.debug( + "cursor_restored iteration_count=%s execution_start_ts=%s", + self.data_layer_2.iteration_count, + self.data_layer_2.execution_start_ts, + ) + except Exception: + logger.warning("cursor_restore_failed", exc_info=True) diff --git a/context_intelligence_server/status.py b/context_intelligence_server/status.py index 0ae0049f..3ca5778c 100644 --- a/context_intelligence_server/status.py +++ b/context_intelligence_server/status.py @@ -13,6 +13,16 @@ # Resolved once at import time — never changes within a process lifetime. SERVER_VERSION: str = _pkg_version("context-intelligence-server") +# Baseline data point only (§10.2 of the cursor-durability spec): a plain integer +# answering ONLY "does the server's expected schema match the DB's stored schema, +# yes/no". Covers BOTH the graph schema (indexes/constraints) AND the additive +# `Iteration.iteration_scope` property — one number for the whole current graph +# shape. No semantic/dotted versioning. Comparison/upgrade/migration logic that +# ACTS on this value is deliberately deferred to a later phase — this is a passive +# data point, exposed read-only via /version and written create-if-absent to +# Neo4j (see `ensure_neo4j_schema` in neo4j_store.py). +SCHEMA_VERSION: int = 1 + if TYPE_CHECKING: from context_intelligence_server.registry import SessionRegistry diff --git a/docs/azure-deployment.md b/docs/azure-deployment.md index bbca231d..cba2821b 100644 --- a/docs/azure-deployment.md +++ b/docs/azure-deployment.md @@ -639,7 +639,9 @@ overrides: ## Updating the server (build & deploy a new version — Neo4j-safe) Runbook for shipping a new version (e.g. `v6.7.0`) without disturbing Neo4j. -Placeholders in ``. +Placeholders in ``. (Illustrative version numbers below are +NOT kept in lockstep with the current release — e.g. the deploy-safe boot fix +ships as `6.7.1`; substitute whatever version you are actually shipping.) **Pre-flight** - Repo `pyproject.toml` version == the version you're shipping. @@ -696,6 +698,19 @@ az containerapp update -n -g \ expect the new revision Running/Healthy, no `access_mode` validation error, and Neo4j connected on **both** clients. +> ⚠️ **`GET /status`'s `schema_health` field is NOT a liveness/readiness signal.** +> Since 6.7.1 the server never crash-loops on graph migration/reachability +> state (deploy-safe boot) — a genuine data conflict or an unreachable graph +> at boot is reported as `schema_health: "degraded"` or `"unknown"` on +> `GET /status` while the server continues to boot and serve. **Do NOT wire +> `schema_health` (or `/status` at all) to a Container Apps/Kubernetes +> liveness or readiness probe** — doing so would recreate the exact +> crash-loop this fix removes, one layer up, the moment a normal ACA +> cold-start race or credential rotation makes the probe transiently +> unreachable. Use a plain HTTP-200 check against `/status` (or `/version`) +> for liveness if one is needed; treat `schema_health` as an operator/ +> automation signal to read, not a gate to enforce. + ### Neo4j safety — guarantees & do-NOT-touch list **Why `amplifier-online up` cannot disturb Neo4j:** the Neo4j VM is a **separate diff --git a/docs/local-development.md b/docs/local-development.md index 38a198ad..9624a4b3 100644 --- a/docs/local-development.md +++ b/docs/local-development.md @@ -124,7 +124,7 @@ Point the server at the generated config and start it with uv: ```bash export AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_CONFIG_FILE="$(pwd)/server-config.yaml" uv sync -uv run uvicorn context_intelligence_server.main:app --host 127.0.0.1 --port 8000 +uv run uvicorn context_intelligence_server.main:asgi_app --host 127.0.0.1 --port 8000 ``` > `uvicorn --reload` is for **local dev only**. For a persistent/shared run, use @@ -171,7 +171,7 @@ Show me the API token once and remind me to save it. ``` Set AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_CONFIG_FILE to the server-config.yaml in the repo root, run `uv sync`, then start the server with -`uv run uvicorn context_intelligence_server.main:app --host 127.0.0.1 --port 8000`. +`uv run uvicorn context_intelligence_server.main:asgi_app --host 127.0.0.1 --port 8000`. Confirm it's healthy by curling http://127.0.0.1:8000/status. ``` diff --git a/docs/maintenance-mode.md b/docs/maintenance-mode.md new file mode 100644 index 00000000..82104d11 --- /dev/null +++ b/docs/maintenance-mode.md @@ -0,0 +1,371 @@ +# Maintenance Mode + +Runtime, no-restart gate that refuses ingest and query while the graph's +identity constraint is absent or a repair operation is in flight — the +live counterpart to the cold-start refusal described in the [README's +"Cold Start No Longer Auto-Migrates"](../README.md) section. All state +lives in one seam: `context_intelligence_server/maintenance.py`'s +`MaintenanceCoordinator` (module singleton `coordinator`). The HTTP gate +(`maintenance_gate_middleware`), the ingest drain loop +(`registry.drain_worker`), and `GET /status` all read from this one +coordinator, so they can never disagree about whether the server is in +maintenance. + +## Why this exists + +`node_node_id_workspace_unique` is the `:Node` uniqueness constraint that +makes every write path's `MERGE` safe. If that constraint is missing — +because the graph has duplicate legacy nodes it conflicts with — the +schema is in a state where **both writes and reads are unsafe**: a write +can silently create a duplicate node instead of merging onto the existing +one, and a read (`/cypher`) can return results computed against that +duplicated, inconsistent shape. Likewise, while an operator-triggered +repair (dedup + `:Node` backfill + constraint creation) is actually +executing, the graph is mid-mutation — reads and writes during that +window would race the repair. + +Maintenance mode is the server's answer to both cases: refuse the request +with a `503` and a `Retry-After` header instead of serving a request +against schema-unsafe or mid-repair data. + +## The mode state machine + +`MaintenanceCoordinator._derive_mode()` is the **one** place mode is +computed; every caller (`gate_closed()`, `status()`) goes through it, so a +mode transition is always caught regardless of which surface polls first. +`MaintenanceMode` is one of four values: + +| Mode | When | Gates ingest/query? | +|---|---|---| +| `healthy` | Constraint present, no op running, boot-time untagged-node count is 0 | No | +| `degraded` | Constraint present, no op running, but the **boot-time** untagged-node probe found `> 0` nodes lacking the `:Node` label | No | +| `unknown` | The live constraint probe could not run (Neo4j unreachable, credentials rejected, etc.) | No — an unreachable graph can't serve writes/reads anyway, so gating here would only buy an outage with no safety benefit | +| `maintenance` | **Either** a maintenance op is currently running (`op.state == "running"`), **or** the live probe found the `:Node` constraint absent | **Yes** — this is the only mode `gate_closed()` treats as closed | + +Exact derivation order (`maintenance.py:267-291`), evaluated in this +priority: + +1. `op.state == "running"` → `mode="maintenance"`, `reason="maintenance operation in progress"` +2. else `constraint_present is False` → `mode="maintenance"`, `reason=":Node uniqueness constraint absent -- migration required"` +3. else `constraint_present is None` → `mode="unknown"`, `reason="constraint probe could not determine graph state"` +4. else boot-time `untagged_nodes > 0` → `mode="degraded"`, `reason="{n} node(s) lacking the :Node label"` +5. else → `mode="healthy"`, `reason=None` + +`gate_closed()` is exactly `mode == "maintenance"` — `degraded` and +`unknown` never gate (`maintenance.py:295-303`). + +### The live probe + +The constraint check is a cheap catalog read, not a data scan: + +```cypher +SHOW CONSTRAINTS YIELD name WHERE name = 'node_node_id_workspace_unique' RETURN count(*) AS c +``` + +It is **TTL-cached and single-flight** (`maintenance_probe_ttl_seconds`, +default `5.0` seconds): the fast path never takes a lock, and concurrent +callers that see an expired cache collapse into exactly one live probe. +This is what makes maintenance mode **self-clearing with no server +restart** — once the constraint is repaired out-of-band, the next probe +(within the TTL window) sees `constraint_present=True` and the mode flips +back on its own. + +## What happens while gated + +While `mode == "maintenance"`: + +- **Ingest is gated in the drain loop, not at spawn.** The check is the + first statement inside `drain_worker`'s `while True:` loop + (`registry.py:373-388`), before `read_batch`. This placement (rather + than refusing at `get_or_create → start_drain`) is deliberate: gating + *spawn* would be a **latch** — refuse once, never retried — the exact + defect class this feature exists to prevent. Drainers still spawn + during maintenance; they idle in a `_GATED_POLL_INTERVAL` sleep loop + instead. Because everything downstream of the gate check (`read_batch`, + `process`, `flush`, `commit`) is unreachable while gated, the on-disk + offset **never advances** during a gated window — a crash mid-maintenance + and restart replays from the last successfully-flushed offset, exactly + once. +- **`POST /events`** (including dead-letter `?replay=true` re-ingestion, + since it flows through the same drain loop) is refused via the HTTP + gate below before it can even reach the queue-append step. +- **`POST /cypher`** is refused via the same HTTP gate. +- **`GET /status` and `GET /version` stay up.** They are explicitly on the + allow-list (see below) — this is how an operator/automation observes + maintenance mode in the first place. + +### The structured 503 + +`maintenance_gate_middleware` (registered on `app` itself, via +`app.middleware("http")(maintenance_gate_middleware)` at `main.py:466`) +runs on every request whose path is not on the allow-list. When the +coordinator reports `mode == "maintenance"`, it returns the **one** +producer of this response, `maintenance_response()` — deliberately not a +FastAPI `HTTPException` (which would render `{"detail": ...}`, the wrong +contract here): + +```json +{ + "status": "maintenance", + "reason": ":Node uniqueness constraint absent -- migration required", + "retry_after": 30, + "schema_health": "degraded", + "maintenance_started_at": "2026-08-13T12:00:00.123456+00:00" +} +``` + +with an HTTP header `Retry-After: 30` (the numeric value mirrors the +`retry_after` field). `retry_after` is sourced from +`maintenance_retry_after_seconds` (default `30`). + +`reason` is the same human-readable string from the mode derivation above +(e.g. `"maintenance operation in progress"` when an op is running). +`schema_health` in this body is a **coarser, two-state** field than the +one on `/status` (below) — it is `"unknown"` only when the live probe +returned `None`, and `"degraded"` for every other case that reaches this +response, **including** the "op running, constraint actually present" +case. Operators should not read `"degraded"` here as "the constraint is +absent" — check `reason` for the actual cause. + +## `GET /status` fields + +`GET /status` is unauthenticated and always reachable (it's on the +allow-list). The maintenance-relevant fields, set in `main.py`'s +`get_status` handler (`main.py:897-933`): + +| Field | Source | Meaning | +|---|---|---| +| `mode` | live, via `coordinator.status()` | `"healthy"` \| `"degraded"` \| `"unknown"` \| `"maintenance"` — de-latched: reflects the current probe, not a boot-time snapshot | +| `maintenance_started_at` | live | ISO-8601 UTC timestamp the **current** maintenance window opened; `null` when not in maintenance | +| `maintenance_elapsed_seconds` | live | Seconds since that window opened; `null` when not in maintenance | +| `schema_health` | live (computed independently in `main.py`, a **3-state** version, not reused from the 503's 2-state ternary) | `"unknown"` if the probe returned `None`; `"degraded"` if the constraint is absent **or** the boot-time untagged count was `> 0`; else `"healthy"` | +| `untagged_nodes` | **boot-time snapshot**, not live | Count of `:Node`-label-missing nodes at the last server boot. Documented as boot-time — it is not a gate input; the live gate/mode signal is the constraint probe above | +| `schema_checked_at` | `datetime.now(UTC)` at the moment this `/status` call is served | Not the exact instant the constraint probe last ran — the probe result may be up to `maintenance_probe_ttl_seconds` stale | +| `degraded_reason` | **boot-time snapshot**, set once by `_record_schema_health` at startup | Human-readable cause captured at boot (e.g. `":Node uniqueness constraint absent (data conflict)"`, `"N node(s) lacking the :Node label"`, or the probe-unreachable message); **not** recomputed on every call | + +Example (in maintenance, constraint absent): + +```json +{ + "mode": "maintenance", + "maintenance_started_at": "2026-08-13T12:00:00.123456+00:00", + "maintenance_elapsed_seconds": 42.7, + "schema_health": "degraded", + "untagged_nodes": 0, + "schema_checked_at": "2026-08-13T12:00:42.831112+00:00", + "degraded_reason": null +} +``` + +(`degraded_reason` is `null` here deliberately — it is a boot-time field, +and this example assumes the constraint was present at boot and only went +absent later; a boot-time degradation would populate it.) + +Example (healthy): + +```json +{ + "mode": "healthy", + "maintenance_started_at": null, + "maintenance_elapsed_seconds": null, + "schema_health": "healthy", + "untagged_nodes": 0, + "schema_checked_at": "2026-08-13T12:05:00.001112+00:00", + "degraded_reason": null +} +``` + +**Note:** `/status` does not surface the coordinator's internal op record +(`run_id` / `state` / `records_affected` / `error`). That detail is +deliberately kept off the unauthenticated `/status` surface and lives on +the admin-authenticated [`GET /admin/maintenance`](#postget-adminmaintenance--the-live-endpoint-contract) +instead, plus the transition log lines (below). + +## Allow-list — blast radius + +`MAINTENANCE_ALLOW_LIST` (`maintenance.py:100-102`) is an **allow-list, +not a deny-list**, matched on exact path (`request.url.path in +MAINTENANCE_ALLOW_LIST`) — deliberately, so any route added to the app +later is blocked-by-default rather than silently exempt: + +```python +MAINTENANCE_ALLOW_LIST: frozenset[str] = frozenset( + {"/status", "/version", "/admin/maintenance", "/docs", "/openapi.json"} +) +``` + +| Stays reachable during maintenance | Returns `503` during maintenance | +|---|---| +| `GET /status` | `POST /events` (including dead-letter replay) | +| `GET /version` | `POST /cypher` | +| `POST`/`GET /admin/maintenance` (live -- the one `/admin/*` route reachable during maintenance; see [the endpoint contract below](#postget-adminmaintenance--the-live-endpoint-contract)) | `GET /blobs/{session_id}`, `GET /blobs/{session_id}/{key}` | +| `GET /docs`, `GET /openapi.json` (Swagger UI / OpenAPI schema) | `GET /queues/dead-letter` and the other `/queues/*` routes | +| | `GET /admin/identities`, `GET /admin/keys`, `POST /admin/blobs/reclaim`, and every other `/admin/*` route | + +This is **intentional**, not a bug: a blob-reclaim scan +(`POST /admin/blobs/reclaim`) walks the graph to decide which blobs are +still referenced, and a mid-dedup or mid-backfill graph would misclassify +live blobs as orphaned. Every other `/admin/*` route and all data-plane +routes are unsafe against a schema-unsafe or mid-repair graph for the +same underlying reason described in [Why this exists](#why-this-exists). + +The middleware is registered on `app` itself (`main.py:466`), **not** the +auth-wrapped `asgi_app` — so it cannot be bypassed by the bare +`main:app` entrypoint. `BearerTokenMiddleware` wraps `app`, so +authentication still runs first; an unauthenticated request 401s before +it ever reaches this gate. A startup assertion, +`_assert_maintenance_endpoint_allow_listed()` (`main.py:542-559`), fails +loud if `/admin/maintenance`, `/status`, or `/version` are ever missing +from the allow-list — this prevents `/admin/maintenance` from ever +502-ing itself out of existence. + +## Log events to grep for + +| Event | Level | Emitted by | Fields | +|---|---|---|---| +| `maintenance_entered` | INFO | `maintenance.py` (`_handle_transition`) | `reason`, `run_id`, `trigger` (`"op"` or `"constraint"`) | +| `maintenance_completed` | INFO | `maintenance.py` (`_handle_transition`) | `reason`, `run_id`, `duration_seconds` | +| `maintenance_probe_failed` | WARNING | `maintenance.py` (`_run_probe`) | `error` — the live probe raised | +| `maintenance_finish_op_run_id_mismatch` | WARNING | `maintenance.py` (`finish_op`) | `expected`, `got` — a stale/foreign completion signal was ignored | +| `maintenance_quiesce` | INFO | `maintenance_ops.py` (`run_maintenance_operation`) | `run_id`, `seconds` | +| `maintenance_op_succeeded` | INFO | `maintenance_ops.py` | `run_id`, `records_affected` | +| `maintenance_op_failed` | ERROR (exception) | `maintenance_ops.py` | `run_id` (+ traceback) | +| `schema_degraded` | ERROR | `main.py` (`_record_schema_health`, boot only) | reason string | +| `startup_degraded` | ERROR (exception) | `main.py` (`lifespan`, boot only) | the exception that made the whole startup boundary degrade | + +`_handle_transition` logs each open↔closed transition **exactly once**: +it runs with no `await` in its body, so two concurrent callers observing +the same transition can never both log it. + +The last three of the `maintenance_ops.py` events +(`maintenance_quiesce` / `maintenance_op_succeeded` / `maintenance_op_failed`) +fire on every `POST /admin/maintenance` call, since that endpoint schedules +`run_maintenance_operation()` as its background task (see below). They do +**not** fire for `migrations/run.py --apply` or `doctor --fix` — both call +`neo4j_store.run_repair()` directly and never go through +`run_maintenance_operation` (or the coordinator's `try_begin_op`/`finish_op` +bookkeeping). Their effect is still observable via the constraint-probe +transition lines (`maintenance_entered` / `maintenance_completed`) once the +next live probe notices the restored constraint. + +## Clearing maintenance + +The coordinator self-clears the moment its live probe sees the constraint +restored — no restart required, on any of the channels below — but +something has to actually restore it first. Two supported channels exist +today, both ultimately calling the same `neo4j_store.run_repair` (dedup → +`:Node` backfill → constraint create): + +- **`POST /admin/maintenance`** — the network-reachable channel (e.g. + cloud/ACA deployments where the private Neo4j is not directly + reachable). Admin-authenticated HTTP call; schedules + `maintenance_ops.run_maintenance_operation()` as a background task, + which wraps `run_repair` with a quiesce sleep and reports the outcome + back through the coordinator (`try_begin_op`/`finish_op`). Poll + `GET /admin/maintenance` for progress. See + [the endpoint contract below](#postget-adminmaintenance--the-live-endpoint-contract) + for the full request/response shape. +- **`migrations/run.py --apply`** — the local/VM/direct-Neo4j channel + (`migrations/run.py`), for operators who can reach Neo4j directly. + Standalone script, run out-of-band from the server process; it calls + `run_repair` directly with no coordinator/HTTP/admin-auth involvement. + `migrations/run.py --status` gives a read-only report first (constraint + presence, untagged/duplicate counts) without writing anything. + +Both channels run the identical repair logic — which one to use is purely +a matter of which one a given deployment can reach. `doctor --fix` +(`context-intelligence-server doctor --fix`) is an older, still-available +entry point to the same `run_repair` function (`doctor.py`); it remains an +equivalent low-level tool, but it is not the primary lever documented here +for clearing a *live* maintenance window — it predates this endpoint and +was originally documented (see the [README](../README.md)) for the +cold-start-refuses-to-boot case. + +Once `run_repair` successfully re-creates the +`node_node_id_workspace_unique` constraint — via any of the three tools +above — the **next** probe (at most `maintenance_probe_ttl_seconds` later, +whether triggered by an ingest drainer's gate check, a `GET /status` poll, +or a `GET /admin/maintenance` poll) observes `constraint_present=True` and +`mode` flips back to `healthy`/`degraded` on its own. No server restart is +required. + +If the gate is closed because an **op is running** (`op.state == +"running"`) rather than because the constraint is absent, that sub-state +clears only via `coordinator.finish_op(...)`, which +`run_maintenance_operation` calls automatically once the background task +completes (success or failure) — see the endpoint contract below. +`migrations/run.py --apply` and `doctor --fix` never go through the +coordinator, so they can neither open nor close the "op running" +sub-state; they only affect the constraint-presence sub-state. + +## `POST`/`GET /admin/maintenance` — the live endpoint contract + +Both routes are defined in `context_intelligence_server/routers/admin.py` +and are on `maintenance.MAINTENANCE_ALLOW_LIST`, so they stay reachable +even while the gate is closed (enforced structurally at startup by +`main._assert_maintenance_endpoint_allow_listed()` — otherwise the +endpoint would 503 at exactly the moment it exists to unblock). + +- **`POST /admin/maintenance`** — admin-authenticated (`require_admin`, + same 401/403 matrix as every other `/admin/*` route). Single-flight: + `coordinator.try_begin_op()` is a synchronous compare-and-swap with no + `await` between check and set, so two concurrent POSTs can never both + start an op. On winning the CAS, it schedules + `maintenance_ops.run_maintenance_operation()` as a background + `asyncio.Task` (retained via `coordinator.retain_task(...)` so it can't + be garbage-collected mid-run) and returns **promptly** — it does not + await the op's duration (quiesce sleep + the O(graph-size) dedup pass): + - **202** on winning the CAS: `{"run_id": "...", "state": "running", "started_at": "..."}` + - **409** if an op is already running: `{"detail": "maintenance operation already running", "run_id": "", "state": "running"}` + - Re-running on an already-clean graph is a genuine re-scan, not a + short-circuit: each `POST` performs a real `run_repair` call and gets + a fresh `run_id`/`completed_at`, even when `records_affected` comes + back `0`. +- **`GET /admin/maintenance`** — admin-authenticated, same matrix. Returns: + + ```json + { + "mode": "maintenance", + "state": "running", + "run_id": "3f9e...", + "started_at": "2026-08-13T12:00:00.123456+00:00", + "completed_at": null, + "elapsed_seconds": 4.2, + "records_affected": null, + "error": null + } + ``` + + | Field | Meaning | + |---|---| + | `mode` | The coordinator's current mode (`"healthy"` \| `"degraded"` \| `"unknown"` \| `"maintenance"`) — the same value `/status` reports | + | `state` | The op record's state: `"unknown"` \| `"running"` \| `"succeeded"` \| `"failed"`. `"unknown"` with every other field `null` means no op has run yet in this process | + | `run_id` | The op's id (uuid4 hex), `null` if none has run | + | `started_at` / `completed_at` | ISO-8601 UTC timestamps for the op; `completed_at` is `null` while `state == "running"` | + | `elapsed_seconds` | Seconds since the **current maintenance window** opened (the same value as `/status`'s `maintenance_elapsed_seconds`) — not strictly the op's own runtime; `null` when not in maintenance | + | `records_affected` | `duplicates_removed + nodes_tagged` from the last completed `run_repair` call, `null` until one completes | + | `error` | Human-readable exception string if the last op failed, else `null` | + +Neither route is exposed in `/openapi.json` or `/docs` (the whole +`/admin` router is mounted with `include_in_schema=False`) — this affects +schema visibility only, not routing or auth. + +## Do not wire this to a liveness/readiness probe + +The same written prohibition that applies to boot-time `schema_health` +(see the [Azure deployment guide](azure-deployment.md)) applies here, +verbatim, per the code comments in both `main.py` (`_record_schema_health` +docstring and the `/status` handler) and `maintenance.py`: + +> `schema_health`/`mode` MUST NOT be wired to a Kubernetes/ACA liveness or +> readiness probe. Doing so would recreate the exact crash-loop the +> deploy-safe-boot fix removes, one layer up. + +`mode`, `schema_health`, `maintenance_started_at`, and +`maintenance_elapsed_seconds` are **data-migration signals** for an +operator or automation to *read* — not a gate to wire into container +orchestration health checks. `/status` (and `/version`) stay reachable +and return `200` throughout a maintenance window specifically so a plain +HTTP-200 liveness check against either of those paths keeps working +un-interrupted; use that for liveness if one is needed, and treat the +maintenance fields as a separate, human/automation-facing signal. diff --git a/migrations/__init__.py b/migrations/__init__.py new file mode 100644 index 00000000..cda7f6c8 --- /dev/null +++ b/migrations/__init__.py @@ -0,0 +1,6 @@ +"""Out-of-band graph rectification tools. + +Nothing in this package is imported by the server at startup or on the +request path -- see ``run.py`` for the CORE PRINCIPLE this package exists to +honor (migrations run OUT-OF-BAND, never in the server's critical path). +""" diff --git a/migrations/manifest.yaml b/migrations/manifest.yaml new file mode 100644 index 00000000..725a5376 --- /dev/null +++ b/migrations/manifest.yaml @@ -0,0 +1,30 @@ +# Machine-readable upgrade/migration manifest. +# +# Establishes the lean upgrade mechanism (docs/plans/2026-08-13-review-remediation-plan.md +# "Upgrade-path scope"): one ordered entry per server version that requires +# out-of-band action on the graph. No rollback machinery -- these are +# forward-only structural rectifications, not reversible schema migrations. +# +# Consumed by: operators, and Amplifier deployment automation deciding +# whether an upgrade needs an out-of-band step before clients resume. +entries: + - id: "6.8.0-maintenance-mode" + server_version: "6.8.0" + schema_version: 1 + schema_affecting: false + summary: > + Adds maintenance mode (gates ingest + query while the :Node uniqueness + constraint is absent, structured 503 + Retry-After, live re-probe + self-clear, /admin/maintenance execution channel). Does not change any + stored node/edge shape -- schema_version stays 1. A deployment with + pre-existing un-migrated/duplicate nodes will boot into maintenance + mode under 6.8.0 (previously it booted degraded but kept accepting + writes); that graph must be rectified once, out-of-band. + scripts: + - "migrations/run.py --apply" + gating: > + If GET /status reports mode=maintenance or mode=degraded, run the + rectification before clients resume sending events. A healthy graph + (schema_health=healthy) needs no action -- boots normally under 6.8.0 + with zero behavior change. + verify: "GET /status shows schema_health=healthy (equivalently mode=healthy)" diff --git a/migrations/run.py b/migrations/run.py new file mode 100644 index 00000000..2e3509f6 --- /dev/null +++ b/migrations/run.py @@ -0,0 +1,272 @@ +"""Standalone, OUT-OF-BAND graph rectification tool. + +``migrations/run.py`` is the local/VM/direct-Neo4j sibling of +``POST /admin/maintenance`` (the network-reachable channel for cloud +deployments). Both call the SAME underlying repair logic +(``neo4j_store.run_repair`` / ``diagnose``) -- this script adds no new +algorithm, only a CLI around the existing functions (DRY, per +``docs/plans/2026-08-13-ws3-implementation-spec.md`` sec 5.1/5.4). + +WHAT THIS DOES + Rectifies an already-degraded graph: dedups duplicate legacy nodes, + backfills the universal ``:Node`` label, and (re-)creates the + ``:Node`` uniqueness constraint that the running server's maintenance + gate (``context_intelligence_server/maintenance.py``) probes for. + This is NOT a schema-version migration -- WS-3 changes no stored + node/edge shape, so ``SCHEMA_VERSION`` (``status.py``) stays ``1``. + It is structural rectification of pre-existing degraded/un-migrated + data, made necessary by upgrading the server past 6.7.x. + +WHY IT EXISTS -- OUT-OF-BAND, NEVER AT STARTUP + CORE PRINCIPLE (AGENTS.md, workspace root): migrations run OUT-OF-BAND + against a live Neo4j instance -- never inside the server's + startup/critical path. This script is NOT imported by the server and + is NEVER invoked from ``lifespan()`` or any request handler. Run it by + hand (local/VM) or trigger the equivalent in-server channel, + ``POST /admin/maintenance`` (cloud, where a human cannot reach the + private Neo4j directly) -- see the README "Upgrading" section. + +IDEMPOTENT + ``--status`` never writes. ``--apply`` calls ``run_repair``, which is + idempotent (dedup -> :Node backfill -> constraint create, all + ``IF NOT EXISTS`` / MERGE-based) -- re-running against an + already-clean graph is a safe no-op. + +USAGE + python migrations/run.py --status + uv run python migrations/run.py --apply + python -m migrations.run --status + + Connection is resolved the same way the server resolves it: via + ``config.Settings`` (``server-config.yaml`` then + ``AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_*`` env vars), using the + SAME driver constructor the server and ``doctor`` CLI use + (``main.build_neo4j_driver``) so this tool can never connect + differently than the process it is rectifying data for. Pass + ``--neo4j-url`` / ``--neo4j-user`` / ``--neo4j-password`` to override + any of the three independently (e.g. pointing at a different Neo4j + than the local ``server-config.yaml`` without editing it). +""" + +from __future__ import annotations + +import argparse +import asyncio +import sys + +from context_intelligence_server.config import Neo4jClientConfig, get_settings +from context_intelligence_server.main import build_neo4j_driver +from context_intelligence_server.maintenance import _CONSTRAINT_NAME, _PROBE_CYPHER +from context_intelligence_server.neo4j_store import diagnose, run_repair +from context_intelligence_server.status import SCHEMA_VERSION, SERVER_VERSION + +# Self-declared from -> to (spec sec 5.4). This is a server-version / +# structural-rectification step, NOT a schema_version bump -- WS-3 changes +# no stored node/edge shape, so schema_version stays 1 -> 1. +FROM_SERVER_VERSION = "6.7.x" +TO_SERVER_VERSION = "6.8.0" +FROM_SCHEMA_VERSION = 1 +TO_SCHEMA_VERSION = 1 + +_OK = "\033[32m\u2713\033[0m" # green check +_FAIL = "\033[31m\u2717\033[0m" # red x +_WARN = "\033[33m!\033[0m" # yellow warning + + +def _print_banner() -> None: + print("=" * 72) + print("context-intelligence-server -- out-of-band graph rectification") + print(f" server version : {FROM_SERVER_VERSION} -> {TO_SERVER_VERSION}") + print( + f" schema_version : {FROM_SCHEMA_VERSION} -> {TO_SCHEMA_VERSION} (unchanged -- structural only)" + ) + print( + f" running server reports: version={SERVER_VERSION} schema_version={SCHEMA_VERSION}" + ) + print(" OUT-OF-BAND: never runs at server startup or on the request path.") + print(" IDEMPOTENT: safe to re-run; --apply is a no-op on an already-clean graph.") + print("=" * 72) + + +def _resolve_config(args: argparse.Namespace) -> Neo4jClientConfig: + """Same resolution the server uses (``Settings``), with CLI overrides. + + Base config comes from ``get_settings().resolve_neo4j_admin()`` -- the + identical call ``doctor.py`` and ``lifespan()`` make. ``--neo4j-*`` + flags override individual fields without requiring a + ``server-config.yaml`` edit. + """ + admin = get_settings().resolve_neo4j_admin() + return admin.model_copy( + update={ + k: v + for k, v in ( + ("url", args.neo4j_url), + ("username", args.neo4j_user), + ("password", args.neo4j_password), + ) + if v is not None + } + ) + + +async def _constraint_present(driver: object) -> bool | None: + """Tri-state ``:Node`` uniqueness constraint check. + + Reuses the exact catalog-read query the running server's maintenance + gate probes with (``maintenance._PROBE_CYPHER`` / ``_CONSTRAINT_NAME``) + -- one source of truth for "is the constraint present", never a + second hardcoded copy of the constraint name. + """ + try: + async with driver.session() as session: # type: ignore[attr-defined] + result = await session.run(_PROBE_CYPHER) + count = 0 + async for record in result: + count = record["c"] + return count > 0 + except Exception: # noqa: BLE001 -- connectivity/catalog probe, report unknown + return None + + +def _print_diagnosis( + diagnosis: dict[str, int], constraint_present: bool | None +) -> None: + mark = ( + _OK if constraint_present else (_WARN if constraint_present is False else _FAIL) + ) + label = {True: "present", False: "ABSENT", None: "unknown (probe failed)"}[ + constraint_present + ] + print(f" {mark} :Node uniqueness constraint ({_CONSTRAINT_NAME}): {label}") + + untagged = diagnosis["untagged_nodes"] + mark = _OK if untagged == 0 else _WARN + print(f" {mark} Untagged :Node count: {untagged}") + + duplicates = diagnosis["duplicate_nodes"] + mark = _OK if duplicates == 0 else _WARN + print(f" {mark} Duplicate node count: {duplicates}") + + +async def _run_status(driver: object) -> int: + """Read-only report. Never writes. Exits 0 once Neo4j is reachable and + the report has been printed, regardless of whether rectification is + needed -- ``--status`` reports, it does not judge. + """ + try: + await driver.verify_connectivity() # type: ignore[attr-defined] + except Exception as exc: # noqa: BLE001 + print(f" {_FAIL} Neo4j reachable -- {exc}") + return 1 + print(f" {_OK} Neo4j reachable") + + constraint_present = await _constraint_present(driver) + diagnosis = await diagnose(driver) + _print_diagnosis(diagnosis, constraint_present) + + needs_rectification = ( + constraint_present is not True + or diagnosis["untagged_nodes"] + or diagnosis["duplicate_nodes"] + ) + if needs_rectification: + print( + f" {_WARN} Rectification needed -- re-run with --apply (or POST /admin/maintenance)." + ) + else: + print(f" {_OK} Graph is healthy -- no rectification needed.") + return 0 + + +async def _run_apply(driver: object) -> int: + """Rectify: dedup -> :Node backfill -> constraint create, via the + SAME ``run_repair`` the server's ``/admin/maintenance`` endpoint and + ``doctor --fix`` call. Prints before/after counts. + """ + try: + await driver.verify_connectivity() # type: ignore[attr-defined] + except Exception as exc: # noqa: BLE001 + print(f" {_FAIL} Neo4j reachable -- {exc}") + return 1 + print(f" {_OK} Neo4j reachable") + + print("-- before --") + before = await diagnose(driver) + _print_diagnosis(before, await _constraint_present(driver)) + + print("Rectifying (dedup + :Node backfill + constraint create)...") + result = await run_repair(driver) + print( + f" {_OK} {result['duplicates_removed']} duplicate(s) removed, {result['nodes_tagged']} node(s) tagged :Node." + ) + + print("-- after --") + after = await diagnose(driver) + constraint_present = await _constraint_present(driver) + _print_diagnosis(after, constraint_present) + + if ( + constraint_present is True + and after["untagged_nodes"] == 0 + and after["duplicate_nodes"] == 0 + ): + print(f" {_OK} Graph is healthy after rectification.") + return 0 + print(f" {_FAIL} Graph still has issues after rectification -- see counts above.") + return 1 + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="migrations/run.py", + description="Out-of-band Neo4j graph rectification (never at server startup).", + ) + mode = parser.add_mutually_exclusive_group(required=True) + mode.add_argument( + "--status", + action="store_true", + help="Read-only report (constraint presence, untagged/duplicate counts). Exit 0.", + ) + mode.add_argument( + "--apply", + action="store_true", + help="Rectify: dedup + :Node backfill + constraint create. Idempotent.", + ) + parser.add_argument( + "--neo4j-url", + default=None, + help="Override the Neo4j bolt URL (default: from Settings).", + ) + parser.add_argument( + "--neo4j-user", + default=None, + help="Override the Neo4j username (default: from Settings).", + ) + parser.add_argument( + "--neo4j-password", + default=None, + help="Override the Neo4j password (default: from Settings).", + ) + return parser + + +async def _amain(args: argparse.Namespace) -> int: + config = _resolve_config(args) + driver = build_neo4j_driver(config) + try: + if args.apply: + return await _run_apply(driver) + return await _run_status(driver) + finally: + await driver.close() + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + _print_banner() + return asyncio.run(_amain(args)) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/pyproject.toml b/pyproject.toml index 2aaa7caf..03f3f9a4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "context-intelligence-server" -version = "6.7.0" +version = "6.8.0" description = "Context Intelligence Server for Amplifier" requires-python = ">=3.11" dependencies = [ @@ -33,6 +33,7 @@ testpaths = ["tests"] markers = [ "neo4j: marks tests as requiring a live Neo4j container (deselect with -m 'not neo4j')", "integration: marks tests that exercise live async drain-workers or the full HTTP pipeline (deselect with -m 'not integration')", + "deploy_safe_boot: marks REAL-PROCESS boot tests that launch the gunicorn entrypoint as a subprocess (no Neo4j required; deselect with -m 'not deploy_safe_boot')", ] timeout = 30 diff --git a/scripts/relabel_incomplete_sessions.py b/scripts/relabel_incomplete_sessions.py new file mode 100755 index 00000000..9c1e376c --- /dev/null +++ b/scripts/relabel_incomplete_sessions.py @@ -0,0 +1,665 @@ +#!/usr/bin/env python +"""Maintenance script: one-off backfill removing the stale IncompleteSession +false-positive marker from historical Session nodes (Part 2 of the +IncompleteSession relabel fix). + +NOT product code -- integration-tested in +tests/neo4j/test_relabel_incomplete_sessions.py. Never execute this file as +part of the normal application lifecycle. + +Background (see docs/plans/2026-08-12-incomplete-session-relabel-spec.md, +Part 2, and docs/issues/incomplete-session-mislabeling.md) +------------------------------------------------------------------------ +:IncompleteSession is stamped at session:end only when a Session node has no +type label yet. Forked sub-sessions drain in independent, concurrently +draining per-session queues, so a child's session:end can be processed +BEFORE its session:fork/session:start -- stamping the marker before the real +terminal type arrives. Part 1 (SessionLabelStateMachine.classify(), see +handlers/data_layer_2/session.py) heals this FORWARD: every start/fork +transition now strips a stale IncompleteSession marker the moment it is +processed. Part 1 fixes new/future events. This script (Part 2) is the +one-off backfill for nodes that were ALREADY mislabeled before Part 1 shipped +and will never see another start/fork event to heal them forward. + +POST-DEPLOY GATE +---------------- +Run this script ONLY after Part 1 (the heal-forward classify() fix) is +deployed AND verified live. Running --apply before Part 1 is deployed races +fresh mislabeling: new out-of-order nodes can still be stamped +IncompleteSession by the old code while this script is removing the stale +marker from historical nodes, so a subsequent read could see the marker +reappear on a node this script "fixed." Part 1 must be live first so the +false-positive population stops growing before this one-off cleans up the +existing backlog. + +Selector direction correction (found while writing the Neo4j integration +tests for this script -- NOT what the spec draft's literal Cypher assumed) +-------------------------------------------------------------------------- +The spec draft's B1/B2 Cypher fragments write the linked-event check as +``(s)<-[:SOURCED_FROM]-(:SessionStartEvent)`` -- i.e. SessionStartEvent as +the edge SOURCE, Session as the TARGET. Verifying against the shipping +write path shows the real graph is the other way around: + +* ``SessionHandler._handle_start``/``_handle_fork`` + (handlers/data_layer_2/session.py) call + ``graph.upsert_edge(session_id, data_layer_1_node_id, {"type": "SOURCED_FROM"})`` + -- ``session_id`` is the FIRST (src) argument. +* ``Neo4jGraphStore._edge_merge_cypher`` (neo4j_store.py) emits + ``MERGE (src)-[r:{edge_type}]->(dst)`` with src/dst bound to the first/second + ``upsert_edge`` arguments respectively. +* Every other SOURCED_FROM bridge in the codebase (ToolCall -> ToolPreEvent, + Prompt -> PromptSubmitEvent, etc. -- see tests/handlers/data_layer_1/test_default.py + and tests/integration/test_event_pipeline.py) follows the same convention: + the domain entity is the edge SOURCE, the data_layer_1 event node is the + TARGET. + +So the real graph shape is ``(Session)-[:SOURCED_FROM]->(SessionStartEvent)``, +NOT the reverse. This script's selectors use the code-verified direction +``(s)-[:SOURCED_FROM]->(:SessionStartEvent)`` / +``(s)-[:SOURCED_FROM]->(:SessionForkEvent)`` throughout. A selector built on +the spec draft's literal (reversed) arrow would never match a single real +node and would silently do nothing. + +B1 -- concrete execution shape +------------------------------- +A single-statement, server-side batched REMOVE using +``CALL { ... } IN TRANSACTIONS OF $batch_size ROWS`` (the pattern proven in +scripts/tag_legacy_pooled_iterations.py -- NOT the per-node Python loop of +scripts/repair_dual_labels.py). The false-positive selector +(_FALSE_POSITIVE_MATCH below) is the ONE module constant reused by the +diagnostic, count, sample, collect, and apply queries -- one definition of +"false positive." ``apply_relabel()`` additionally collects the node_ids it +actually touched (for the printed "removed N node(s)" report) via ``RETURN +collect(s.node_id)`` after the batched CALL block -- the same "outer +variable survives the per-batch commits" technique +``tag_legacy_pooled_iterations.py`` uses for its ``RETURN count(i)``, just +projecting ids instead of a count. This is a REPORTING count only: the B4 +undo-log is sourced from a separate, EARLIER, read-only collection -- see B4 +below for why. + +B2 -- read-only reconciliation diagnostic, REQUIRED before --apply +-------------------------------------------------------------------- +The selector's core assumption -- "a linked SessionStartEvent/SessionForkEvent +implies the terminal label is already set" -- holds for today's handler code +but is UNVERIFIED for historical nodes written before this fix existed. So +--apply is hard-gated on a read-only diagnostic (``diagnostic_report()``) +that this script always runs first: + +* ``linked_but_untyped`` -- IncompleteSession nodes with a linked start/fork + event but NO terminal label. This is the assumption's blind spot. If this + is > 0, the assumption is FALSIFIED on this DB and --apply REFUSES (exit 1, + no write), printing the count and up to 20 sample node_ids. The operator + must reconcile (or narrow the selector to the terminal-label clause only) + before re-running --apply. +* ``typed_but_unlinked`` -- IncompleteSession nodes with a terminal label but + no linked start/fork event at all. Informational only; does not gate. + +B3 -- this banner (see POST-DEPLOY GATE above). + +B4 -- lightweight rollback +--------------------------- +--apply writes a plain JSON undo-log (--undo-log PATH, default a timestamped +path in the current directory) BEFORE it mutates the graph. The log is +sourced from ``collect_false_positive_ids()`` -- a read-only, pre-mutation +collection of the FULL false-positive candidate set (same _FALSE_POSITIVE_MATCH +selector as everything else) -- NOT from the ids ``apply_relabel()`` reports +as touched. This ordering matters: ``apply_relabel()``'s batched +``CALL { ... } IN TRANSACTIONS OF N ROWS`` COMMITS PER BATCH, so a mid-run +crash can leave some REMOVEs already durable in Neo4j. Writing the undo-log +from the pre-mutation candidate set means that even a crash after batch 1 of +N still leaves a COMPLETE undo-log on disk -- there is no auditability +window where a committed removal has no undo record anywhere. See +``run_apply()`` for the inline safety argument for why logging the (superset) +candidate set instead of the (subset) touched set is safe. The file's +header records the Neo4j host and an ISO-8601 generation timestamp. +--restore PATH re-adds :IncompleteSession to exactly those node_ids via the +same batched ``CALL { ... } IN TRANSACTIONS OF N ROWS`` shape. No +edge/temporal snapshot is needed: this is one non-destructive label REMOVE, +fully recomputable from the node_id list alone. + +B5 -- acceptance check +------------------------ +Both --dry-run and --apply print a population summary before (and --apply +also after) any write: total :IncompleteSession count, and the +genuine-incomplete count (:IncompleteSession AND NOT linked to any +start/fork event at all -- the true, un-mislabeled data-loss population this +script must never touch). This lets the operator see the population move. + +"Only once" guarantees (both hold, no version machinery) +---------------------------------------------------------- +1. Out-of-band -- standalone script, run manually post-upgrade; NEVER at + server startup, and NOT triggered by any product code path. +2. Idempotent -- the false-positive selector's WHERE clause leads with + ``s:IncompleteSession``, so once a node's marker is removed it no longer + matches; re-running --apply against an already-healed graph matches (and + touches) zero rows. Exactly-once degrades safely to at-least-once. + +Leaves untouched the genuine ~0.5% (no linked start/fork event at all -- +these carry no evidence that the labeling race, rather than a real data-loss +event, produced the marker). + +SCHEMA_VERSION is unchanged (this is a data backfill, not a schema change). + +Modes +----- +--dry-run (default) + Read-only report: population summary, B2 diagnostic (informational + here), and the count + a bounded sample (first 20 node_ids) of what + --apply would touch. Writes NOTHING. Always exits 0 (health-check, not + a gate). + +--apply + Runs the B2 diagnostic first. If ``linked_but_untyped`` > 0, REFUSES + (exit 1, no write). Otherwise runs the batched REMOVE + (``apply_relabel``), writes the B4 undo-log, and prints the before/after + population summary (B5). + +--restore PATH + Reads a B4 undo-log written by a previous --apply and re-adds + :IncompleteSession to exactly those node_ids. + +Connection +---------- +Connection details come from :func:`context_intelligence_server.config.get_settings`. +Resolution order (highest first): + +1. Environment variables with prefix ``AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_``. +2. ``server-config.yaml`` in the current working directory. +3. Built-in defaults. + +--neo4j-url/--neo4j-user/--neo4j-password override the resolved settings. +This script is standalone: it connects to Neo4j directly via the ``neo4j`` +driver and does NOT import or start the FastAPI application. + +Exit codes +---------- +* 0 -- --dry-run (always); --apply completed (whether or not any node + needed healing); --restore completed. +* 1 -- --apply refused by the B2 gate (``linked_but_untyped`` > 0); or + --restore could not read the given undo-log file. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from datetime import datetime, timezone +from typing import Any +from urllib.parse import urlparse + +from context_intelligence_server.config import get_settings +from neo4j import GraphDatabase + +DEFAULT_BATCH_SIZE = 500 +DEFAULT_SAMPLE_LIMIT = 20 + +# --------------------------------------------------------------------------- +# Selector fragments -- the ONE place each predicate is defined. Every +# diagnostic / count / sample / apply query below is built from these same +# fragments, so there is exactly one definition of "has a terminal label" and +# "has a linked start/fork event." Both fragments are only ever embedded +# inside a WHERE clause (standard Cypher pattern-predicate usage) -- never +# used as a bare expression inside RETURN/CASE. +# --------------------------------------------------------------------------- + +# A Session node carries a real terminal type. +_HAS_TERMINAL_LABEL = "(s:RootSession OR s:SubSession OR s:ForkedSession)" + +# A Session node has its own linked SessionStartEvent/SessionForkEvent. +# Direction verified against the shipping write path -- see "Selector +# direction correction" in the module docstring above. +_HAS_LINKED_START_OR_FORK = ( + "( (s)-[:SOURCED_FROM]->(:SessionStartEvent) " + "OR (s)-[:SOURCED_FROM]->(:SessionForkEvent) )" +) + +# B1 -- the ONE false-positive selector. An IncompleteSession node is a +# false positive if it already carries a real terminal label, OR has its own +# linked start/fork event. This is the exact set --apply removes the stale +# marker from. ``WHERE s:IncompleteSession`` leads the predicate, which is +# also what makes the operation idempotent: once removed, a node no longer +# matches (see "Only once guarantees" above). +_FALSE_POSITIVE_MATCH = ( + "MATCH (s:Session) WHERE s:IncompleteSession " + f"AND ( {_HAS_TERMINAL_LABEL} OR {_HAS_LINKED_START_OR_FORK} )" +) + + +# --------------------------------------------------------------------------- +# Read-only helpers +# --------------------------------------------------------------------------- + + +def population_summary(session) -> dict[str, int]: + """B5 acceptance check: total IncompleteSession count + genuine-incomplete count. + + genuine_incomplete = IncompleteSession AND NOT linked to any + SessionStartEvent/SessionForkEvent at all -- the true data-loss + population this fix must never touch. + """ + total = session.run( + "MATCH (s:Session) WHERE s:IncompleteSession RETURN count(s) AS n" + ).single()["n"] + + genuine = session.run( + "MATCH (s:Session) WHERE s:IncompleteSession " + f"AND NOT {_HAS_LINKED_START_OR_FORK} " + "RETURN count(s) AS n" + ).single()["n"] + + return {"total": total or 0, "genuine_incomplete": genuine or 0} + + +def diagnostic_report(session) -> dict[str, Any]: + """B2 read-only reconciliation diagnostic. No writes. + + Returns: + { + "linked_but_untyped": int, + "linked_but_untyped_samples": list[str], # up to 20 node_ids + "typed_but_unlinked": int, + } + + linked_but_untyped is the assumption's blind spot: an IncompleteSession + node with a linked start/fork event but NO terminal label yet. If this + is > 0 anywhere in the graph, the selector's core assumption is + FALSIFIED on this DB and --apply must refuse. + + typed_but_unlinked is informational only (never gates --apply): an + IncompleteSession node with a terminal label but no linked start/fork + event at all. + """ + linked_but_untyped_rows = session.run( + "MATCH (s:Session) WHERE s:IncompleteSession " + f"AND {_HAS_LINKED_START_OR_FORK} AND NOT {_HAS_TERMINAL_LABEL} " + "RETURN s.node_id AS node_id" + ) + linked_but_untyped_ids = [row["node_id"] for row in linked_but_untyped_rows] + + typed_but_unlinked = ( + session.run( + "MATCH (s:Session) WHERE s:IncompleteSession " + f"AND {_HAS_TERMINAL_LABEL} AND NOT {_HAS_LINKED_START_OR_FORK} " + "RETURN count(s) AS n" + ).single()["n"] + or 0 + ) + + return { + "linked_but_untyped": len(linked_but_untyped_ids), + "linked_but_untyped_samples": linked_but_untyped_ids[:DEFAULT_SAMPLE_LIMIT], + "typed_but_unlinked": typed_but_unlinked, + } + + +def count_false_positives(session) -> int: + """Count of IncompleteSession nodes the B1 selector matches (i.e. how + many --apply would touch).""" + return ( + session.run(f"{_FALSE_POSITIVE_MATCH} RETURN count(s) AS n").single()["n"] or 0 + ) + + +def sample_false_positives(session, limit: int = DEFAULT_SAMPLE_LIMIT) -> list[str]: + """Bounded sample of node_ids the B1 selector matches. + + ``limit`` is always an internal int constant (never user-controlled + text), so it is safe to interpolate directly into the LIMIT clause. + """ + rows = session.run( + f"{_FALSE_POSITIVE_MATCH} RETURN s.node_id AS node_id LIMIT {limit}" + ) + return [row["node_id"] for row in rows] + + +def collect_false_positive_ids(session) -> list[str]: + """Read-only collection of the FULL (unbounded) set of node_ids the B1 + selector matches -- i.e. every candidate --apply would touch. + + This is the pre-mutation read ``run_apply`` uses to source the B4 + undo-log BEFORE calling ``apply_relabel`` (see "undo-log-before-mutation" + in ``run_apply``'s docstring for the ordering rationale). Unlike + ``sample_false_positives`` (bounded by ``DEFAULT_SAMPLE_LIMIT``, for + human-readable reporting), this has no LIMIT -- it must capture every + candidate so the undo-log is complete even if a subsequent crash + prevents ``apply_relabel`` from reporting its own touched set. + """ + result = session.run(f"{_FALSE_POSITIVE_MATCH} RETURN collect(s.node_id) AS ids") + return list(result.single()["ids"] or []) + + +# --------------------------------------------------------------------------- +# Mutating operations +# --------------------------------------------------------------------------- + + +def apply_relabel(session, batch_size: int = DEFAULT_BATCH_SIZE) -> list[str]: + """B1 -- the single-statement, server-side batched REMOVE. + + Strips :IncompleteSession from every node matched by + _FALSE_POSITIVE_MATCH, batched via + ``CALL { ... } IN TRANSACTIONS OF $batch_size ROWS`` (the pattern in + scripts/tag_legacy_pooled_iterations.py). Returns the node_ids actually + touched -- needed for the B4 undo-log. + + Idempotent: because the outer MATCH re-requires ``s:IncompleteSession``, + a node healed by a previous run no longer matches; re-running this + function against an already-healed graph returns []. + + This function performs NO gate check. Callers that must honor the B2 + reconciliation gate use ``run_apply()``, not this function directly. + """ + result = session.run( + f"{_FALSE_POSITIVE_MATCH} " + "CALL { WITH s REMOVE s:IncompleteSession } IN TRANSACTIONS OF $batch_size ROWS " + "RETURN collect(s.node_id) AS touched_ids", + batch_size=batch_size, + ) + return list(result.single()["touched_ids"] or []) + + +def restore_ids( + session, node_ids: list[str], batch_size: int = DEFAULT_BATCH_SIZE +) -> int: + """B4 --restore: re-add :IncompleteSession to exactly the given node_ids. + + Uses the same batched ``CALL { ... } IN TRANSACTIONS OF N ROWS`` shape as + apply_relabel. Returns the number of nodes actually restored (node_ids + that still exist in the graph). + """ + result = session.run( + "UNWIND $node_ids AS nid " + "MATCH (s:Session {node_id: nid}) " + "CALL { WITH s SET s:IncompleteSession } IN TRANSACTIONS OF $batch_size ROWS " + "RETURN count(s) AS restored", + node_ids=node_ids, + batch_size=batch_size, + ) + return result.single()["restored"] or 0 + + +# --------------------------------------------------------------------------- +# Undo-log (B4) +# --------------------------------------------------------------------------- + + +def _neo4j_host(url: str) -> str: + """Return a host-only fragment of a Neo4j URL for the undo-log header.""" + return urlparse(url).hostname or url + + +def default_undo_log_path() -> str: + """Default timestamped undo-log path, used when --undo-log is omitted.""" + ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + return f"relabel_incomplete_sessions_undo_{ts}.json" + + +def write_undo_log(path: str, neo4j_url: str, node_ids: list[str]) -> None: + """Write a B4 undo-log: header (Neo4j host + ISO timestamp) + touched node_ids.""" + payload = { + "neo4j_host": _neo4j_host(neo4j_url), + "generated_at": datetime.now(timezone.utc).isoformat(), + "node_ids": node_ids, + } + with open(path, "w", encoding="utf-8") as fh: + json.dump(payload, fh, indent=2) + + +def read_undo_log(path: str) -> dict[str, Any]: + """Read a B4 undo-log written by write_undo_log.""" + with open(path, encoding="utf-8") as fh: + return json.load(fh) + + +# --------------------------------------------------------------------------- +# Reporting / orchestration +# --------------------------------------------------------------------------- + + +def run_dry_run(session) -> int: + """Read-only report. Writes nothing. Always returns 0 (health-check, + not a gate).""" + before = population_summary(session) + print("DRY RUN -- relabel_incomplete_sessions.py (no writes)\n") + print("POPULATION SUMMARY:") + print(f" total IncompleteSession : {before['total']}") + print(f" genuine_incomplete (no link) : {before['genuine_incomplete']}\n") + + diag = diagnostic_report(session) + print("RECONCILIATION DIAGNOSTIC (B2 -- gates --apply; informational here):") + print(f" linked_but_untyped : {diag['linked_but_untyped']}") + print(f" typed_but_unlinked : {diag['typed_but_unlinked']} (informational only)") + if diag["linked_but_untyped"]: + print("\n --apply would REFUSE (see module docstring / B2). Sample node_ids:") + for node_id in diag["linked_but_untyped_samples"]: + print(f" - {node_id}") + print() + + would_touch = count_false_positives(session) + sample = sample_false_positives(session) + print( + f"WOULD APPLY -- {would_touch} node(s) would have :IncompleteSession removed." + ) + if sample: + print(f"Sample node_ids (up to {DEFAULT_SAMPLE_LIMIT}):") + for node_id in sample: + print(f" - {node_id}") + return 0 + + +def run_apply(session, batch_size: int, undo_log_path: str, neo4j_url: str) -> int: + """B2-gated --apply: diagnostic first, write only if the gate is clear. + + 1. Print the BEFORE population summary (B5). + 2. Run the B2 reconciliation diagnostic. If linked_but_untyped > 0, + REFUSE (no write): print the count and up to 20 sample node_ids, + return 1. + 3. Otherwise: collect the FULL false-positive candidate set (read-only), + write the B4 undo-log from THAT set, THEN run apply_relabel() to + mutate, print the AFTER population summary, return 0. + + undo-log-before-mutation (W-1 fix) + ----------------------------------- + Step 3 writes the undo-log BEFORE calling apply_relabel(), not after. + apply_relabel()'s batched ``CALL { ... } IN TRANSACTIONS OF N ROWS`` + COMMITS PER BATCH -- each batch's REMOVE is durable in Neo4j the moment + that batch completes, well before apply_relabel() returns. If the + process crashed mid-run under the OLD ordering (mutate first, log + after), some REMOVEs would already be committed with NO undo record + anywhere -- an auditability hole. Logging the pre-mutation candidate set + first closes that window: even a crash after the very first batch still + leaves a COMPLETE undo-log on disk. + + Why logging the (pre-mutation) candidate set instead of apply_relabel's + (post-mutation) touched set is safe: + - candidate_ids is collected via the exact same _FALSE_POSITIVE_MATCH + selector apply_relabel() uses, moments before apply_relabel() runs + -- so candidate_ids is a SUPERSET of (on a clean run, EQUAL to) + whatever apply_relabel() actually removes. + - restore_ids() (the --restore consumer of this log) is idempotent + over a superset: it SETs :IncompleteSession on each node_id that + still exists. For a node that WAS removed, this correctly restores + the marker. For a candidate that -- for any reason -- was NOT + removed (still carries :IncompleteSession), the SET is a no-op: the + label is already present. So restoring from the candidate superset + is always safe -- it never incorrectly re-adds a marker that + shouldn't exist, and it never fails to restore a node that WAS + removed. + - On a clean run (the common case, and the only case this script's + idempotent selector allows in practice) candidate_ids == the ids + apply_relabel() reports touched, so the printed "removed N node(s)" + count is UNCHANGED by this reordering -- only the undo-log's source + and its write timing move earlier. + """ + before = population_summary(session) + print("POPULATION SUMMARY (before):") + print(f" total IncompleteSession : {before['total']}") + print(f" genuine_incomplete (no link) : {before['genuine_incomplete']}\n") + + diag = diagnostic_report(session) + print("RECONCILIATION DIAGNOSTIC (B2):") + print(f" linked_but_untyped : {diag['linked_but_untyped']}") + print(f" typed_but_unlinked : {diag['typed_but_unlinked']} (informational only)\n") + + if diag["linked_but_untyped"] > 0: + print( + "APPLY REFUSED -- the selector's assumption (a linked " + "SessionStartEvent/SessionForkEvent implies the terminal label is " + f"already set) does NOT hold on this DB: {diag['linked_but_untyped']} " + "node(s) are linked but carry no terminal label.\n" + ) + print("Sample node_ids (up to 20):") + for node_id in diag["linked_but_untyped_samples"]: + print(f" - {node_id}") + print( + "\nReconcile these nodes (or narrow the selector to the " + "terminal-label clause only) before re-running --apply." + ) + return 1 + + # Read-only pre-mutation capture, THEN write the undo-log, THEN mutate. + # See the "undo-log-before-mutation (W-1 fix)" note in this function's + # docstring for why this order (and logging the candidate superset + # rather than apply_relabel's touched subset) is safe. + candidate_ids = collect_false_positive_ids(session) + write_undo_log(undo_log_path, neo4j_url, candidate_ids) + print( + f"Undo log written to {undo_log_path} ({len(candidate_ids)} node_id(s), " + "pre-mutation candidate set).\n" + ) + + touched_ids = apply_relabel(session, batch_size) + print(f"APPLIED -- removed :IncompleteSession from {len(touched_ids)} node(s).\n") + + after = population_summary(session) + print("POPULATION SUMMARY (after):") + print(f" total IncompleteSession : {after['total']}") + print(f" genuine_incomplete (no link) : {after['genuine_incomplete']}") + return 0 + + +def run_restore(session, path: str, batch_size: int) -> int: + """B4 --restore: re-add :IncompleteSession to exactly the ids in *path*.""" + try: + snap = read_undo_log(path) + except (OSError, json.JSONDecodeError) as exc: + print(f"RESTORE FAILED -- could not read undo-log {path}: {exc}") + return 1 + + node_ids = snap.get("node_ids", []) + restored = restore_ids(session, node_ids, batch_size) + print( + f"RESTORED -- re-added :IncompleteSession to {restored}/{len(node_ids)} " + f"node(s) from {path} (generated_at={snap.get('generated_at')})." + ) + return 0 + + +# --------------------------------------------------------------------------- +# Entry point +# --------------------------------------------------------------------------- + + +def main() -> int: + """Parse arguments, connect to Neo4j, and run the requested mode.""" + parser = argparse.ArgumentParser( + prog="relabel_incomplete_sessions.py", + description=( + "One-off maintenance tool (Part 2 of the IncompleteSession relabel " + "fix): remove the stale :IncompleteSession false-positive marker " + "from historical Session nodes that already carry a real terminal " + "label or a linked SessionStartEvent/SessionForkEvent. " + "POST-DEPLOY GATE -- run only after Part 1 (heal-forward classify()) " + "is deployed and verified live. --apply is itself gated on a " + "read-only reconciliation diagnostic (B2); see module docstring." + ), + ) + parser.add_argument( + "--dry-run", + action="store_true", + help=( + "Read-only report: population summary, B2 diagnostic, and the " + "count + bounded sample of what --apply would touch. Writes " + "nothing. This is the default mode." + ), + ) + parser.add_argument( + "--apply", + action="store_true", + help=( + "Run the B2 diagnostic; if clear, remove :IncompleteSession from " + "the false-positive set (batched, idempotent), write a B4 " + "undo-log, and print the before/after population summary." + ), + ) + parser.add_argument( + "--restore", + metavar="PATH", + default=None, + help="Re-add :IncompleteSession to exactly the node_ids in a B4 undo-log file.", + ) + parser.add_argument( + "--undo-log", + metavar="PATH", + default=None, + help=( + "Path to write the B4 undo-log (used with --apply). Defaults to " + "a timestamped path in the current directory." + ), + ) + parser.add_argument( + "--batch-size", + type=int, + default=DEFAULT_BATCH_SIZE, + metavar="N", + help=f"Rows per transaction for --apply/--restore (default: {DEFAULT_BATCH_SIZE}).", + ) + parser.add_argument( + "--neo4j-url", + metavar="URL", + default=None, + help="Neo4j Bolt URL (overrides server-config.yaml / env var)", + ) + parser.add_argument( + "--neo4j-user", + metavar="USER", + default=None, + help="Neo4j username (overrides server-config.yaml / env var)", + ) + parser.add_argument( + "--neo4j-password", + metavar="PW", + default=None, + help="Neo4j password (overrides server-config.yaml / env var)", + ) + args = parser.parse_args() + + modes_selected = sum(bool(m) for m in (args.dry_run, args.apply, args.restore)) + if modes_selected > 1: + parser.error("--dry-run, --apply, and --restore are mutually exclusive") + + settings = get_settings() + neo4j_url = args.neo4j_url or settings.neo4j_url + neo4j_user = args.neo4j_user or settings.neo4j_user + neo4j_password = args.neo4j_password or settings.neo4j_password + + print(f"Connecting to Neo4j at {neo4j_url} as {neo4j_user}\n") + driver = GraphDatabase.driver(neo4j_url, auth=(neo4j_user, neo4j_password)) + try: + with driver.session() as neo4j_session: + if args.restore: + return run_restore(neo4j_session, args.restore, args.batch_size) + if args.apply: + undo_log_path = args.undo_log or default_undo_log_path() + return run_apply( + neo4j_session, args.batch_size, undo_log_path, neo4j_url + ) + # --dry-run is the default: no writes unless --apply is explicit. + return run_dry_run(neo4j_session) + finally: + driver.close() + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/tag_legacy_pooled_iterations.py b/scripts/tag_legacy_pooled_iterations.py new file mode 100755 index 00000000..39a4fdf9 --- /dev/null +++ b/scripts/tag_legacy_pooled_iterations.py @@ -0,0 +1,374 @@ +#!/usr/bin/env python +"""Maintenance tool: TAG (non-destructive) legacy Iteration nodes that were +confirmed MERGEd across >=2 distinct OrchestratorRuns before the I5 fix. + +**NOT product code -- no unit tests.** This is a standalone, one-off +maintenance script. The regression that PREVENTS new corruption is covered +by ``tests/handlers/data_layer_2/test_iteration.py`` +(``TestIterationRunScopingP21``); this script only deals with historical data +already in a live graph. + +Background (I5 / I3, see CHANGELOG.md) +--------------------------------------- +Before the I5 fix, ``Iteration.node_id`` was the bare shape +``{session_id}::iteration::{N}``, with ``N`` a per-*session* (not per-*run*) +counter. Because the counter could restart (e.g. drainer restart/replay), +two DIFFERENT ``OrchestratorRun``s could reuse the same ``N`` and MERGE onto +the SAME bare-id Iteration node, clobbering its usage/message properties +(last-write-wins). After I5, new Iteration node_ids are run-scoped: +``{session_id}::orch_run::{ts}::iteration::{N}`` -- these can never collide +across runs and are excluded from consideration by this script entirely. + +A **bare-id node_id does NOT mean the node is corrupt.** A council review + +live-data verification established that only nodes whose bare id is reached +by ``HAS_PART`` from **two or more distinct** ``OrchestratorRun`` nodes are +provably corrupt (pooled). The rest of the bare-id population (the large +majority -- ~93-95% of bare-id nodes in live data) are clean, single-run +nodes created before the fix and must NOT be touched: a run=1 bare-id node is +exactly what a healthy pre-fix Iteration looked like. + +What this script does +---------------------- +This script TAGS -- and only tags -- the confirmed-corrupt subset with a +non-destructive marker property ``data_quality = 'legacy_pooled_pre_fix'``. +It never deletes or restructures anything. The destructive cleanup (e.g. +splitting or deleting pooled nodes) is a SEPARATE, gated follow-up and is +explicitly OUT OF SCOPE here. + +CONFIRMED-CORRUPT SELECTOR (the ONLY nodes this script ever touches):: + + MATCH (run:OrchestratorRun)-[:HAS_PART]->(i:Iteration) + WHERE NOT i.node_id CONTAINS '::orch_run::' + WITH i, count(DISTINCT run) AS runs + WHERE runs >= 2 + +Selector-sanity note (does not over-claim) +------------------------------------------- +``runs >= 2`` is a **confirmed lower bound** on corruption, not an exhaustive +enumeration of it. A bare-id node with exactly one surviving ``HAS_PART`` +parent (``runs == 1``) could -- in principle -- have been pooled across MORE +runs whose ``OrchestratorRun`` node or edge was later pruned/expired, leaving +only one parent behind; such a node would be indistinguishable, from current +graph state, from a genuinely single-run node. This script deliberately does +**not** attempt to catch that case: it tags only what is provably corroborated +by graph structure today (>=2 live distinct parents), and explicitly leaves +``runs == 1`` (``confirmed_clean_single_run``) and ``runs == 0`` +(``no_run_edge`` -- no ``HAS_PART`` parent survives at all, so corruption +cannot be confirmed or denied) untouched. + +Nodes this script explicitly does NOT touch +-------------------------------------------- +* ``runs == 1`` -- confirmed clean single-run bare-id node. Leave alone. +* ``runs == 0`` -- no surviving ``OrchestratorRun`` parent edge at all, so + pooling cannot be confirmed. Leave alone (see selector-sanity note above). +* Any run-scoped node (``node_id`` contains ``'::orch_run::'``) -- these are + post-I5 and structurally cannot collide across runs. + +DEPLOYMENT GATE +--------------- +This script is **not a deployment gate**. It performs a single additive, +idempotent ``SET`` of one marker property on a provably-corrupt subset of +historical nodes; it does not race the I5 write-path fix (which governs only +newly-created Iteration nodes) and carries no risk of mixed-type or +mixed-shape state. It may be run at any time, before or after I5 is deployed +to a given server, and re-run as often as desired (see Idempotency below). +Running it is a maintenance convenience, not a precondition for shipping I5. + +Connection +---------- +Connection details come from :func:`context_intelligence_server.config.get_settings`. +Resolution order (highest first): + +1. Environment variables with prefix ``AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_``. +2. ``server-config.yaml`` in the current working directory. +3. Built-in defaults. + +To target a DTU, override env vars:: + + AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_NEO4J_URL=bolt://localhost:7688 \\ + AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_NEO4J_PASSWORD=testpassword \\ + uv run python scripts/tag_legacy_pooled_iterations.py --dry-run + +This script is standalone: it connects to Neo4j directly via the ``neo4j`` +driver and does NOT import or start the FastAPI application. + +Idempotency +----------- +The write query only touches nodes where +``i.data_quality IS NULL OR i.data_quality <> 'legacy_pooled_pre_fix'``, so a +second ``--apply`` run against an already-tagged graph matches zero rows and +performs zero writes. The tag itself is never removed by this script (it is +a permanent, non-destructive marker); un-tagging is out of scope. + +Modes +----- +--dry-run (default) + Read-only report: total bare-id Iteration count, broken down into + ``confirmed_corrupt`` (runs >= 2), ``confirmed_clean_single_run`` + (runs == 1), ``no_run_edge`` (runs == 0), and how many of the + ``confirmed_corrupt`` set are NOT yet tagged (i.e. would be written by + ``--apply``). Writes NOTHING. Always exits 0 (this is a health-check / + reporting mode, not a gate). + +--apply + ``SET i.data_quality = 'legacy_pooled_pre_fix'`` on the confirmed-corrupt + set, batched via ``CALL { ... } IN TRANSACTIONS OF N ROWS``. Only writes + rows not already carrying the tag (idempotent). Prints the number of rows + tagged, then re-runs the untagged-count verification query and reports + it (must be 0 after a successful apply). + +Exit codes +---------- +* 0 -- success (dry-run report printed; or apply completed and verification + found zero confirmed-corrupt nodes remaining untagged). +* 1 -- apply completed but verification found untagged confirmed-corrupt + nodes remaining (should not happen; signals a bug or concurrent writer). +""" + +from __future__ import annotations + +import argparse +import sys + +from context_intelligence_server.config import get_settings +from neo4j import GraphDatabase + +DEFAULT_BATCH_SIZE = 500 + +TAG_VALUE = "legacy_pooled_pre_fix" + +# --------------------------------------------------------------------------- +# Selector -- the ONLY nodes this script ever touches. +# Shared verbatim (as a fragment) across classify/apply/verify so there is +# exactly one place that defines "confirmed corrupt". +# --------------------------------------------------------------------------- +_CONFIRMED_CORRUPT_MATCH = ( + "MATCH (run:OrchestratorRun)-[:HAS_PART]->(i:Iteration) " + "WHERE NOT i.node_id CONTAINS '::orch_run::' " + "WITH i, count(DISTINCT run) AS runs " + "WHERE runs >= 2" +) + +_UNTAGGED_GUARD = ( + "WITH DISTINCT i WHERE i.data_quality IS NULL OR i.data_quality <> $tag_value" +) + + +# --------------------------------------------------------------------------- +# Read-only helpers +# --------------------------------------------------------------------------- + + +def classify(session) -> dict[str, int]: + """Return bucketed counts over ALL bare-id Iteration nodes. No writes. + + Buckets: + - total: every bare-id (pre-fix-shaped) Iteration node + - confirmed_corrupt: runs >= 2 (the ONLY set this script will ever tag) + - confirmed_clean_single_run: runs == 1 (leave alone) + - no_run_edge: runs == 0, i.e. no surviving OrchestratorRun HAS_PART + parent at all (leave alone -- corruption cannot be confirmed) + - confirmed_corrupt_untagged: of confirmed_corrupt, how many are NOT yet + carrying the tag (i.e. how many --apply would write) + """ + result = session.run( + "MATCH (i:Iteration) " + "WHERE NOT i.node_id CONTAINS '::orch_run::' " + "OPTIONAL MATCH (run:OrchestratorRun)-[:HAS_PART]->(i) " + "WITH i, count(DISTINCT run) AS runs " + "RETURN " + " count(i) AS total, " + " sum(CASE WHEN runs >= 2 THEN 1 ELSE 0 END) AS confirmed_corrupt, " + " sum(CASE WHEN runs = 1 THEN 1 ELSE 0 END) AS confirmed_clean_single_run, " + " sum(CASE WHEN runs = 0 THEN 1 ELSE 0 END) AS no_run_edge" + ) + row = result.single() + counts = { + "total": row["total"] or 0, + "confirmed_corrupt": row["confirmed_corrupt"] or 0, + "confirmed_clean_single_run": row["confirmed_clean_single_run"] or 0, + "no_run_edge": row["no_run_edge"] or 0, + } + counts["confirmed_corrupt_untagged"] = count_untagged(session) + return counts + + +def count_untagged(session) -> int: + """Return the count of confirmed-corrupt Iteration nodes NOT yet tagged. + + Standalone re-runnable verification query: after a successful --apply + this must return 0. + """ + result = session.run( + f"{_CONFIRMED_CORRUPT_MATCH} {_UNTAGGED_GUARD} RETURN count(i) AS untagged", + tag_value=TAG_VALUE, + ) + return result.single()["untagged"] or 0 + + +# --------------------------------------------------------------------------- +# Mutating operation +# --------------------------------------------------------------------------- + + +def tag_confirmed_corrupt(session, batch_size: int) -> int: + """SET data_quality='legacy_pooled_pre_fix' on the confirmed-corrupt, + not-yet-tagged subset, batched via CALL { ... } IN TRANSACTIONS OF N ROWS. + + Idempotent: rows already carrying the tag are excluded by the guard + before the CALL, so re-running matches (and writes) zero rows. + + Returns the number of rows tagged. + """ + result = session.run( + f"{_CONFIRMED_CORRUPT_MATCH} {_UNTAGGED_GUARD} " + "CALL { " + " WITH i " + " SET i.data_quality = $tag_value " + "} IN TRANSACTIONS OF $batch_size ROWS " + "RETURN count(i) AS tagged", + tag_value=TAG_VALUE, + batch_size=batch_size, + ) + return result.single()["tagged"] or 0 + + +# --------------------------------------------------------------------------- +# Reporting +# --------------------------------------------------------------------------- + + +def run_dry_run(session) -> int: + """Print classification report. Always returns 0 (health-check, not a gate).""" + counts = classify(session) + print("DRY RUN -- confirmed-corrupt Iteration nodes (TAG ONLY, no writes):\n") + print(f"{'Bucket':<35} {'Count':>10}") + print("-" * 46) + print(f"{'total (all bare-id Iterations)':<35} {counts['total']:>10}") + print(f"{'confirmed_corrupt (runs >= 2)':<35} {counts['confirmed_corrupt']:>10}") + print( + f"{'confirmed_clean_single_run (runs == 1)':<35} " + f"{counts['confirmed_clean_single_run']:>10}" + ) + print(f"{'no_run_edge (runs == 0)':<35} {counts['no_run_edge']:>10}") + print("-" * 46) + print( + f"{'confirmed_corrupt NOT yet tagged':<35} " + f"{counts['confirmed_corrupt_untagged']:>10}" + ) + print( + f"\n --apply would tag {counts['confirmed_corrupt_untagged']} node(s). " + f"{counts['confirmed_clean_single_run'] + counts['no_run_edge']} bare-id " + "node(s) are left untouched (not confirmed corrupt).\n" + ) + return 0 + + +def run_apply(session, batch_size: int) -> int: + """Tag the confirmed-corrupt set, then verify zero remain untagged. + + Returns the process exit code (0 = verified clean, 1 = residual untagged). + """ + print("APPLY -- tagging confirmed-corrupt Iteration nodes:\n") + tagged = tag_confirmed_corrupt(session, batch_size) + print(f" Tagged {tagged} node(s) with data_quality='{TAG_VALUE}'.\n") + + remaining = count_untagged(session) + print("VERIFICATION -- confirmed-corrupt nodes still missing the tag:\n") + print(f" remaining untagged: {remaining}") + if remaining: + print( + "\nAPPLY INCOMPLETE -- " + f"{remaining} confirmed-corrupt node(s) remain untagged after apply. " + "This should not happen; investigate for a concurrent writer or a bug." + ) + return 1 + print("\nAPPLY COMPLETE -- all confirmed-corrupt nodes are tagged.") + return 0 + + +# --------------------------------------------------------------------------- +# Entry point +# --------------------------------------------------------------------------- + + +def main() -> int: + """Parse arguments, connect to Neo4j, and run dry-run or apply.""" + parser = argparse.ArgumentParser( + prog="tag_legacy_pooled_iterations.py", + description=( + "Maintenance tool: non-destructively TAG (data_quality=" + f"'{TAG_VALUE}') legacy bare-id Iteration nodes confirmed pooled " + "across >=2 distinct OrchestratorRuns before the I5 fix. Never " + "touches run-scoped nodes, single-run bare-id nodes, or bare-id " + "nodes with no surviving run parent. The destructive cleanup is " + "a separate, gated follow-up -- out of scope here." + ), + ) + parser.add_argument( + "--dry-run", + action="store_true", + help=( + "Report bucketed counts (confirmed_corrupt / " + "confirmed_clean_single_run / no_run_edge) and how many " + "confirmed-corrupt nodes would be tagged. Writes nothing. " + "This is the default mode." + ), + ) + parser.add_argument( + "--apply", + action="store_true", + help=( + "Tag the confirmed-corrupt set (batched, idempotent), then " + "verify zero remain untagged." + ), + ) + parser.add_argument( + "--batch-size", + type=int, + default=DEFAULT_BATCH_SIZE, + metavar="N", + help=f"Rows per transaction for --apply (default: {DEFAULT_BATCH_SIZE}).", + ) + parser.add_argument( + "--neo4j-url", + metavar="URL", + default=None, + help="Neo4j Bolt URL (overrides server-config.yaml / env var)", + ) + parser.add_argument( + "--neo4j-user", + metavar="USER", + default=None, + help="Neo4j username (overrides server-config.yaml / env var)", + ) + parser.add_argument( + "--neo4j-password", + metavar="PW", + default=None, + help="Neo4j password (overrides server-config.yaml / env var)", + ) + args = parser.parse_args() + + if args.apply and args.dry_run: + parser.error("--dry-run and --apply are mutually exclusive") + + settings = get_settings() + neo4j_url = args.neo4j_url or settings.neo4j_url + neo4j_user = args.neo4j_user or settings.neo4j_user + neo4j_password = args.neo4j_password or settings.neo4j_password + + print(f"Connecting to Neo4j at {neo4j_url} as {neo4j_user}\n") + driver = GraphDatabase.driver(neo4j_url, auth=(neo4j_user, neo4j_password)) + try: + with driver.session() as neo_session: + if args.apply: + return run_apply(neo_session, args.batch_size) + # --dry-run is the default: no writes unless --apply is explicit. + return run_dry_run(neo_session) + finally: + driver.close() + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/conftest.py b/tests/conftest.py index ecfd94d9..088581e6 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -178,6 +178,34 @@ def reset_registry() -> Generator[None, None, None]: registry._write_semaphore = None +@pytest.fixture(autouse=True) +def reset_maintenance_coordinator() -> Generator[None, None, None]: + """Ensure each test starts with (and leaves) a pristine MaintenanceCoordinator. + + ``context_intelligence_server.maintenance.coordinator`` is a process-wide + singleton shared by ``registry.py``'s drain-loop gate and ``main.py``'s + HTTP gate/status. Any test that runs the REAL ``lifespan()`` context + manager (there are many, pre-dating WS-3a) now ALSO calls + ``coordinator.bind_driver(...)`` against whatever mock Neo4j driver that + test configured -- often a driver mock whose canned responses were never + designed to answer the maintenance constraint-probe query. Without this + reset, that leaks into the coordinator's TTL-cached probe/op state and + can spuriously close the gate for every OTHER test in the session (test + pollution via a shared singleton). Reset by copying in a fresh instance's + attributes rather than replacing the object, since other modules hold a + direct reference to THIS object (``from ...maintenance import coordinator``). + """ + from context_intelligence_server.maintenance import MaintenanceCoordinator + from context_intelligence_server.maintenance import coordinator as _coordinator + + def _reset() -> None: + _coordinator.__dict__.update(vars(MaintenanceCoordinator())) + + _reset() + yield + _reset() + + @pytest.fixture async def client() -> AsyncGenerator[httpx.AsyncClient, None]: async with httpx.AsyncClient( diff --git a/tests/handlers/data_layer_2/test_iteration.py b/tests/handlers/data_layer_2/test_iteration.py index 77512c4f..ca5bc709 100644 --- a/tests/handlers/data_layer_2/test_iteration.py +++ b/tests/handlers/data_layer_2/test_iteration.py @@ -3,10 +3,15 @@ Covers: - handled_events == frozenset({'provider:request', 'llm:request', 'llm:response'}) - provider:request creates Iteration:SST_EVENT node keyed as - '{session_id}::iteration::{iteration_number}' with session_id, iteration_number, - and started_at; sets active_iteration_id cursor + '{session_id}::iteration::{iteration_number}' (no active orchestrator run) or + '{session_id}::orch_run::{execution_start_ts}::iteration::{iteration_number}' + (P2.1 fix, run-scoped when execution_start_ts is set) with session_id, + iteration_number, and started_at; sets active_iteration_id cursor - E06: OrchestratorRun -[:HAS_PART {sst_semantic: 'CONTAINS'}]-> Iteration - created when execution_start_ts cursor is set; NOT created when None + created when execution_start_ts cursor is set (target is the run-scoped + iteration_id); NOT created when None +- P2.1: two iterations sharing the same iteration_number under DIFFERENT + orchestrator runs get DISTINCT node_ids (no cross-run collision) - llm:request enriches active Iteration with provider, model, message_count, has_system; noop when active_iteration_id is None - llm:response enriches active Iteration with usage_input, usage_output, usage_cache_write; @@ -16,11 +21,12 @@ from __future__ import annotations +import logging + from context_intelligence_server.handlers.data_layer_2.iteration import IterationHandler from context_intelligence_server.services import HookStateService from context_intelligence_server.utils import make_node_id - # --------------------------------------------------------------------------- # 1. TestIterationHandlerHandledEvents # --------------------------------------------------------------------------- @@ -172,7 +178,10 @@ class TestE06HasPartEdge: async def test_e06_has_part_edge_created_when_execution_start_ts_is_set( self, services: HookStateService ) -> None: - """E06 edge must be created when execution_start_ts cursor is set before provider:request.""" + """E06 edge must be created when execution_start_ts cursor is set before provider:request. + + P2.1: the edge target is now the run-scoped iteration_id, not the bare shape. + """ handler = IterationHandler(services) # Simulate that execution:start previously fired and set the cursor services.data_layer_2.execution_start_ts = "2026-01-01T00:00:00Z" @@ -185,7 +194,7 @@ async def test_e06_has_part_edge_created_when_execution_start_ts_is_set( }, ) orch_run_id = "s1::orch_run::2026-01-01T00:00:00Z" - iteration_id = "s1::iteration::1" + iteration_id = "s1::orch_run::2026-01-01T00:00:00Z::iteration::1" edge = await services.graph.get_edge(orch_run_id, iteration_id) assert edge is not None, ( f"E06 HAS_PART edge from '{orch_run_id}' to '{iteration_id}' must exist " @@ -197,6 +206,9 @@ async def test_e06_has_part_edge_created_when_execution_start_ts_is_set( assert edge.get("sst_semantic") == "CONTAINS", ( f"E06 edge must have sst_semantic='CONTAINS'. Got: {edge.get('sst_semantic')}" ) + assert services.data_layer_2.active_iteration_id == iteration_id, ( + "active_iteration_id cursor must be set to the run-scoped iteration_id" + ) async def test_e06_not_created_when_execution_start_ts_is_none( self, services: HookStateService @@ -218,6 +230,111 @@ async def test_e06_not_created_when_execution_start_ts_is_none( f"Only SOURCED_FROM edge should exist when execution_start_ts is None. " f"Got {len(services.graph._edges)} edges: {list(services.graph._edges.keys())}" ) + # Falls back to the bare (pre-fix-shaped) id when no orchestrator run is active + assert services.data_layer_2.active_iteration_id == "s1::iteration::1" + + +# --------------------------------------------------------------------------- +# 3b. TestIterationRunScopingP21 +# --------------------------------------------------------------------------- + + +class TestIterationRunScopingP21: + """P2.1 (I5/I3 fix): Iteration node_id is run-scoped and does not collide across runs.""" + + async def test_same_iteration_number_under_different_runs_gets_distinct_node_ids( + self, services: HookStateService + ) -> None: + """Two iterations sharing iteration_number=1 under DIFFERENT orchestrator runs + must produce DISTINCT node_ids and both nodes must independently exist. + + This reproduces the real-world collision: iteration_count is a per-session + (not per-run) counter, so after a drainer restart/replay recreates + DataLayer2State the counter can restart from zero and reproduce a prior + run's iteration_number under a NEW orchestrator run. Before the P2.1 fix, + both runs' first iteration would MERGE onto the bare + 's1::iteration::1' node_id. After the fix, each run's iteration_number=1 is + prefixed with its own orch_run_id and the two nodes stay distinct. + """ + handler = IterationHandler(services) + + # --- Run 1: execution_start_ts = t1, iteration_count increments 0 -> 1 + services.data_layer_2.execution_start_ts = "2026-01-01T00:00:00Z" + await handler( + "provider:request", + {"session_id": "s1", "timestamp": "2026-01-01T00:00:01Z"}, + ) + run1_iteration_id = services.data_layer_2.active_iteration_id + + # --- Simulate a drainer restart: a fresh DataLayer2State would have + # iteration_count=0 again. Reproduce that here directly (this is exactly + # what a freshly (re)constructed HookStateService looks like), then a + # genuinely NEW orchestrator run begins with a different timestamp. + services.data_layer_2.iteration_count = 0 + services.data_layer_2.execution_start_ts = "2026-01-02T00:00:00Z" + await handler( + "provider:request", + {"session_id": "s1", "timestamp": "2026-01-02T00:00:01Z"}, + ) + run2_iteration_id = services.data_layer_2.active_iteration_id + + assert run1_iteration_id == "s1::orch_run::2026-01-01T00:00:00Z::iteration::1" + assert run2_iteration_id == "s1::orch_run::2026-01-02T00:00:00Z::iteration::1" + assert run1_iteration_id != run2_iteration_id, ( + "Iteration node_ids for the same iteration_number under different " + "orchestrator runs must be distinct (no cross-run collision)." + ) + + node1 = await services.graph.get_node(run1_iteration_id) + node2 = await services.graph.get_node(run2_iteration_id) + assert node1 is not None, ( + f"Run 1's Iteration node '{run1_iteration_id}' must exist" + ) + assert node2 is not None, ( + f"Run 2's Iteration node '{run2_iteration_id}' must exist" + ) + assert node1.get("iteration_number") == 1 + assert node2.get("iteration_number") == 1 + assert node1.get("started_at") == "2026-01-01T00:00:01Z", ( + "Run 1's Iteration node must retain its own started_at, not run 2's " + "(proves the two nodes were never merged together)" + ) + assert node2.get("started_at") == "2026-01-02T00:00:01Z" + + # --- Regression guard: exactly ONE distinct HAS_PART parent per Iteration. + # + # This is the invariant the I5 fix guarantees and the one a revert of + # run-scoping would break: pre-fix, both runs' iteration_number=1 would + # MERGE onto the SAME bare node_id, so that one node would end up with + # TWO distinct OrchestratorRun HAS_PART parents. Post-fix, each run's + # Iteration node_id is unique to that run, so each node has exactly one. + run1_orch_run_id = "s1::orch_run::2026-01-01T00:00:00Z" + run2_orch_run_id = "s1::orch_run::2026-01-02T00:00:00Z" + + def has_part_parents(iteration_id: str) -> list[str]: + """Distinct HAS_PART parent ids pointing at *iteration_id* in the fake graph.""" + return [ + src + for (src, dst), data in services.graph._edges.items() + if dst == iteration_id and data.get("type") == "HAS_PART" + ] + + run1_parents = has_part_parents(run1_iteration_id) + run2_parents = has_part_parents(run2_iteration_id) + + assert run1_parents == [run1_orch_run_id], ( + f"Run 1's Iteration node '{run1_iteration_id}' must have exactly ONE " + f"HAS_PART parent (its own OrchestratorRun). Got: {run1_parents!r}" + ) + assert run2_parents == [run2_orch_run_id], ( + f"Run 2's Iteration node '{run2_iteration_id}' must have exactly ONE " + f"HAS_PART parent (its own OrchestratorRun). Got: {run2_parents!r}" + ) + assert set(run1_parents).isdisjoint(run2_parents), ( + "No Iteration node may be shared (MERGEd) across the two " + "OrchestratorRuns -- each run's iterations must be distinct nodes " + "with distinct, non-overlapping HAS_PART parents." + ) # --------------------------------------------------------------------------- @@ -597,3 +714,172 @@ async def test_llm_response_creates_sourced_from_edge( assert edge.get("type") == "SOURCED_FROM", ( f"Edge type must be 'SOURCED_FROM'. Got: {edge.get('type')!r}" ) + + +# --------------------------------------------------------------------------- +# TC-10 / BLOCKER-2 -- iteration_scope completeness (spec §10.1, §10.4) +# +# The Iteration node is upsert_node'd from THREE sites: provider:request, +# llm:request, llm:response. Each site stamps the additive 'iteration_scope' +# ('run' | 'unscoped') property independently, sourced from the SAME cursor +# field (execution_start_ts) -- so a node can never be created/updated +# without a scope value, regardless of which of the three sites happens to +# be the one that actually writes it (e.g. a dead-lettered provider:request +# whose active_iteration_id mutation nonetheless survives to a later +# llm:request/llm:response). +# --------------------------------------------------------------------------- + + +class TestIterationScopeCompleteness: + """iteration_scope stamped at ALL THREE upsert_node call sites.""" + + async def test_provider_request_unscoped_with_no_execution_start( + self, services: HookStateService + ) -> None: + """TC-10: provider:request with NO preceding execution:start -> + iteration_scope == 'unscoped'.""" + assert services.data_layer_2.execution_start_ts is None + handler = IterationHandler(services) + await handler( + "provider:request", + {"session_id": "s1", "timestamp": "2026-01-01T00:00:00Z"}, + ) + node = await services.graph.get_node("s1::iteration::1") + assert node is not None + assert node.get("iteration_scope") == "unscoped", ( + f"Expected iteration_scope='unscoped'. Got: {node!r}" + ) + + async def test_provider_request_run_scoped_with_execution_start( + self, services: HookStateService + ) -> None: + """TC-10: a normal (execution:start already seen) run stamps 'run'.""" + services.data_layer_2.execution_start_ts = "2026-01-01T00:00:00Z" + handler = IterationHandler(services) + await handler( + "provider:request", + {"session_id": "s1", "timestamp": "2026-01-01T00:00:01Z"}, + ) + node_id = "s1::orch_run::2026-01-01T00:00:00Z::iteration::1" + node = await services.graph.get_node(node_id) + assert node is not None + assert node.get("iteration_scope") == "run", ( + f"Expected iteration_scope='run'. Got: {node!r}" + ) + + async def test_unscoped_emitted_log_is_info_not_warning( + self, services: HookStateService, caplog + ) -> None: + """Spec §10.4: the D6 log line for an unscoped iteration must be + INFO (or rate-limited), NEVER WARNING -- a loop-basic unscoped + session is a normal case, not an alert-worthy anomaly.""" + handler = IterationHandler(services) + with caplog.at_level( + logging.INFO, + logger="context_intelligence_server.handlers.data_layer_2.iteration", + ): + await handler( + "provider:request", + {"session_id": "s1", "timestamp": "2026-01-01T00:00:00Z"}, + ) + unscoped_records = [ + r for r in caplog.records if "unscoped_iteration_emitted" in r.message + ] + assert unscoped_records, ( + "expected an 'unscoped_iteration_emitted' log record to be emitted" + ) + assert all(r.levelno == logging.INFO for r in unscoped_records), ( + "unscoped_iteration_emitted must log at INFO, got levels: " + f"{[r.levelname for r in unscoped_records]}" + ) + assert not any(r.levelno >= logging.WARNING for r in caplog.records), ( + "no WARNING (or higher) should be emitted for a normal unscoped iteration" + ) + + async def test_llm_request_stamps_unscoped_independent_of_provider_request( + self, services: HookStateService + ) -> None: + """BLOCKER-2's completeness gap: llm:request must stamp iteration_scope + on its OWN upsert_node call, even when the node was never created by + provider:request (e.g. a dead-lettered provider:request whose + active_iteration_id cursor mutation nonetheless survives). Simulated + here by setting the cursor directly, bypassing provider:request.""" + services.data_layer_2.active_iteration_id = "s1::iteration::99" + handler = IterationHandler(services) + await handler( + "llm:request", + { + "session_id": "s1", + "timestamp": "2026-01-01T00:00:00Z", + "provider": "anthropic", + "model": "claude", + }, + ) + node = await services.graph.get_node("s1::iteration::99") + assert node is not None + assert node.get("iteration_scope") == "unscoped", ( + f"llm:request must stamp iteration_scope='unscoped'. Got: {node!r}" + ) + + async def test_llm_request_stamps_run_scope_when_execution_start_ts_set( + self, services: HookStateService + ) -> None: + services.data_layer_2.execution_start_ts = "2026-01-01T00:00:00Z" + node_id = "s1::orch_run::2026-01-01T00:00:00Z::iteration::1" + services.data_layer_2.active_iteration_id = node_id + handler = IterationHandler(services) + await handler( + "llm:request", + { + "session_id": "s1", + "timestamp": "2026-01-01T00:00:01Z", + "provider": "anthropic", + "model": "claude", + }, + ) + node = await services.graph.get_node(node_id) + assert node is not None + assert node.get("iteration_scope") == "run", ( + f"llm:request must stamp iteration_scope='run'. Got: {node!r}" + ) + + async def test_llm_response_stamps_unscoped_independent_of_provider_request( + self, services: HookStateService + ) -> None: + """Same completeness gap as llm:request, for the third site.""" + services.data_layer_2.active_iteration_id = "s1::iteration::99" + handler = IterationHandler(services) + await handler( + "llm:response", + { + "session_id": "s1", + "timestamp": "2026-01-01T00:00:00Z", + "usage": {"input_tokens": 1, "output_tokens": 1}, + }, + ) + node = await services.graph.get_node("s1::iteration::99") + assert node is not None + assert node.get("iteration_scope") == "unscoped", ( + f"llm:response must stamp iteration_scope='unscoped'. Got: {node!r}" + ) + + async def test_llm_response_stamps_run_scope_when_execution_start_ts_set( + self, services: HookStateService + ) -> None: + services.data_layer_2.execution_start_ts = "2026-01-01T00:00:00Z" + node_id = "s1::orch_run::2026-01-01T00:00:00Z::iteration::1" + services.data_layer_2.active_iteration_id = node_id + handler = IterationHandler(services) + await handler( + "llm:response", + { + "session_id": "s1", + "timestamp": "2026-01-01T00:00:01Z", + "usage": {"input_tokens": 1, "output_tokens": 1}, + }, + ) + node = await services.graph.get_node(node_id) + assert node is not None + assert node.get("iteration_scope") == "run", ( + f"llm:response must stamp iteration_scope='run'. Got: {node!r}" + ) diff --git a/tests/handlers/data_layer_2/test_session.py b/tests/handlers/data_layer_2/test_session.py index 19983caf..42b437e3 100644 --- a/tests/handlers/data_layer_2/test_session.py +++ b/tests/handlers/data_layer_2/test_session.py @@ -2078,6 +2078,14 @@ class TestClassifyMatrix: remove, because by the time a node reaches one of those states its StubSession marker has already been cleared by the transition that got it there. + + "IncompleteSession" heal-forward: EVERY start/fork cell (including the + former pure no-ops) now also removes "IncompleteSession" — see + _heal_forward() in session.py. A node may carry a stale IncompleteSession + marker from an earlier out-of-order session:end regardless of which + start/fork branch it reaches, so every start/fork transition strips it. + The end/None cells are unchanged — session:end still stamps + IncompleteSession when genuinely no type is known at end. """ CASES: list[tuple[str, str | None, bool, list[str], list[str]]] = [ @@ -2085,78 +2093,78 @@ class TestClassifyMatrix: # ------------------------------------------------------------------ # event=start # ------------------------------------------------------------------ - ("start", "ForkedSession", True, [], []), - ("start", "ForkedSession", False, [], []), - ("start", "SubSession", True, [], []), - ("start", "SubSession", False, [], []), - ("start", "RootSession", False, [], []), + ("start", "ForkedSession", True, [], ["IncompleteSession"]), + ("start", "ForkedSession", False, [], ["IncompleteSession"]), + ("start", "SubSession", True, [], ["IncompleteSession"]), + ("start", "SubSession", False, [], ["IncompleteSession"]), + ("start", "RootSession", False, [], ["IncompleteSession"]), ( "start", "RootSession", True, ["SubSession", "SST_EVENT"], - ["RootSession", "StubSession"], + ["RootSession", "StubSession", "IncompleteSession"], ), ( "start", None, True, ["Session", "SubSession", "SST_EVENT"], - ["StubSession"], + ["StubSession", "IncompleteSession"], ), ( "start", None, False, ["RootSession", "Session", "SST_EVENT"], - ["StubSession"], + ["StubSession", "IncompleteSession"], ), # ------------------------------------------------------------------ # event=fork # ------------------------------------------------------------------ - ("fork", "ForkedSession", True, [], []), - ("fork", "ForkedSession", False, [], []), + ("fork", "ForkedSession", True, [], ["IncompleteSession"]), + ("fork", "ForkedSession", False, [], ["IncompleteSession"]), ( "fork", "RootSession", True, ["ForkedSession", "SST_EVENT"], - ["RootSession", "StubSession"], + ["RootSession", "StubSession", "IncompleteSession"], ), ( "fork", "RootSession", False, ["ForkedSession", "SST_EVENT"], - ["RootSession", "StubSession"], + ["RootSession", "StubSession", "IncompleteSession"], ), ( "fork", "SubSession", True, ["ForkedSession", "SST_EVENT"], - ["SubSession", "StubSession"], + ["SubSession", "StubSession", "IncompleteSession"], ), ( "fork", "SubSession", False, ["ForkedSession", "SST_EVENT"], - ["SubSession", "StubSession"], + ["SubSession", "StubSession", "IncompleteSession"], ), ( "fork", None, True, ["Session", "ForkedSession", "SST_EVENT"], - ["StubSession"], + ["StubSession", "IncompleteSession"], ), ( "fork", None, False, ["Session", "ForkedSession", "SST_EVENT"], - ["StubSession"], + ["StubSession", "IncompleteSession"], ), # ------------------------------------------------------------------ # event=end @@ -2165,7 +2173,8 @@ class TestClassifyMatrix: # not a fabricated Root/Sub terminal. has_parent is irrelevant here — # the server never guesses; it marks and surfaces the health signal. # IncompleteSession is a confirmed (if incomplete) terminal, so - # StubSession is cleared here too. + # StubSession is cleared here too. UNCHANGED by heal-forward: end is + # not a start/fork transition. ("end", None, True, ["IncompleteSession", "SST_EVENT"], ["StubSession"]), ("end", None, False, ["IncompleteSession", "SST_EVENT"], ["StubSession"]), ("end", "RootSession", True, [], []), @@ -2882,3 +2891,127 @@ async def test_current_type_returns_none_for_incomplete_session_node( "_current_type must ignore IncompleteSession and return None, " "so a late start/fork can still classify the session normally" ) + + # ----------------------------------------------------------------------- + # Heal-forward: out-of-order end -> fork/start race + # + # A forked sub-session's session:end can drain, in an independent queue, + # BEFORE its session:fork/session:start. classify() sees current_type=None + # at end -> stamps IncompleteSession. When the real fork/start is then + # processed, it must strip the stale marker (heal-forward), leaving the + # node with ONLY the real terminal label. See + # docs/issues/incomplete-session-mislabeling.md and + # docs/plans/2026-08-12-incomplete-session-relabel-spec.md. + # ----------------------------------------------------------------------- + + async def test_out_of_order_end_then_fork_heals_incomplete_session( + self, services: HookStateService + ) -> None: + """end processed BEFORE fork: stale IncompleteSession must be stripped + the moment the real session:fork arrives, leaving ForkedSession only. + """ + handler = SessionHandler(services) + + # session:end drains first (simulating the cross-queue race) — stamps + # IncompleteSession on the bare node. + await handler( + "session:end", + {"session_id": "s-race-fork", "timestamp": "2026-01-01T01:00:00Z"}, + ) + node = await services.graph.get_node("s-race-fork") + assert node is not None + assert "IncompleteSession" in node["labels"], ( + "Precondition: out-of-order end must stamp the stale marker" + ) + + # The real session:fork arrives late. + await handler( + "session:fork", + { + "session_id": "s-race-fork", + "parent_id": "p-race", + "timestamp": "2026-01-01T00:00:00Z", + }, + ) + + node = await services.graph.get_node("s-race-fork") + assert node is not None + labels = node["labels"] + assert "ForkedSession" in labels, "Real terminal must be assigned by fork" + assert "IncompleteSession" not in labels, ( + "Heal-forward: session:fork must strip the stale IncompleteSession " + "marker left by the out-of-order session:end" + ) + + async def test_out_of_order_end_then_start_heals_incomplete_session( + self, services: HookStateService + ) -> None: + """end processed BEFORE start (no parent): stale IncompleteSession must + be stripped the moment the real session:start arrives, leaving + RootSession only. + """ + handler = SessionHandler(services) + + await handler( + "session:end", + {"session_id": "s-race-root", "timestamp": "2026-01-01T01:00:00Z"}, + ) + node = await services.graph.get_node("s-race-root") + assert node is not None + assert "IncompleteSession" in node["labels"], ( + "Precondition: out-of-order end must stamp the stale marker" + ) + + await handler( + "session:start", + {"session_id": "s-race-root", "timestamp": "2026-01-01T00:00:00Z"}, + ) + + node = await services.graph.get_node("s-race-root") + assert node is not None + labels = node["labels"] + assert "RootSession" in labels, "Real terminal must be assigned by start" + assert "IncompleteSession" not in labels, ( + "Heal-forward: session:start must strip the stale IncompleteSession " + "marker left by the out-of-order session:end" + ) + + async def test_out_of_order_end_then_start_with_parent_heals_to_subsession( + self, services: HookStateService + ) -> None: + """end processed BEFORE start (with parent): stale IncompleteSession + must be stripped, leaving SubSession only. + """ + handler = SessionHandler(services) + + await handler( + "session:end", + { + "session_id": "s-race-sub", + "parent_id": "p-race-sub", + "timestamp": "2026-01-01T01:00:00Z", + }, + ) + node = await services.graph.get_node("s-race-sub") + assert node is not None + assert "IncompleteSession" in node["labels"], ( + "Precondition: out-of-order end must stamp the stale marker" + ) + + await handler( + "session:start", + { + "session_id": "s-race-sub", + "parent_id": "p-race-sub", + "timestamp": "2026-01-01T00:00:00Z", + }, + ) + + node = await services.graph.get_node("s-race-sub") + assert node is not None + labels = node["labels"] + assert "SubSession" in labels, "Real terminal must be assigned by start" + assert "IncompleteSession" not in labels, ( + "Heal-forward: session:start must strip the stale IncompleteSession " + "marker left by the out-of-order session:end" + ) diff --git a/tests/integration/test_cursor_durability.py b/tests/integration/test_cursor_durability.py new file mode 100644 index 00000000..9bf1fe20 --- /dev/null +++ b/tests/integration/test_cursor_durability.py @@ -0,0 +1,834 @@ +"""Integration -- durable handler cursor across a worker rebuild (I5b). + +Exercises the REAL, unmocked handler pipeline (``setup_handlers`` + +``process_event``) against the default in-memory ``GraphState``, so **no +Neo4j is required** -- this file runs under the repo's no-Neo4j gate +(``uv run pytest tests/ -q``, AGENTS.md). + +``tests/integration/test_crash_recovery.py`` patches +``context_intelligence_server.registry.process_event``; with dispatch +mocked, no handler ever touches ``DataLayer2State``/``DataLayer3State``, so +that suite is structurally incapable of catching cursor loss. This file +deliberately does **not** patch ``process_event`` -- the whole point is to +prove the cross-handler cursor (``execution_start_ts``, ``iteration_count``, +``pending_tool_block_ids``, ...) survives a worker rebuild. + +Because ``GraphState`` is an in-memory, per-``HookStateService`` store (not a +shared backing store like Neo4j), a simulated worker rebuild uses a FRESH +``HookStateService`` -- exactly as it would in production, where the new +worker's in-process state starts empty and only the persisted ``.offset`` +cursor bridges the gap. Assertions therefore check each worker's own graph +for the nodes/edges IT was responsible for creating, never a private +``GraphState._nodes`` dict -- always through the public +``await worker.services.graph.get_node(...)`` / ``get_edge(...)`` accessors. +""" + +from __future__ import annotations + +import asyncio +import json +import time +from pathlib import Path + +from context_intelligence_server import registry as registry_module +from context_intelligence_server.pipeline import setup_handlers +from context_intelligence_server.queue_manager import QueueManager +from context_intelligence_server.registry import SessionRegistry, SessionWorker +from context_intelligence_server.services import HookStateService + +WORKSPACE = "/ws" + +# make_node_id (data_layer_1 default handler, invoked for EVERY event) parses +# "timestamp" as ISO-8601, so all event timestamps below must be valid ISO +# strings -- unlike orch_run_id/iteration_id, which embed the raw string +# verbatim and impose no format requirement of their own. +T0 = "2026-01-01T00:00:00+00:00" +T1 = "2026-01-01T00:00:01+00:00" +EXEC_START_TS = "2026-01-01T00:00:02+00:00" # execution:start's timestamp +T3 = "2026-01-01T00:00:03+00:00" +T4 = "2026-01-01T00:00:04+00:00" +T5 = "2026-01-01T00:00:05+00:00" +T5A = "2026-01-01T00:00:06+00:00" +T5B = "2026-01-01T00:00:07+00:00" +T6 = "2026-01-01T00:00:08+00:00" +T7 = "2026-01-01T00:00:09+00:00" +T8 = "2026-01-01T00:00:10+00:00" + + +def _line(event: str, data: dict) -> bytes: + return json.dumps({"event": event, "workspace": WORKSPACE, "data": data}).encode( + "utf-8" + ) + + +def _first_triplet(sid: str) -> list[bytes]: + """session:start -> prompt -> execution:start -> provider/llm triplet.""" + return [ + _line("session:start", {"session_id": sid, "timestamp": T0}), + _line("prompt:submit", {"session_id": sid, "timestamp": T1, "prompt": "hi"}), + _line("execution:start", {"session_id": sid, "timestamp": EXEC_START_TS}), + _line("provider:request", {"session_id": sid, "timestamp": T3}), + _line( + "llm:request", + { + "session_id": sid, + "timestamp": T4, + "provider": "anthropic", + "model": "claude", + }, + ), + _line( + "llm:response", + { + "session_id": sid, + "timestamp": T5, + "usage": {"input_tokens": 10, "output_tokens": 5}, + }, + ), + ] + + +def _second_triplet(sid: str) -> list[bytes]: + """A second provider/llm triplet, driven AFTER the state-reset boundary.""" + return [ + _line("provider:request", {"session_id": sid, "timestamp": T6}), + _line( + "llm:request", + { + "session_id": sid, + "timestamp": T7, + "provider": "anthropic", + "model": "claude", + }, + ), + _line( + "llm:response", + { + "session_id": sid, + "timestamp": T8, + "usage": {"input_tokens": 3, "output_tokens": 2}, + }, + ), + ] + + +def _worker(sid: str) -> SessionWorker: + return SessionWorker( + session_id=sid, + workspace=WORKSPACE, + services=HookStateService(workspace=WORKSPACE), + ) + + +async def _drain_to_idle( + reg: SessionRegistry, worker: SessionWorker, timeout: float = 5.0 +) -> None: + """Start drain_worker and poll until the log is fully committed, then cancel. + + "Idle" (queue empty) is the natural point at which a real process could + be interrupted between batches -- exactly the T1 (crash-restart) and T2 + (stale-reap) triggers described in the spec, both of which occur once a + batch has already committed and the drainer is polling an empty log. + """ + task = asyncio.create_task(reg.drain_worker(worker, flush_timeout=10.0)) + deadline = time.monotonic() + timeout + reached_idle = False + while time.monotonic() < deadline: + await asyncio.sleep(0.01) + if (await reg.queue_manager.read_batch(worker.session_id, 10)).lines == []: + reached_idle = True + break + task.cancel() + try: + await task + except asyncio.CancelledError: + pass + if not reached_idle: + raise AssertionError("drain did not reach idle within timeout") + + +# --------------------------------------------------------------------------- +# TC-1 -- T1: crash restart +# --------------------------------------------------------------------------- + + +async def test_tc1_crash_restart_preserves_cursor() -> None: + sid = "cursor-tc1" + reg1 = SessionRegistry() + for raw in _first_triplet(sid): + await reg1.queue_manager.append(sid, raw) + + w1 = _worker(sid) + reg1._register_for_test(w1) + await _drain_to_idle(reg1, w1) + + node1 = f"{sid}::orch_run::{EXEC_START_TS}::iteration::1" + assert await w1.services.graph.get_node(node1) is not None + assert await w1.services.graph.get_node(f"{sid}::iteration::1") is None + assert w1.services.data_layer_2.iteration_count == 1 + + # "Crash": w1/reg1 are simply discarded (never deregistered) -- a fresh + # SessionRegistry over the SAME on-disk queues dir simulates a process + # restart recovering from the persisted .offset cursor. + reg2 = SessionRegistry() + for raw in _second_triplet(sid): + await reg2.queue_manager.append(sid, raw) + + w2 = _worker(sid) + reg2._register_for_test(w2) + await _drain_to_idle(reg2, w2) + + node2 = f"{sid}::orch_run::{EXEC_START_TS}::iteration::2" + assert await w2.services.graph.get_node(node2) is not None + assert await w2.services.graph.get_node(f"{sid}::iteration::2") is None + # iteration_count CONTINUED at 2 -- no reset back to 1. + assert w2.services.data_layer_2.iteration_count == 2 + assert w2.services.data_layer_2.execution_start_ts == EXEC_START_TS + + +# --------------------------------------------------------------------------- +# TC-2 -- T2: stale-worker reap +# --------------------------------------------------------------------------- + + +async def test_tc2_stale_reap_preserves_cursor() -> None: + sid = "cursor-tc2" + reg = SessionRegistry() + for raw in _first_triplet(sid): + await reg.queue_manager.append(sid, raw) + + w1 = _worker(sid) + reg._register_for_test(w1) + await _drain_to_idle(reg, w1) + + # Simulate the stale-reap path: _deregister pops the worker WITHOUT + # touching .log/.offset (registry.py _deregister) -- the queue files + # survive on disk exactly as they would after a real stale-session reap. + reg._deregister(sid) + + for raw in _second_triplet(sid): + await reg.queue_manager.append(sid, raw) + + w2 = _worker(sid) + reg._register_for_test(w2) + await _drain_to_idle(reg, w2) + + node2 = f"{sid}::orch_run::{EXEC_START_TS}::iteration::2" + orch_run_id = f"{sid}::orch_run::{EXEC_START_TS}" + assert await w2.services.graph.get_node(node2) is not None + assert await w2.services.graph.get_node(f"{sid}::iteration::1") is None + assert await w2.services.graph.get_node(f"{sid}::iteration::2") is None + # E06 HAS_PART edge from the SAME orchestrator run -- proves + # execution_start_ts (not just iteration_count) was carried through. + edge = await w2.services.graph.get_edge(orch_run_id, node2) + assert edge is not None + assert edge["type"] == "HAS_PART" + + +# --------------------------------------------------------------------------- +# TC-3 -- sibling cursor state (pending_tool_block_ids / E09) survives +# --------------------------------------------------------------------------- + + +async def test_tc3_pending_tool_block_ids_survives_rebuild() -> None: + sid = "cursor-tc3" + reg1 = SessionRegistry() + lines = _first_triplet(sid) + [ + _line( + "content_block:start", + {"session_id": sid, "timestamp": T5A, "block_index": 0}, + ), + _line( + "content_block:end", + { + "session_id": sid, + "timestamp": T5B, + "block_index": 0, + "block": {"type": "tool_call", "id": "toolblock-1"}, + }, + ), + ] + for raw in lines: + await reg1.queue_manager.append(sid, raw) + + w1 = _worker(sid) + reg1._register_for_test(w1) + await _drain_to_idle(reg1, w1) + + block_node_id = f"{sid}::block::1::0" + assert w1.services.data_layer_2.pending_tool_block_ids == { + "toolblock-1": block_node_id + } + + # Rebuild: fresh registry + fresh HookStateService (new in-memory graph), + # restoring ONLY from the persisted cursor. + reg2 = SessionRegistry() + await reg2.queue_manager.append( + sid, + _line( + "tool:pre", + { + "session_id": sid, + "timestamp": T6, + "tool_call_id": "toolblock-1", + "tool_name": "bash", + }, + ), + ) + w2 = _worker(sid) + reg2._register_for_test(w2) + await _drain_to_idle(reg2, w2) + + # E09: ContentBlock -[:CAUSED]-> ToolCall -- only possible if + # pending_tool_block_ids was restored from the persisted cursor. + edge = await w2.services.graph.get_edge(block_node_id, "toolblock-1") + assert edge is not None + assert edge["type"] == "CAUSED" + + +# --------------------------------------------------------------------------- +# TC-4 -- atomicity: offset and cursor always come from the SAME JSON write +# --------------------------------------------------------------------------- + + +async def test_tc4_offset_and_cursor_written_atomically() -> None: + sid = "cursor-tc4" + reg = SessionRegistry() + for raw in _first_triplet(sid): + await reg.queue_manager.append(sid, raw) + + w = _worker(sid) + reg._register_for_test(w) + await _drain_to_idle(reg, w) + + settings = registry_module.get_settings() + offset_path = Path(settings.queues_path) / f"{sid}.offset" + record = json.loads(offset_path.read_text(encoding="utf-8")) + assert record["v"] == 1 + assert isinstance(record["offset"], int) and record["offset"] > 0 + assert record["cursor"]["dl2"]["iteration_count"] == 1 + assert record["cursor"]["dl2"]["execution_start_ts"] == EXEC_START_TS + + # The manager's own reader agrees exactly with the raw file -- there is + # only ever ONE record, never a separate cursor file that could skew. + committed, cursor = reg.queue_manager._read_offset_record(sid) + assert committed == record["offset"] + assert cursor == record["cursor"] + + +# --------------------------------------------------------------------------- +# TC-4b -- crash-before-commit replay idempotence (edge preservation) +# --------------------------------------------------------------------------- + + +async def test_tc4b_crash_before_commit_replay_is_idempotent() -> None: + sid = "cursor-tc4b" + reg = SessionRegistry() + lines = _first_triplet(sid) + [ + _line( + "content_block:start", + {"session_id": sid, "timestamp": T5A, "block_index": 0}, + ), + _line( + "content_block:end", + { + "session_id": sid, + "timestamp": T5B, + "block_index": 0, + "block": {"type": "tool_call", "id": "toolblock-2"}, + }, + ), + ] + for raw in lines: + await reg.queue_manager.append(sid, raw) + + # Simulate a crash AFTER dispatch but BEFORE commit: drive the batch + # through the exact same dispatch step drain_worker uses + # (SessionRegistry._process_batch), then never call commit -- the + # offset file never advances past 0. + batch = await reg.queue_manager.read_batch(sid, max_items=100) + w_crashed = _worker(sid) + handlers = setup_handlers(w_crashed.services) + await reg._process_batch(w_crashed, batch, handlers) + + node_id = f"{sid}::orch_run::{EXEC_START_TS}::iteration::1" + assert await w_crashed.services.graph.get_node(node_id) is not None + assert (await reg.queue_manager.read_batch(sid, 10)).lines != [] # uncommitted + + # Re-drain from scratch (fresh worker, offset still 0, cursor still + # None) via the REAL drain loop -- this is the replay. + w_replay = _worker(sid) + reg._register_for_test(w_replay) + await _drain_to_idle(reg, w_replay) + + replayed_node = await w_replay.services.graph.get_node(node_id) + assert replayed_node is not None + # Identical node_id, identical iteration_number -- no double-increment + # from replaying a batch the crashed attempt already touched in memory. + assert replayed_node["iteration_number"] == 1 + assert w_replay.services.data_layer_2.iteration_count == 1 + + # E09 edge preserved across the crash-before-commit boundary too. + block_node_id = f"{sid}::block::1::0" + edge = await w_replay.services.graph.get_edge(block_node_id, "toolblock-2") + assert edge is None or edge["type"] == "CAUSED" # created on tool:pre, not here + # No tool:pre in this script; the important, load-bearing assertion is + # that pending_tool_block_ids itself round-tripped through the replay: + assert w_replay.services.data_layer_2.pending_tool_block_ids == { + "toolblock-2": block_node_id + } + + +# --------------------------------------------------------------------------- +# TC-5 -- legacy bare-integer .offset compatibility +# --------------------------------------------------------------------------- + + +async def test_tc5_legacy_bare_int_offset_compat() -> None: + sid = "cursor-tc5" + reg = SessionRegistry() + first = _line("session:start", {"session_id": sid, "timestamp": T0}) + second = _line( + "prompt:submit", {"session_id": sid, "timestamp": T1, "prompt": "hi"} + ) + await reg.queue_manager.append(sid, first) + await reg.queue_manager.append(sid, second) + + # Pre-write a bare-integer offset (pre-upgrade shape) covering only the + # first (newline-terminated) line. + settings = registry_module.get_settings() + offset_path = Path(settings.queues_path) / f"{sid}.offset" + first_line_len = len(first) + 1 # +1 for the newline append() adds + offset_path.write_text(str(first_line_len), encoding="utf-8") + + committed, cursor = reg.queue_manager._read_offset_record(sid) + assert committed == first_line_len + assert cursor is None + + w = _worker(sid) + reg._register_for_test(w) + await _drain_to_idle(reg, w) # no exception; drains the remaining line + + # The next commit rewrites the file in the new JSON form. + text = offset_path.read_text(encoding="utf-8").strip() + assert text.startswith("{") + record = json.loads(text) + assert record["v"] == 1 + assert record["offset"] == first_line_len + len(second) + 1 + + +# --------------------------------------------------------------------------- +# TC-6 -- cursor is JSON-round-trippable (regression guard for future fields) +# --------------------------------------------------------------------------- + + +def test_tc6_cursor_json_round_trip() -> None: + baseline = HookStateService(workspace=WORKSPACE) + snapshot = baseline.snapshot_cursor() + + # JSON-safety regression guard: every field must survive a JSON round-trip. + round_tripped = json.loads(json.dumps(snapshot)) + assert round_tripped == snapshot + + mutated = HookStateService(workspace=WORKSPACE) + mutated.data_layer_2.iteration_count = 99 + mutated.data_layer_2.execution_start_ts = "should-be-overwritten" + mutated.restore_cursor(round_tripped) + + assert mutated.data_layer_2 == baseline.data_layer_2 + assert mutated.data_layer_3 == baseline.data_layer_3 + + +# --------------------------------------------------------------------------- +# TC-8 -- recovery_reconcile_dead preserves the cursor (R2 guard) +# --------------------------------------------------------------------------- + + +async def test_tc8_recovery_reconcile_dead_preserves_cursor() -> None: + # DO NOT skip or xfail this test (spec \u00a710.4): recovery_reconcile_dead is + # the ONLY offset writer besides commit(). If it ever drops the cursor, + # a startup that skips a dead-lettered line silently wipes cross-handler + # state on the recovery path -- the exact bug this change fixes, + # reintroduced through the one writer that isn't commit(). + sid = "cursor-tc8" + settings = registry_module.get_settings() + qm = QueueManager(queues_dir=Path(settings.queues_path)) + line = _line("provider:request", {"session_id": sid, "timestamp": T0}) + await qm.append(sid, line) + + cursor = { + "dl2": { + "execution_start_ts": "2026-01-01T00:00:09+00:00", + "active_iteration_id": None, + "pending_tool_block_ids": {}, + "last_prompt_id": None, + "last_completed_orch_run_id": None, + "iteration_count": 7, + }, + "dl3": {"active_recipe_run_stack": [], "active_recipe_step_id": None}, + } + # Commit a real cursor at offset 0 (line not yet consumed), then + # dead-letter the pending line so recovery_reconcile_dead has something + # to skip past -- mirroring the dead_letter-then-crash-before-commit window. + await qm.commit(sid, 0, cursor) + await qm.dead_letter(sid, line, "boom") + + skipped = await qm.recovery_reconcile_dead() + assert skipped == 1 + + new_offset, new_cursor = qm._read_offset_record(sid) + assert new_offset == len(line) + 1 # +1 for the newline append() added + assert new_cursor == cursor # cursor carried through UNMODIFIED + + +# --------------------------------------------------------------------------- +# TC-9 -- delete_drained removes the cursor (no stale-restore on a recycled key) +# --------------------------------------------------------------------------- + + +async def test_tc9_delete_drained_removes_cursor() -> None: + sid = "cursor-tc9" + settings = registry_module.get_settings() + qm = QueueManager(queues_dir=Path(settings.queues_path)) + await qm.commit(sid, 10, {"dl2": {"iteration_count": 3}, "dl3": {}}) + assert await qm.read_cursor(sid) is not None + + await qm.delete_drained(sid) + assert await qm.read_cursor(sid) is None + + +# --------------------------------------------------------------------------- +# BLOCKER-1 -- phantom-cursor guard (R6xD1): a dead-lettered line's in-memory +# cursor mutation must not survive into the committed cursor snapshot. +# --------------------------------------------------------------------------- + + +async def test_blocker1_dead_lettered_line_does_not_leak_cursor_mutation() -> None: + """A line that fails ALL flush retries is dead-lettered and its graph + write is discarded (``_handle_exhausted_batch``) -- but ``IterationHandler`` + mutates ``active_iteration_id``/``iteration_count`` in memory BEFORE the + graph write (``iteration.py`` ~86-100). Without a guard, the unconditional + ``qm.commit(..., worker.services.snapshot_cursor())`` at the end of every + iteration of the per-line loop persists that mutation anyway, so the + durable cursor ends up pointing at a node that was NEVER written -- a + phantom that survives a restart (R6, promoted to BLOCKER-1 by persisting + the cursor at all -- see spec \u00a710.1). + """ + sid = "cursor-blocker1" + reg = SessionRegistry() + w = _worker(sid) + handlers = setup_handlers(w.services) + + # Clean baseline: session:start -> prompt -> execution:start, drained + # normally so execution_start_ts is set while iteration_count / + # active_iteration_id are still at their untouched defaults. + setup_lines = [ + _line("session:start", {"session_id": sid, "timestamp": T0}), + _line("prompt:submit", {"session_id": sid, "timestamp": T1, "prompt": "hi"}), + _line("execution:start", {"session_id": sid, "timestamp": EXEC_START_TS}), + ] + for raw in setup_lines: + await reg.queue_manager.append(sid, raw) + reg._register_for_test(w) + await _drain_to_idle(reg, w) + + pre_cursor = await reg.queue_manager.read_cursor(sid) + assert pre_cursor is not None + assert pre_cursor["dl2"]["iteration_count"] == 0 + assert pre_cursor["dl2"]["active_iteration_id"] is None + + # The poison line: provider:request mutates active_iteration_id / + # iteration_count BEFORE its graph write (iteration.py _handle_provider_ + # request). Force that write to fail -- standing in for a flush that + # exhausted every retry -- so the line is routed down the dead-letter/ + # discard path exactly as _handle_exhausted_batch runs it once + # drain_worker's batch-level retry budget is spent. + poison_raw = _line("provider:request", {"session_id": sid, "timestamp": T3}) + await reg.queue_manager.append(sid, poison_raw) + batch = await reg.queue_manager.read_batch(sid, max_items=10) + assert batch.lines == [poison_raw] + + # IterationHandler is an ENRICHER (pipeline step 5), dispatched AFTER the + # DefaultHandler's own Event-node upsert_node call (step 4). Only fail the + # write that creates the Iteration node itself (identified by its + # ``labels``) -- otherwise the boom fires on the DefaultHandler's raw + # Event-node write first and IterationHandler's mutation (which happens + # BEFORE its own upsert_node call) never even runs, which would not + # reproduce the phantom at all. + phantom_iteration_id = f"{sid}::orch_run::{EXEC_START_TS}::iteration::1" + original_upsert_node = w.services.graph.upsert_node + + async def _boom_upsert_node(node_id: str, data: dict) -> None: + if "Iteration" in (data.get("labels") or []): + raise RuntimeError("simulated write failure (flush retries exhausted)") + await original_upsert_node(node_id, data) + + w.services.graph.upsert_node = _boom_upsert_node # type: ignore[method-assign] + + await reg._handle_exhausted_batch(w, batch, handlers) + + # The write really never landed -- confirms this is the discard path, + # not a passing line. + assert await w.services.graph.get_node(phantom_iteration_id) is None + dead_letters = await reg.queue_manager.read_dead_letters(sid) + assert len(dead_letters) == 1 + + # THE BUG (BLOCKER-1): the committed cursor must NOT carry the + # dead-lettered line's mutation forward -- it must match the clean + # pre-line snapshot, not point at the never-written node. + persisted = await reg.queue_manager.read_cursor(sid) + assert persisted is not None + assert persisted["dl2"]["iteration_count"] == 0, ( + "phantom cursor: iteration_count advanced past a dead-lettered line" + ) + assert persisted["dl2"]["active_iteration_id"] is None, ( + "phantom cursor: active_iteration_id points at a never-written node" + ) + + +async def test_blocker1_successful_line_after_poison_keeps_its_own_mutation() -> None: + """Selectivity guard: only the DISCARDED line's mutation rolls back. A + line that succeeds -- even immediately after a dead-lettered one in the + same exhausted-batch pass -- must keep its own cursor mutation and its + own graph write. Proves the fix rolls back per-line, not the whole batch. + """ + sid = "cursor-blocker1b" + reg = SessionRegistry() + w = _worker(sid) + handlers = setup_handlers(w.services) + + setup_lines = [ + _line("session:start", {"session_id": sid, "timestamp": T0}), + _line("prompt:submit", {"session_id": sid, "timestamp": T1, "prompt": "hi"}), + _line("execution:start", {"session_id": sid, "timestamp": EXEC_START_TS}), + ] + for raw in setup_lines: + await reg.queue_manager.append(sid, raw) + reg._register_for_test(w) + await _drain_to_idle(reg, w) + + poison_raw = _line("provider:request", {"session_id": sid, "timestamp": T3}) + good_raw = _line("provider:request", {"session_id": sid, "timestamp": T4}) + await reg.queue_manager.append(sid, poison_raw) + await reg.queue_manager.append(sid, good_raw) + batch = await reg.queue_manager.read_batch(sid, max_items=10) + assert batch.lines == [poison_raw, good_raw] + + # Fail only the FIRST Iteration-node creation call (the poisoned line's). + # Identified by ``labels`` rather than call order or node_id, since the + # good line's computed node_id depends on whether the guard rolled the + # counter back (that dependency is exactly what this test proves). + original_upsert_node = w.services.graph.upsert_node + iteration_upserts = 0 + + async def _flaky_upsert_node(node_id: str, data: dict) -> None: + nonlocal iteration_upserts + if "Iteration" in (data.get("labels") or []): + iteration_upserts += 1 + if iteration_upserts == 1: + raise RuntimeError("simulated write failure (flush retries exhausted)") + await original_upsert_node(node_id, data) + + w.services.graph.upsert_node = _flaky_upsert_node # type: ignore[method-assign] + + await reg._handle_exhausted_batch(w, batch, handlers) + + dead_letters = await reg.queue_manager.read_dead_letters(sid) + assert len(dead_letters) == 1 + + # The final committed cursor reflects ONLY the successful line's + # mutation: the counter advanced exactly once (0 -> 1), not twice -- + # this only holds if the poisoned line's increment was rolled back + # before the good line ran. + persisted = await reg.queue_manager.read_cursor(sid) + assert persisted is not None + assert persisted["dl2"]["iteration_count"] == 1 + + # The successful line's own node must exist and carry iteration_number 1 + # (not 2) -- proving its mutation was NOT disturbed by the earlier + # line's rollback. + good_iteration_id = persisted["dl2"]["active_iteration_id"] + assert good_iteration_id is not None + good_node = await w.services.graph.get_node(good_iteration_id) + assert good_node is not None + assert good_node["iteration_number"] == 1 + + +# --------------------------------------------------------------------------- +# BLOCKER-2 -- iteration_scope completeness (spec §10.1): dead-letter -> +# enrichment ordering. The Iteration node is upsert_node'd from THREE sites +# (provider:request, llm:request, llm:response); iteration_scope must be +# stamped at all three so a dead-lettered provider:request followed by a +# surviving llm:request/llm:response can never leave the node with neither +# value. +# --------------------------------------------------------------------------- + + +async def test_blocker2_dead_lettered_provider_request_then_surviving_llm_events() -> ( + None +): + """Drive a provider:request down the dead-letter path -- exactly as + BLOCKER-1's own reproduction does (the Iteration-node write fails, is + dead-lettered, and BLOCKER-1's phantom-cursor guard rolls + ``active_iteration_id`` back to its pre-line value) -- then append a + 'surviving' llm:request/llm:response pair for the same run, as a client + that is unaware the provider:request failed server-side would. + + The invariant under test: whatever the Iteration node ends up looking + like -- present or genuinely absent -- it must NEVER be present without a + valid ``iteration_scope``. That is the literal defect BLOCKER-2 exists to + close: a node written by one of the three sites (llm:request/llm:response) + that has properties but no scope at all, because only provider:request + used to stamp it. + + SURPRISE (see task's evidence requirements): with BLOCKER-1's rollback + already in place, ``active_iteration_id`` resets to ``None`` (this run's + pre-line value, since this is the run's first iteration) once the + provider:request line is dead-lettered. ``_handle_llm_request`` / + ``_handle_llm_response`` both early-return when the cursor is ``None`` + (iteration.py:163-165, :198-200), so in THIS exact ordering no phantom + node is created at all -- the node is legitimately absent, not + present-but-scopeless. BLOCKER-1 and BLOCKER-2 therefore compose as + defense-in-depth: BLOCKER-1 prevents the enrichers from writing to a + phantom id in the first place; BLOCKER-2 additionally guarantees that IF + they ever do write (e.g. a future change loosens BLOCKER-1's rollback, or + active_iteration_id survives because a PRIOR iteration in the same run + was the one dead-lettered instead of the first), the node still cannot + end up scope-less. + """ + sid = "cursor-blocker2" + reg = SessionRegistry() + w = _worker(sid) + handlers = setup_handlers(w.services) + + setup_lines = [ + _line("session:start", {"session_id": sid, "timestamp": T0}), + _line("prompt:submit", {"session_id": sid, "timestamp": T1, "prompt": "hi"}), + _line("execution:start", {"session_id": sid, "timestamp": EXEC_START_TS}), + ] + for raw in setup_lines: + await reg.queue_manager.append(sid, raw) + reg._register_for_test(w) + await _drain_to_idle(reg, w) + + # Dead-letter the provider:request: its own Iteration-node write fails, + # standing in for flush retries exhausted (BLOCKER-1's exact setup). + phantom_iteration_id = f"{sid}::orch_run::{EXEC_START_TS}::iteration::1" + original_upsert_node = w.services.graph.upsert_node + + async def _boom_upsert_node(node_id: str, data: dict) -> None: + if "Iteration" in (data.get("labels") or []): + raise RuntimeError("simulated write failure (flush retries exhausted)") + await original_upsert_node(node_id, data) + + w.services.graph.upsert_node = _boom_upsert_node # type: ignore[method-assign] + + poison_raw = _line("provider:request", {"session_id": sid, "timestamp": T3}) + await reg.queue_manager.append(sid, poison_raw) + batch = await reg.queue_manager.read_batch(sid, max_items=10) + assert batch.lines == [poison_raw] + await reg._handle_exhausted_batch(w, batch, handlers) + + dead_letters = await reg.queue_manager.read_dead_letters(sid) + assert len(dead_letters) == 1 + # BLOCKER-1 confirmed: the cursor was rolled back, not left phantom. + assert w.services.data_layer_2.active_iteration_id is None + assert await w.services.graph.get_node(phantom_iteration_id) is None + + # Restore the real upsert_node -- a "surviving" llm:request/llm:response + # pair, emitted by a client that doesn't know the provider:request failed + # server-side, must be able to write normally. + w.services.graph.upsert_node = original_upsert_node # type: ignore[method-assign] + + for raw in ( + _line( + "llm:request", + { + "session_id": sid, + "timestamp": T4, + "provider": "anthropic", + "model": "claude", + }, + ), + _line( + "llm:response", + { + "session_id": sid, + "timestamp": T5, + "usage": {"input_tokens": 1, "output_tokens": 1}, + }, + ), + ): + await reg.queue_manager.append(sid, raw) + tail_batch = await reg.queue_manager.read_batch(sid, max_items=10) + await reg._process_batch(w, tail_batch, handlers) + await reg._flush_barrier(w) + await reg.queue_manager.commit( + sid, tail_batch.end_offset, w.services.snapshot_cursor() + ) + + # THE INVARIANT: never present-but-scopeless. In this repo's current code + # (BLOCKER-1 active) the node is legitimately absent -- see the docstring + # "SURPRISE" note above. + node = await w.services.graph.get_node(phantom_iteration_id) + assert node is None, ( + "with BLOCKER-1's rollback active, active_iteration_id is None so " + "the surviving llm:* events must early-return and write nothing -- " + f"if this now fails, BLOCKER-1's guard regressed. Got node: {node!r}" + ) + if node is not None: # pragma: no cover -- documents the invariant even + # if BLOCKER-1's rollback semantics ever change so a write DOES land. + assert node.get("iteration_scope") in ("run", "unscoped"), ( + f"Iteration node present without a valid iteration_scope: {node!r}" + ) + + +async def test_blocker2_llm_events_stamp_scope_when_cursor_survives() -> None: + """Direct, ordering-independent proof of the completeness fix: even + without going through the dead-letter machinery at all, if + ``active_iteration_id`` ever points at a node that llm:request/ + llm:response are the FIRST to write (simulating any path -- present or + future -- by which the cursor mutation outlives the node's own creation + write), the enrichers stamp iteration_scope on their own, independent of + provider:request. Uses ``IterationHandler`` directly (not the full + registry/drain machinery) since this is a targeted unit-level proof of + sites 2 and 3, not a durability/ordering scenario. + """ + from context_intelligence_server.handlers.data_layer_2.iteration import ( + IterationHandler, + ) + + sid = "cursor-blocker2b" + services = HookStateService(workspace=WORKSPACE) + handler = IterationHandler(services) + + # No execution:start -- unscoped branch. + services.data_layer_2.active_iteration_id = f"{sid}::iteration::99" + await handler( + "llm:request", + { + "session_id": sid, + "timestamp": T4, + "provider": "anthropic", + "model": "claude", + }, + ) + node = await services.graph.get_node(f"{sid}::iteration::99") + assert node is not None + assert node.get("iteration_scope") == "unscoped" + + # execution_start_ts now set -- run-scoped branch, llm:response site. + services.data_layer_2.execution_start_ts = EXEC_START_TS + run_scoped_id = f"{sid}::orch_run::{EXEC_START_TS}::iteration::7" + services.data_layer_2.active_iteration_id = run_scoped_id + await handler( + "llm:response", + { + "session_id": sid, + "timestamp": T5, + "usage": {"input_tokens": 1, "output_tokens": 1}, + }, + ) + node2 = await services.graph.get_node(run_scoped_id) + assert node2 is not None + assert node2.get("iteration_scope") == "run" diff --git a/tests/integration/test_data_layer_3_delegation_skill.py b/tests/integration/test_data_layer_3_delegation_skill.py index ad56d110..8295d83f 100644 --- a/tests/integration/test_data_layer_3_delegation_skill.py +++ b/tests/integration/test_data_layer_3_delegation_skill.py @@ -236,8 +236,10 @@ async def test_skill_loaded_during_iteration_creates_e05(self) -> None: handlers, ) - # iteration_id = SESSION_ID::iteration::1 (first iteration) - iteration_id = f"{SESSION_ID}::iteration::1" + # P2.1: execution:start fired at T1 before provider:request, so IterationHandler + # run-scopes the iteration_id as '{session_id}::orch_run::{T1}::iteration::1' + # (not the bare '{session_id}::iteration::1' shape). + iteration_id = f"{SESSION_ID}::orch_run::{T1}::iteration::1" skill_load_id = f"{SESSION_ID}::skill::{SKILL_NAME}::{T2}" # Verify active_iteration_id was set by IterationHandler diff --git a/tests/neo4j/test_blob_reclaim.py b/tests/neo4j/test_blob_reclaim.py new file mode 100644 index 00000000..c53f88f5 --- /dev/null +++ b/tests/neo4j/test_blob_reclaim.py @@ -0,0 +1,757 @@ +"""Tier 3 -- Neo4j end-to-end tests for POST /admin/blobs/reclaim. + +Covers the design in +``docs/plans/2026-08-12-blob-reclaim-endpoint-spec.md`` (see the "Council +amendment -- AUTHORITATIVE" section, which supersedes conflicting earlier +text): the ONE shared ``_select_orphans`` selection path, the B1 Event.data +invariant, the B2 structural-JSON-extraction requirement (no regex, no +APOC), and the B3 durable undrained-queue safety gate. + +Requires Docker and the ``docker`` Python package -- skipped via the +``neo4j_container`` fixture in ``tests/neo4j/conftest.py`` when unavailable. + +Run explicitly: + cd amplifier-context-intelligence + uv run pytest tests/neo4j/test_blob_reclaim.py -v -m neo4j +""" + +from __future__ import annotations + +import json +import os +import time +from collections.abc import AsyncGenerator +from pathlib import Path +from typing import Any + +import httpx +import pytest +from context_intelligence_server.config import get_settings +from context_intelligence_server.handlers.data_layer_1.default import DefaultHandler +from context_intelligence_server.registry import SessionWorker +from context_intelligence_server.services import HookStateService +from neo4j import AsyncGraphDatabase + +pytestmark = pytest.mark.neo4j + +# --------------------------------------------------------------------------- +# Fixture-data helpers +# --------------------------------------------------------------------------- + +_OLD_AGE_SECONDS = 7_200.0 # 2 hours -- comfortably past any min_age_minutes used + + +def _write_blob( + root: Path, session_id: str, key: str, *, age_seconds: float = 0.0 +) -> Path: + """Write a fake blob file at the exact layout AsyncDiskBlobStore uses. + + ``age_seconds > 0`` back-dates the file's mtime via ``os.utime`` so tests + can exercise the min_age_minutes gate deterministically. + """ + p = root / session_id / "blobs" / f"{key}.json" + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(json.dumps({"payload": "x" * 32}), encoding="utf-8") + if age_seconds > 0: + ts = time.time() - age_seconds + os.utime(p, (ts, ts)) + return p + + +async def _create_event_node( + driver: Any, *, node_id: str, workspace: str, data: dict[str, Any] +) -> None: + """Create a bare :Event node carrying *data* as a JSON string, mirroring + what DefaultHandler persists in production (node_props["data"] = + json.dumps(data)).""" + async with driver.session() as session: + await session.run( + "CREATE (:Event {node_id: $node_id, workspace: $workspace, data: $data})", + {"node_id": node_id, "workspace": workspace, "data": json.dumps(data)}, + ) + + +async def _create_carrier_node( + driver: Any, + *, + label: str, + node_id: str, + workspace: str, + prop_name: str, + prop_value: str, +) -> None: + """Create a bare node of *label* carrying a single non-``data`` property. + + Used to seed a reference that lives ONLY on the given carrier property + (``tool_input`` / ``prompt`` / ``response``) with deliberately NO :Event + node anywhere in the graph -- proving the reference-scan hardening + (docs/plans/2026-08-12-blob-reclaim-reference-scan-hardening.md) finds it + via the new per-property UNION branches, not the pre-existing + ``:Event.data`` branch. + """ + async with driver.session() as session: + await session.run( + f"CREATE (n:{label} {{node_id: $node_id, workspace: $workspace}}) " + f"SET n.{prop_name} = $prop_value", + {"node_id": node_id, "workspace": workspace, "prop_value": prop_value}, + ) + + +# --------------------------------------------------------------------------- +# Fixtures +# --------------------------------------------------------------------------- + + +@pytest.fixture +async def neo4j_driver( + neo4j_container: dict[str, Any], +) -> AsyncGenerator[Any, None]: + """A standalone async driver for seeding/verifying graph state directly.""" + driver = AsyncGraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + yield driver + await driver.close() + + +@pytest.fixture +async def admin_client( + tmp_path: Path, + neo4j_container: dict[str, Any], + monkeypatch: pytest.MonkeyPatch, +) -> AsyncGenerator[httpx.AsyncClient, None]: + """A live ASGI client for /admin/blobs/reclaim wired to the real container. + + - ``require_admin`` is overridden to a no-op (auth enforcement is proven + separately in ``tests/routers/test_blob_reclaim_auth.py``). + - ``blob_path`` is redirected to ``tmp_path/blobs`` via the env-var + + ``get_settings.cache_clear()`` pattern (mirrors + ``tests/integration/test_blob_pipeline.py::integration_env``) so the + route's own ``get_settings()`` call sees it. + - ``queues_path`` is redirected to the SAME ``tmp_path/queues`` the + autouse ``safe_settings`` fixture (tests/conftest.py) already points + ``registry.queue_manager`` at -- no extra wiring needed. + - ``app.state.neo4j_query_driver`` is a REAL driver against the live test + container (not a mock), so the reference scan runs for real. + """ + import context_intelligence_server.main as main_module + from context_intelligence_server.routers.admin import require_admin + + blob_dir = tmp_path / "blobs" + blob_dir.mkdir() + + get_settings.cache_clear() + monkeypatch.setenv("AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_BLOB_PATH", str(blob_dir)) + + main_module.create_asgi_app() + main_module.app.dependency_overrides[require_admin] = lambda: None + + query_driver = AsyncGraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + monkeypatch.setattr( + main_module.app.state, "neo4j_query_driver", query_driver, raising=False + ) + monkeypatch.setattr( + main_module.app.state, "neo4j_query_access_mode", "READ", raising=False + ) + + try: + async with httpx.AsyncClient( + transport=httpx.ASGITransport(app=main_module.app), + base_url="http://test", + ) as client: + yield client + finally: + main_module.app.dependency_overrides.pop(require_admin, None) + await query_driver.close() + get_settings.cache_clear() + # neo4j_container is session-scoped -- clean up between tests. + cleanup_driver = AsyncGraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + async with cleanup_driver.session() as session: + await session.run("MATCH (n) DETACH DELETE n") + await cleanup_driver.close() + + +# --------------------------------------------------------------------------- +# T-orphan / T-referenced / T-workspace-safety / T-in-flight-recent +# --------------------------------------------------------------------------- + + +async def test_orphan_listed_in_dry_run_and_deleted_on_apply( + admin_client: httpx.AsyncClient, tmp_path: Path +) -> None: + """T-orphan: unreferenced, old, drained blob -> dry-run lists it, apply deletes it.""" + blob_dir = tmp_path / "blobs" + sid = "orphan-sess-1" + path = _write_blob(blob_dir, sid, "node1__result", age_seconds=_OLD_AGE_SECONDS) + uri = f"ci-blob://{sid}/node1__result" + + dry = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 15} + ) + assert dry.status_code == 200 + dry_body = dry.json() + assert dry_body["dry_run"] is True + assert dry_body["orphans_found"] == 1 + assert uri in dry_body["sample"] + assert dry_body["reclaimable_bytes"] > 0 + assert dry_body["rescanned"] is False + assert dry_body["deleted"] == 0 + assert path.exists(), "dry-run must never delete" + + apply_resp = await admin_client.post( + "/admin/blobs/reclaim", + json={"dry_run": False, "min_age_minutes": 15, "max_delete": 10}, + ) + assert apply_resp.status_code == 200 + apply_body = apply_resp.json() + assert apply_body["dry_run"] is False + assert apply_body["rescanned"] is True + assert apply_body["orphans_found"] == 1 + assert apply_body["deleted"] == 1 + assert apply_body["deleted_bytes"] > 0 + assert not path.exists(), "apply must delete the orphan file" + + +async def test_referenced_blob_never_a_candidate( + admin_client: httpx.AsyncClient, + tmp_path: Path, + neo4j_driver: Any, +) -> None: + """T-referenced: a blob referenced by :Event.data is never a candidate.""" + blob_dir = tmp_path / "blobs" + sid = "referenced-sess-1" + path = _write_blob(blob_dir, sid, "node1__result", age_seconds=_OLD_AGE_SECONDS) + uri = f"ci-blob://{sid}/node1__result" + + await _create_event_node( + neo4j_driver, + node_id="evt-ref-1", + workspace="ws-a", + data={"session_id": sid, "result": {"$blob_ref": uri}}, + ) + + resp = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 15} + ) + assert resp.status_code == 200 + body = resp.json() + assert body["referenced_uris"] >= 1 + assert body["orphans_found"] == 0 + assert uri not in body["sample"] + + apply_resp = await admin_client.post( + "/admin/blobs/reclaim", + json={"dry_run": False, "min_age_minutes": 15, "max_delete": 10}, + ) + assert apply_resp.status_code == 200 + assert apply_resp.json()["deleted"] == 0 + assert path.exists(), "a referenced blob must never be deleted" + + +async def test_cross_workspace_reference_still_protects_blob( + admin_client: httpx.AsyncClient, + tmp_path: Path, + neo4j_driver: Any, +) -> None: + """T-workspace-safety: reference from a DIFFERENT workspace still protects the blob. + + Proves the reference scan is global (never workspace-filtered) -- a + per-workspace scan would wrongly treat this blob as orphaned. + """ + blob_dir = tmp_path / "blobs" + sid = "cross-ws-sess-1" + path = _write_blob(blob_dir, sid, "node1__result", age_seconds=_OLD_AGE_SECONDS) + uri = f"ci-blob://{sid}/node1__result" + + # The referencing node lives in "workspace-b" -- a DIFFERENT workspace + # than any the reclaim request could plausibly scope to (the endpoint + # accepts no workspace parameter at all -- the scan is always global). + await _create_event_node( + neo4j_driver, + node_id="evt-cross-ws-1", + workspace="workspace-b", + data={"session_id": sid, "result": {"$blob_ref": uri}}, + ) + + resp = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 15} + ) + assert resp.status_code == 200 + body = resp.json() + assert body["orphans_found"] == 0 + assert uri not in body["sample"] + assert path.exists() + + +async def test_fresh_blob_skipped_as_recent( + admin_client: httpx.AsyncClient, tmp_path: Path +) -> None: + """T-in-flight-recent: a freshly-written, unreferenced blob is excluded via skipped_recent.""" + blob_dir = tmp_path / "blobs" + sid = "fresh-sess-1" + path = _write_blob(blob_dir, sid, "node1__result", age_seconds=0.0) + + resp = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 60} + ) + assert resp.status_code == 200 + body = resp.json() + assert body["orphans_found"] == 0 + assert body["skipped_recent"] >= 1 + assert path.exists() + + +# --------------------------------------------------------------------------- +# T-pending-queue (B3 -- undrained-queue gate, both durable-log and +# live-worker OR clauses) +# --------------------------------------------------------------------------- + + +async def test_undrained_queue_skips_even_old_blob( + admin_client: httpx.AsyncClient, tmp_path: Path +) -> None: + """T-pending-queue (durable): an undrained .log skips the blob even though + it is old -- the durable gate takes priority over the age gate.""" + from context_intelligence_server.main import registry as shared_registry + + blob_dir = tmp_path / "blobs" + sid = "pending-queue-sess-1" + path = _write_blob(blob_dir, sid, "node1__result", age_seconds=_OLD_AGE_SECONDS) + + # Append WITHOUT committing -- committed offset (0) stays behind the + # complete-data end, so is_fully_drained(sid) is False. + qm = shared_registry.queue_manager + await qm.append( + sid, + json.dumps( + { + "event": "tool:pre", + "workspace": "w", + "data": {"session_id": sid, "timestamp": "2024-01-01T00:00:00+00:00"}, + } + ).encode("utf-8"), + ) + assert not await qm.is_fully_drained(sid), "test setup: queue must be undrained" + + resp = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 15} + ) + assert resp.status_code == 200 + body = resp.json() + assert body["orphans_found"] == 0 + assert body["skipped_pending_session"] >= 1 + assert body["skipped_recent"] == 0, ( + "an old blob gated by the durable undrained-queue check must be " + "counted as skipped_pending_session, NOT skipped_recent" + ) + assert path.exists() + + +async def test_live_worker_skips_even_drained_old_blob( + admin_client: httpx.AsyncClient, tmp_path: Path +) -> None: + """T-pending-queue (live-worker OR clause): a registered live worker skips + the blob even when its queue has no undrained data at all (drained).""" + from context_intelligence_server.main import registry as shared_registry + + blob_dir = tmp_path / "blobs" + sid = "live-worker-sess-1" + path = _write_blob(blob_dir, sid, "node1__result", age_seconds=_OLD_AGE_SECONDS) + + # No .log file exists for this session at all -- is_fully_drained(sid) + # reads True (0 >= 0). Only the live-worker registration should gate it. + qm = shared_registry.queue_manager + assert await qm.is_fully_drained(sid), "test setup: queue must read as drained" + + worker = SessionWorker( + session_id=sid, workspace="w", services=HookStateService(workspace="w") + ) + shared_registry._register_for_test(worker) + assert sid in shared_registry.active_sessions() + + resp = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 15} + ) + assert resp.status_code == 200 + body = resp.json() + assert body["orphans_found"] == 0 + assert body["skipped_pending_session"] >= 1 + assert path.exists() + + +# --------------------------------------------------------------------------- +# T-b2-specialchars (structural JSON extraction, not regex) +# --------------------------------------------------------------------------- + + +async def test_special_characters_in_session_id_classified_referenced( + admin_client: httpx.AsyncClient, + tmp_path: Path, + neo4j_driver: Any, +) -> None: + """T-b2-specialchars: a session_id containing a literal quote and a + non-ASCII character, when referenced, MUST be classified referenced. + + The named regex pattern from the pre-amendment design (``ci-blob://[^"\\\\]+``) + truncates at the first unescaped `"` in the SERIALIZED JSON string, + misclassifying this exact case as orphan. Structural `json.loads` + + recursive walk (B2) handles it correctly because the quote and the + non-ASCII character are just ordinary characters in the DECODED string -- + json.loads has already resolved all escaping before the walk runs. + """ + blob_dir = tmp_path / "blobs" + # A literal double-quote (queue_manager._validate_session_id blocks only + # '/', '\\', and '\\0' -- NOT '"') and a non-ASCII character (û). + sid = 'weird"session-û' + path = _write_blob(blob_dir, sid, "node1__result", age_seconds=_OLD_AGE_SECONDS) + uri = f"ci-blob://{sid}/node1__result" + + await _create_event_node( + neo4j_driver, + node_id="evt-special-1", + workspace="ws-special", + data={"session_id": sid, "result": {"$blob_ref": uri}}, + ) + + resp = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 15} + ) + assert resp.status_code == 200 + body = resp.json() + assert body["referenced_uris"] >= 1 + assert body["orphans_found"] == 0, ( + f"special-character URI {uri!r} must be classified referenced via " + "structural JSON extraction -- a regex-based extractor would " + "truncate at the embedded quote and misclassify it as orphan" + ) + assert uri not in body["sample"] + assert path.exists() + + +# --------------------------------------------------------------------------- +# T-dry-apply-parity / T-idempotence / T-max-delete-required / T-max-delete-cap +# --------------------------------------------------------------------------- + + +async def test_dry_run_and_apply_select_the_same_candidate_set( + admin_client: httpx.AsyncClient, tmp_path: Path +) -> None: + """T-dry-apply-parity: dry-run's candidate set == apply's deleted set for + identical fixtures (proves the ONE shared _select_orphans path).""" + blob_dir = tmp_path / "blobs" + sids_and_keys = [ + ("parity-sess-1", "n1__result"), + ("parity-sess-2", "n2__result"), + ("parity-sess-3", "n3__result"), + ] + expected_uris = set() + for sid, key in sids_and_keys: + _write_blob(blob_dir, sid, key, age_seconds=_OLD_AGE_SECONDS) + expected_uris.add(f"ci-blob://{sid}/{key}") + + dry = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 15} + ) + dry_body = dry.json() + assert set(dry_body["sample"]) == expected_uris + assert dry_body["orphans_found"] == len(expected_uris) + + apply_resp = await admin_client.post( + "/admin/blobs/reclaim", + json={"dry_run": False, "min_age_minutes": 15, "max_delete": 100}, + ) + apply_body = apply_resp.json() + assert set(apply_body["sample"]) == expected_uris + assert apply_body["deleted"] == len(expected_uris) + + +async def test_second_apply_is_idempotent( + admin_client: httpx.AsyncClient, tmp_path: Path +) -> None: + """T-idempotence: a second apply call deletes zero (files already gone).""" + blob_dir = tmp_path / "blobs" + sid = "idempotent-sess-1" + _write_blob(blob_dir, sid, "node1__result", age_seconds=_OLD_AGE_SECONDS) + + first = await admin_client.post( + "/admin/blobs/reclaim", + json={"dry_run": False, "min_age_minutes": 15, "max_delete": 10}, + ) + assert first.json()["deleted"] == 1 + + second = await admin_client.post( + "/admin/blobs/reclaim", + json={"dry_run": False, "min_age_minutes": 15, "max_delete": 10}, + ) + second_body = second.json() + assert second_body["orphans_found"] == 0 + assert second_body["deleted"] == 0 + + +async def test_apply_without_max_delete_is_422( + admin_client: httpx.AsyncClient, +) -> None: + """T-max-delete-required: dry_run=false without max_delete -> 422.""" + resp = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": False, "min_age_minutes": 15} + ) + assert resp.status_code == 422 + + +async def test_max_delete_cap_is_honored( + admin_client: httpx.AsyncClient, tmp_path: Path +) -> None: + """T-max-delete-cap: totals stay authoritative while the cap limits deletions.""" + blob_dir = tmp_path / "blobs" + for i in range(3): + _write_blob( + blob_dir, f"cap-sess-{i}", "n__result", age_seconds=_OLD_AGE_SECONDS + ) + + resp = await admin_client.post( + "/admin/blobs/reclaim", + json={"dry_run": False, "min_age_minutes": 15, "max_delete": 1}, + ) + assert resp.status_code == 200 + body = resp.json() + assert body["orphans_found"] == 3, "totals must reflect the FULL candidate set" + assert body["deleted"] == 1, "deletions must be capped at max_delete" + assert body["reclaimable_bytes"] > body["deleted_bytes"] or body["deleted"] == 0 + + +# --------------------------------------------------------------------------- +# T-b1-invariant (Event.data is the complete reference carrier) +# --------------------------------------------------------------------------- + + +async def test_b1_event_data_carries_every_blob_ref( + neo4j_services: Any, +) -> None: + """T-b1-invariant: an event carrying a $blob_ref-shaped value in tool_input + AND in each BLOB_FIELD must have every exact URI present in the + persisted :Event.data -- pinning the invariant the reclaim scan depends + on (B1 of the council amendment). + + Uses DefaultHandler directly against a real Neo4j-backed HookStateService + (``neo4j_services`` fixture, tests/neo4j/conftest.py) so this is a real + handler run, not a synthetic Event.data string. + """ + handler = DefaultHandler(neo4j_services) + session_id = "b1-invariant-sess-1" + tool_input_ref = "ci-blob://b1-invariant-sess-1/pre-existing-blob" + + # blob_processor.BLOB_FIELDS: {"raw", "result", "messages", "mount_plan", + # "context_snapshot", "debug"}. Simulate each already offloaded to a + # $blob_ref (as blob_processor.process_event_data would have done + # upstream of DefaultHandler in the real pipeline). + blob_field_refs = { + field: f"ci-blob://{session_id}/node1__{field}" + for field in ( + "raw", + "result", + "messages", + "mount_plan", + "context_snapshot", + "debug", + ) + } + + data: dict[str, Any] = { + "session_id": session_id, + "timestamp": "2024-01-01T00:00:00+00:00", + "tool_input": {"$blob_ref": tool_input_ref}, + **{k: {"$blob_ref": v} for k, v in blob_field_refs.items()}, + } + + await handler("tool:pre", data) + await neo4j_services.graph.flush() + + driver = neo4j_services.graph._driver + async with driver.session() as session: + result = await session.run( + "MATCH (n:Event) WHERE n.event_name = 'tool:pre' " + "AND n.data CONTAINS $sid RETURN n.data AS data", + {"sid": session_id}, + ) + rows = [record["data"] async for record in result] + + assert len(rows) == 1, f"expected exactly one persisted Event node, got {rows}" + persisted = json.loads(rows[0]) + + assert persisted["tool_input"]["$blob_ref"] == tool_input_ref + for field, expected_uri in blob_field_refs.items(): + assert persisted[field]["$blob_ref"] == expected_uri, ( + f"BLOB_FIELD {field!r} ref must survive verbatim into persisted " + "Event.data -- this is the invariant the reclaim scan relies on" + ) + + +# --------------------------------------------------------------------------- +# Reference-scan hardening (docs/plans/2026-08-12-blob-reclaim-reference- +# scan-hardening.md): a ref living ONLY on a non-Event.data carrier property +# must still be classified referenced. Each of these tests seeds NO :Event +# node at all -- proving the new per-property UNION branches (not the +# pre-existing Event.data branch) find the reference. Every one of these +# FAILS against the old Event.data-only scan and PASSES after the hardening. +# --------------------------------------------------------------------------- + + +async def test_toolcall_tool_input_only_reference_is_protected( + admin_client: httpx.AsyncClient, + tmp_path: Path, + neo4j_driver: Any, +) -> None: + """A $blob_ref-shaped reference living ONLY on ToolCall.tool_input (no + Event.data anywhere containing it) must be classified referenced. + + Structural-extraction path: tool_input is stored as the JSON string + ``{"$blob_ref": ""}`` (neo4j_store._sanitize_properties + JSON-serializes dict property values on write), so this pins the + json.loads + _collect_blob_refs branch of _extract_blob_refs_from_value + for a non-``data`` carrier. + """ + blob_dir = tmp_path / "blobs" + sid = "toolcall-only-sess-1" + path = _write_blob(blob_dir, sid, "node1__result", age_seconds=_OLD_AGE_SECONDS) + uri = f"ci-blob://{sid}/node1__result" + + await _create_carrier_node( + neo4j_driver, + label="ToolCall", + node_id="toolcall-only-1", + workspace="ws-toolcall", + prop_name="tool_input", + prop_value=json.dumps({"$blob_ref": uri}), + ) + + resp = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 15} + ) + assert resp.status_code == 200 + body = resp.json() + assert body["referenced_uris"] >= 1 + assert body["orphans_found"] == 0, ( + f"{uri!r} is referenced only via ToolCall.tool_input -- an " + "Event.data-only scan would misclassify it as orphan and delete it" + ) + assert uri not in body["sample"] + assert path.exists() + + +async def test_plain_string_tool_input_carrier_is_protected( + admin_client: httpx.AsyncClient, + tmp_path: Path, + neo4j_driver: Any, +) -> None: + """A bare-string tool_input (NOT the {"$blob_ref": ...} JSON wrapper) + mentioning a ci-blob:// URI must still be classified referenced. + + Regex-fallback path: a plain string is written through verbatim (never + JSON-serialized), so json.loads on it raises and + _extract_blob_refs_from_value falls back to the bare ci-blob://[^"\\s]+ + token regex. Pins that fallback for a lifted plain-string carrier. + """ + blob_dir = tmp_path / "blobs" + sid = "plain-string-sess-1" + path = _write_blob(blob_dir, sid, "node1__result", age_seconds=_OLD_AGE_SECONDS) + uri = f"ci-blob://{sid}/node1__result" + + await _create_carrier_node( + neo4j_driver, + label="ToolCall", + node_id="plain-string-toolcall-1", + workspace="ws-plain", + prop_name="tool_input", + prop_value=f"see {uri} for the prior result", + ) + + resp = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 15} + ) + assert resp.status_code == 200 + body = resp.json() + assert body["referenced_uris"] >= 1 + assert body["orphans_found"] == 0, ( + f"{uri!r} is referenced only via a plain-string tool_input -- the " + "regex fallback must extract it even though the property is not " + "valid JSON" + ) + assert uri not in body["sample"] + assert path.exists() + + +async def test_prompt_prompt_only_reference_is_protected( + admin_client: httpx.AsyncClient, + tmp_path: Path, + neo4j_driver: Any, +) -> None: + """A $blob_ref-shaped reference living ONLY on Prompt.prompt (no + Event.data anywhere containing it) must be classified referenced.""" + blob_dir = tmp_path / "blobs" + sid = "prompt-only-sess-1" + path = _write_blob(blob_dir, sid, "node1__result", age_seconds=_OLD_AGE_SECONDS) + uri = f"ci-blob://{sid}/node1__result" + + await _create_carrier_node( + neo4j_driver, + label="Prompt", + node_id="prompt-only-1", + workspace="ws-prompt", + prop_name="prompt", + prop_value=json.dumps({"$blob_ref": uri}), + ) + + resp = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 15} + ) + assert resp.status_code == 200 + body = resp.json() + assert body["referenced_uris"] >= 1 + assert body["orphans_found"] == 0, ( + f"{uri!r} is referenced only via Prompt.prompt -- an Event.data-only " + "scan would misclassify it as orphan and delete it" + ) + assert uri not in body["sample"] + assert path.exists() + + +async def test_orchestrator_run_response_only_reference_is_protected( + admin_client: httpx.AsyncClient, + tmp_path: Path, + neo4j_driver: Any, +) -> None: + """A $blob_ref-shaped reference living ONLY on OrchestratorRun.response + (no Event.data anywhere containing it) must be classified referenced.""" + blob_dir = tmp_path / "blobs" + sid = "orchrun-only-sess-1" + path = _write_blob(blob_dir, sid, "node1__result", age_seconds=_OLD_AGE_SECONDS) + uri = f"ci-blob://{sid}/node1__result" + + await _create_carrier_node( + neo4j_driver, + label="OrchestratorRun", + node_id="orchrun-only-1", + workspace="ws-orchrun", + prop_name="response", + prop_value=json.dumps({"$blob_ref": uri}), + ) + + resp = await admin_client.post( + "/admin/blobs/reclaim", json={"dry_run": True, "min_age_minutes": 15} + ) + assert resp.status_code == 200 + body = resp.json() + assert body["referenced_uris"] >= 1 + assert body["orphans_found"] == 0, ( + f"{uri!r} is referenced only via OrchestratorRun.response -- an " + "Event.data-only scan would misclassify it as orphan and delete it" + ) + assert uri not in body["sample"] + assert path.exists() diff --git a/tests/neo4j/test_cursor_durability_e2e.py b/tests/neo4j/test_cursor_durability_e2e.py new file mode 100644 index 00000000..4b571923 --- /dev/null +++ b/tests/neo4j/test_cursor_durability_e2e.py @@ -0,0 +1,707 @@ +"""Tier 3 -- REAL Neo4j end-to-end proof for I5b (durable handler cursor). + +Reproduces a realistic "server restart mid-session" / "stale-session reap +mid-session" against a LIVE Neo4j container and asserts the spec's Sec 10.3 +DTU acceptance criteria: + + 1. Duplicate Iteration nodes == 0 across the restart (no bare-shape + re-pooling). + 2. Edge parity: E06 HAS_PART, E09 CAUSED, E14 TRIGGERS, E15 ENABLES present + across the restart boundary (proves FULL-cursor persistence -- not just + the node_id -- because E09 specifically requires + ``pending_tool_block_ids`` to have survived the rebuild). + 3. ``iteration_scope`` tallies: every Iteration node carries "run" or + "unscoped"; none are missing/neither. + +This test drives the REAL code path a crash-restart/reap hits: +``get_or_create -> start_drain -> drain_worker -> restore_cursor(read_cursor)``. +It does NOT use ``_register_for_test`` to bypass ``get_or_create`` (unlike +most other tests/neo4j/ files) -- the whole point here is to exercise +``SessionRegistry.get_or_create``'s settings-derived construction path, which +is where BOTH triggers (T1 crash-restart via main.py's recovery loop, T2 +stale-reap via drain_worker's own idle branch) actually spawn their rebuilt +worker in production. + +Two scenarios: + - test_cursor_durability_survives_crash_restart_e2e (T1): the pre-restart + registry/worker object is simply discarded (its drain task cancelled) + WITHOUT calling delete_drained -- .log/.offset survive on disk exactly + as they would after a process crash. A brand-new SessionRegistry, over + the SAME on-disk queues dir and the SAME Neo4j, picks the session back + up. + - test_cursor_durability_survives_stale_reap_e2e (T2): the registry's own + ``_deregister`` is called directly (mirrors the real idle-reap branch in + ``drain_worker``, which calls ``_safe_close`` + ``_deregister`` + returns) + -- this is the sharper trigger because it has NO recovery path today + without I5b: ``_deregister`` intentionally leaves .log/.offset on disk. + +Run: + uv run pytest tests/neo4j/test_cursor_durability_e2e.py -v -m neo4j +""" + +from __future__ import annotations + +import asyncio +import json +from pathlib import Path +from typing import Any + +import pytest +from context_intelligence_server.config import Neo4jClientConfig +from context_intelligence_server.neo4j_store import ( + ensure_neo4j_schema, + ensure_schema_version_baseline, +) +from context_intelligence_server.registry import SessionRegistry, SessionWorker +from context_intelligence_server.status import SCHEMA_VERSION +from neo4j import AsyncGraphDatabase + +pytestmark = pytest.mark.neo4j + +WORKSPACE = "cursor-durability-e2e" + +# Fixed timestamps -- deterministic node_ids, no wall-clock flakiness. +T0 = "2026-08-11T09:00:00+00:00" # session:start +TP1 = "2026-08-11T09:00:01+00:00" # prompt:submit #1 +T1 = "2026-08-11T10:00:00+00:00" # execution:start (the run-scoping ts) +TPR1 = "2026-08-11T10:00:01+00:00" # provider:request iter1 +TLQ1 = "2026-08-11T10:00:02+00:00" # llm:request iter1 +TLR1 = "2026-08-11T10:00:03+00:00" # llm:response iter1 +TCB0S = "2026-08-11T10:00:04+00:00" # content_block:start block0 +TCB0E = "2026-08-11T10:00:05+00:00" # content_block:end block0 (tool_call) +TPR2 = "2026-08-11T10:00:06+00:00" # provider:request iter2 +TLQ2 = "2026-08-11T10:00:07+00:00" # llm:request iter2 +TLR2 = "2026-08-11T10:00:08+00:00" # llm:response iter2 + +# --- post-restart timestamps --- +TTPRE = "2026-08-11T10:05:00+00:00" # tool:pre (fires E09 -- pending_tool_block_ids) +TTPOST = "2026-08-11T10:05:01+00:00" # tool:post +TPR3 = "2026-08-11T10:05:02+00:00" # provider:request iter3 (continuation, not reset) +TLQ3 = "2026-08-11T10:05:03+00:00" # llm:request iter3 +TLR3 = "2026-08-11T10:05:04+00:00" # llm:response iter3 +TOC = "2026-08-11T10:05:05+00:00" # orchestrator:complete +TP2 = "2026-08-11T10:05:06+00:00" # prompt:submit #2 (fires E15) + +TOOL_CALL_ID = "toolblock-1" + + +def _line(event: str, workspace: str, data: dict[str, Any]) -> bytes: + """Encode an appended event line exactly as POST /events stores it.""" + return json.dumps({"event": event, "workspace": workspace, "data": data}).encode( + "utf-8" + ) + + +def _pre_restart_lines(sid: str) -> list[bytes]: + """session:start .. provider:request/llm:*(iter2) -- the pre-restart half. + + Deliberately stops with content_block:end (tool_call) cached in + ``pending_tool_block_ids`` but WITHOUT the matching tool:pre -- that pop + happens post-restart, so E09 can only be created if the FULL DataLayer2State + (not just execution_start_ts/iteration_count) survived the rebuild. + """ + return [ + _line("session:start", WORKSPACE, {"session_id": sid, "timestamp": T0}), + _line( + "prompt:submit", + WORKSPACE, + {"session_id": sid, "timestamp": TP1, "prompt": "do the thing"}, + ), + _line("execution:start", WORKSPACE, {"session_id": sid, "timestamp": T1}), + _line( + "provider:request", + WORKSPACE, + {"session_id": sid, "timestamp": TPR1}, + ), + _line( + "llm:request", + WORKSPACE, + { + "session_id": sid, + "timestamp": TLQ1, + "provider": "anthropic", + "model": "claude", + "message_count": 1, + }, + ), + _line( + "llm:response", + WORKSPACE, + { + "session_id": sid, + "timestamp": TLR1, + "usage": {"input_tokens": 10, "output_tokens": 5}, + }, + ), + _line( + "content_block:start", + WORKSPACE, + {"session_id": sid, "timestamp": TCB0S, "block_index": 0}, + ), + _line( + "content_block:end", + WORKSPACE, + { + "session_id": sid, + "timestamp": TCB0E, + "block_index": 0, + "block": {"type": "tool_call", "id": TOOL_CALL_ID}, + }, + ), + _line( + "provider:request", + WORKSPACE, + {"session_id": sid, "timestamp": TPR2}, + ), + _line( + "llm:request", + WORKSPACE, + { + "session_id": sid, + "timestamp": TLQ2, + "provider": "anthropic", + "model": "claude", + "message_count": 2, + }, + ), + _line( + "llm:response", + WORKSPACE, + { + "session_id": sid, + "timestamp": TLR2, + "usage": {"input_tokens": 20, "output_tokens": 8}, + }, + ), + ] + + +def _post_restart_lines(sid: str) -> list[bytes]: + """tool:pre/post(iter2's block) -> provider:request(iter3) -> orchestrator:complete + -> prompt:submit#2 -- the post-restart half, all for the SAME orch run T1. + """ + return [ + _line( + "tool:pre", + WORKSPACE, + { + "session_id": sid, + "timestamp": TTPRE, + "tool_call_id": TOOL_CALL_ID, + "tool_name": "bash", + "tool_input": "echo hi", + }, + ), + _line( + "tool:post", + WORKSPACE, + { + "session_id": sid, + "timestamp": TTPOST, + "tool_call_id": TOOL_CALL_ID, + "result": {"output": "hi"}, + }, + ), + _line( + "provider:request", + WORKSPACE, + {"session_id": sid, "timestamp": TPR3}, + ), + _line( + "llm:request", + WORKSPACE, + { + "session_id": sid, + "timestamp": TLQ3, + "provider": "anthropic", + "model": "claude", + "message_count": 3, + }, + ), + _line( + "llm:response", + WORKSPACE, + { + "session_id": sid, + "timestamp": TLR3, + "usage": {"input_tokens": 30, "output_tokens": 12}, + }, + ), + _line( + "orchestrator:complete", + WORKSPACE, + { + "session_id": sid, + "timestamp": TOC, + "orchestrator": "test-orchestrator", + "turn_count": 1, + }, + ), + _line( + "prompt:submit", + WORKSPACE, + {"session_id": sid, "timestamp": TP2, "prompt": "do another thing"}, + ), + ] + + +class _SettingsProxy: + """Minimal settings stand-in pointed at the REAL Neo4j fixture + a tmp queues dir. + + ``SessionRegistry.get_or_create`` calls ``settings.resolve_neo4j_admin()`` + directly (doc 12, the Neo4j two-client split) and reads several scalar + fields off ``get_settings()`` -- this mirrors tests/conftest.py's + ``safe_settings`` proxy shape exactly, but points ``neo4j_url`` / + ``neo4j_user`` / ``neo4j_password`` at the LIVE test container instead of + the (unreachable) real default settings, so ``get_or_create`` builds a + genuine ``Neo4jGraphStore`` -- not a stub, not ``_register_for_test``. + """ + + def __init__( + self, queues_dir: Path, blob_dir: Path, container: dict[str, Any] + ) -> None: + self.queues_path = str(queues_dir) + self.blob_path = str(blob_dir) + self.neo4j_url = container["bolt_url"] + self.neo4j_user = container["user"] + self.neo4j_password = container["password"] + self.stale_session_timeout = 3600.0 + self.write_concurrency = 4 + self.max_delivery_attempts = 3 + self.neo4j_flush_chunk_rows = 100 + self.neo4j_flush_chunk_bytes = 4_194_304 + self.neo4j_lock_timeout: float | None = None + + def resolve_neo4j_admin(self) -> Neo4jClientConfig: + return Neo4jClientConfig( + url=self.neo4j_url, + username=self.neo4j_user, + password=self.neo4j_password, + access_mode="WRITE", + ) + + +async def _drain_until( + predicate: Any, + *, + timeout: float = 30.0, + interval: float = 0.05, +) -> bool: + """Poll *predicate* (a zero-arg callable) until truthy or *timeout* elapses.""" + loop = asyncio.get_event_loop() + deadline = loop.time() + timeout + while True: + if predicate(): + return True + if loop.time() >= deadline: + return False + await asyncio.sleep(interval) + + +async def _cancel_and_await(worker: SessionWorker) -> None: + """Cancel a worker's drain task and await its (CancelledError) completion.""" + if worker.task is None: + return + worker.task.cancel() + try: + await worker.task + except asyncio.CancelledError: + pass + + +async def _append_all(qm: Any, sid: str, lines: list[bytes]) -> None: + for raw in lines: + await qm.append(sid, raw) + + +def _make_settings_proxy( + tmp_path: Path, neo4j_container: dict[str, Any] +) -> _SettingsProxy: + return _SettingsProxy( + queues_dir=tmp_path / "queues", + blob_dir=tmp_path / "blobs", + container=neo4j_container, + ) + + +async def _run_query( + neo4j_container: dict[str, Any], query: str, **params: Any +) -> list[Any]: + """One-shot query against the live container; returns all result rows.""" + driver = AsyncGraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + try: + async with driver.session() as session: + result = await session.run(query, params) + return [record async for record in result] + finally: + await driver.close() + + +# --------------------------------------------------------------------------- +# T1 -- crash restart +# --------------------------------------------------------------------------- + + +async def test_cursor_durability_survives_crash_restart_e2e( + neo4j_container: dict[str, Any], + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + sid = "cursor-e2e-crash" + orch_run_id = f"{sid}::orch_run::{T1}" + + # Schema (indexes/constraints) active before any MERGE -- mirrors lifespan. + admin_driver = AsyncGraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + try: + await ensure_neo4j_schema(admin_driver) + await ensure_schema_version_baseline(admin_driver) + finally: + await admin_driver.close() + + proxy = _make_settings_proxy(tmp_path, neo4j_container) + monkeypatch.setattr( + "context_intelligence_server.registry.get_settings", lambda: proxy + ) + + # --------------------------------------------------------------- + # PHASE 1 -- pre-restart: real registry_A, real get_or_create, real store. + # --------------------------------------------------------------- + reg_a = SessionRegistry() + worker_a = reg_a.get_or_create(sid, WORKSPACE) + assert worker_a.task is not None + + pre_lines = _pre_restart_lines(sid) + await _append_all(reg_a.queue_manager, sid, pre_lines) + + wrote = await _drain_until( + lambda: reg_a.pipeline_counters()["written_total"] >= len(pre_lines), + timeout=30.0, + ) + assert wrote, "pre-restart batch did not commit within the window" + + # --- Evidence requirement: .offset now contains a JSON record with a + # non-null cursor (execution_start_ts=T1, iteration_count>0). --- + offset_path = tmp_path / "queues" / f"{sid}.offset" + rec = json.loads(offset_path.read_text(encoding="utf-8")) + assert rec["v"] == 1 + assert rec["cursor"] is not None, ".offset cursor must be non-null after commit" + assert rec["cursor"]["dl2"]["execution_start_ts"] == T1 + assert rec["cursor"]["dl2"]["iteration_count"] == 2 + assert rec["cursor"]["dl2"]["pending_tool_block_ids"] == { + TOOL_CALL_ID: f"{sid}::block::1::0" + } + + # --------------------------------------------------------------- + # "CRASH": discard reg_a/worker_a WITHOUT delete_drained -- .log/.offset + # survive on disk. Cancel the task (test-harness cleanup only; production + # would have the OS kill the process instead). + # --------------------------------------------------------------- + await _cancel_and_await(worker_a) + + # --------------------------------------------------------------- + # PHASE 2 -- "restart": a BRAND NEW SessionRegistry + fresh + # HookStateService/DataLayer2State + new Neo4jGraphStore over the SAME + # queues dir + SAME Neo4j, via get_or_create -> start_drain -> drain_worker. + # --------------------------------------------------------------- + # Mirrors lifespan calling ensure_schema_version_baseline on every + # "startup" -- a fresh admin driver, used once, then closed. + restart_driver = AsyncGraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + try: + await ensure_schema_version_baseline(restart_driver) + finally: + await restart_driver.close() + + reg_b = SessionRegistry() + worker_b = reg_b.get_or_create(sid, WORKSPACE) + assert worker_b.task is not None + + # HONESTY CHECK: the fresh worker's DataLayer2State is genuinely at + # defaults THIS INSTANT -- the drain task has been scheduled but the + # event loop has not yet given it a turn to run restore_cursor. If this + # assertion ever fails, the "fresh state" claim below is not proven. + assert worker_b.services.data_layer_2.execution_start_ts is None + assert worker_b.services.data_layer_2.iteration_count == 0 + assert worker_b.services.data_layer_2.pending_tool_block_ids == {} + + # Let the drain task actually run far enough to call restore_cursor + # (read_cursor is asyncio.to_thread -- needs a real await, not just + # asyncio.sleep(0)). + restored = await _drain_until( + lambda: worker_b.services.data_layer_2.execution_start_ts is not None, + timeout=10.0, + ) + assert restored, "restore_cursor did not populate execution_start_ts in time" + assert worker_b.services.data_layer_2.execution_start_ts == T1 + assert worker_b.services.data_layer_2.iteration_count == 2 + assert worker_b.services.data_layer_2.pending_tool_block_ids == { + TOOL_CALL_ID: f"{sid}::block::1::0" + } + + # --------------------------------------------------------------- + # PHASE 3 -- append MORE events for the SAME run after the restart. + # --------------------------------------------------------------- + post_lines = _post_restart_lines(sid) + await _append_all(reg_b.queue_manager, sid, post_lines) + + wrote2 = await _drain_until( + lambda: reg_b.pipeline_counters()["written_total"] >= len(post_lines), + timeout=30.0, + ) + assert wrote2, "post-restart batch did not commit within the window" + + await _cancel_and_await(worker_b) + + # ================================================================= + # Sec 10.3 acceptance-criteria assertions (real Cypher, real Neo4j) + # ================================================================= + iter_rows = await _run_query( + neo4j_container, + "MATCH (i:Iteration {session_id: $sid}) " + "RETURN i.node_id AS node_id, i.iteration_number AS n, " + "i.iteration_scope AS scope ORDER BY i.iteration_number", + sid=sid, + ) + + # (a) Duplicate-free. + node_ids = [r["node_id"] for r in iter_rows] + assert len(node_ids) == len(set(node_ids)), ( + f"duplicate Iteration node_ids for session {sid}: {node_ids}" + ) + bare = [nid for nid in node_ids if "::orch_run::" not in nid] + assert bare == [], f"bare (pre-fix-shape) Iteration node_ids found: {bare}" + assert len(node_ids) == 3, f"expected exactly 3 Iteration nodes, got {node_ids}" + + # (b) Run-scoped continuity: same run prefix, iteration_number continues + # 1, 2, 3 across the restart boundary (no restart-to-1 collision). + expected_prefix = f"{orch_run_id}::iteration::" + for r in iter_rows: + assert r["node_id"].startswith(expected_prefix), ( + f"Iteration {r['node_id']} is not scoped to {orch_run_id}" + ) + assert [r["n"] for r in iter_rows] == [1, 2, 3], ( + f"iteration_number must continue 1,2,3 across the restart, got " + f"{[r['n'] for r in iter_rows]}" + ) + + # (d) iteration_scope: every Iteration node carries a value (never + # missing/null), and since execution_start_ts was active throughout, + # all three must be "run" (never "unscoped"). + scopes = [r["scope"] for r in iter_rows] + assert all(s is not None for s in scopes), f"missing iteration_scope: {scopes}" + assert scopes == ["run", "run", "run"], f"expected all-'run' scopes, got {scopes}" + + # (c) Edge parity across the restart boundary. + iter3_id = f"{orch_run_id}::iteration::3" + + e06_rows = await _run_query( + neo4j_container, + "MATCH (o:OrchestratorRun {node_id: $orid})-[r:HAS_PART]->(i:Iteration) " + "WHERE i.node_id = $iter3 RETURN count(r) AS c", + orid=orch_run_id, + iter3=iter3_id, + ) + assert e06_rows[0]["c"] >= 1, ( + "E06 HAS_PART edge missing from the post-restart (iteration 3) " + "OrchestratorRun -> Iteration" + ) + + # E09: ContentBlock -[:CAUSED]-> ToolCall. This can ONLY exist if + # pending_tool_block_ids (cached pre-restart at content_block:end) + # survived the rebuild and was consumed by the post-restart tool:pre -- + # i.e. it proves FULL DataLayer2State restore, not just the node_id. + e09_rows = await _run_query( + neo4j_container, + "MATCH (b:ContentBlock {session_id: $sid})-[r:CAUSED]->(t:ToolCall) " + "WHERE t.tool_call_id = $tcid RETURN count(r) AS c", + sid=sid, + tcid=TOOL_CALL_ID, + ) + assert e09_rows[0]["c"] >= 1, ( + "E09 CAUSED edge missing -- pending_tool_block_ids did not survive " + "the worker rebuild" + ) + + # E14: Prompt -[:TRIGGERS]-> OrchestratorRun (created at execution:start + # from the pre-restart prompt:submit's last_prompt_id cursor). + e14_rows = await _run_query( + neo4j_container, + "MATCH (p:Prompt {session_id: $sid})-[r:TRIGGERS]->(o:OrchestratorRun) " + "WHERE o.node_id = $orid RETURN count(r) AS c", + sid=sid, + orid=orch_run_id, + ) + assert e14_rows[0]["c"] >= 1, "E14 TRIGGERS edge missing" + + # E15: OrchestratorRun -[:ENABLES]-> Prompt (created at the post-restart + # second prompt:submit from orchestrator:complete's + # last_completed_orch_run_id cursor). + e15_rows = await _run_query( + neo4j_container, + "MATCH (o:OrchestratorRun {node_id: $orid})-[r:ENABLES]->(p:Prompt) " + "WHERE p.session_id = $sid RETURN count(r) AS c", + orid=orch_run_id, + sid=sid, + ) + assert e15_rows[0]["c"] >= 1, "E15 ENABLES edge missing" + + # (e) SchemaMeta: exactly ONE singleton node exists. + schema_rows = await _run_query( + neo4j_container, + "MATCH (m:SchemaMeta {id: 'singleton'}) " + "RETURN count(m) AS c, m.schema_version AS v", + ) + assert schema_rows[0]["c"] == 1, ( + f"expected exactly one :SchemaMeta singleton, got {schema_rows[0]['c']}" + ) + assert schema_rows[0]["v"] == SCHEMA_VERSION + + +# --------------------------------------------------------------------------- +# T2 -- stale-worker reap (the sharper trigger: no recovery path today) +# --------------------------------------------------------------------------- + + +async def test_cursor_durability_survives_stale_reap_e2e( + neo4j_container: dict[str, Any], + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + sid = "cursor-e2e-reap" + orch_run_id = f"{sid}::orch_run::{T1}" + + admin_driver = AsyncGraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + try: + await ensure_neo4j_schema(admin_driver) + finally: + await admin_driver.close() + + proxy = _make_settings_proxy(tmp_path, neo4j_container) + monkeypatch.setattr( + "context_intelligence_server.registry.get_settings", lambda: proxy + ) + + reg = SessionRegistry() + worker1 = reg.get_or_create(sid, WORKSPACE) + assert worker1.task is not None + + pre_lines = _pre_restart_lines(sid) + await _append_all(reg.queue_manager, sid, pre_lines) + wrote = await _drain_until( + lambda: reg.pipeline_counters()["written_total"] >= len(pre_lines), + timeout=30.0, + ) + assert wrote, "pre-reap batch did not commit within the window" + + # --------------------------------------------------------------- + # T2 trigger: the registry's OWN reap path -- _deregister pops the + # worker from the dict WITHOUT touching .log/.offset (registry.py's + # real idle-reap branch calls _safe_close + _deregister + return; we + # call _deregister directly and cancel the task ourselves to mirror + # that same-coroutine self-termination without waiting out the real + # 30s idle-detection window). + # --------------------------------------------------------------- + reg._deregister(sid) + assert sid not in reg._workers, "_deregister must have removed the worker" + await _cancel_and_await(worker1) + + # Files must still be on disk -- this IS what makes T2 have no recovery + # path without I5b. + assert (tmp_path / "queues" / f"{sid}.log").exists() + assert (tmp_path / "queues" / f"{sid}.offset").exists() + + # The "next event" rebuilds a fresh worker via the SAME registry (mirrors + # the real production flow: the next POST /events calls get_or_create + # again on a registry that no longer has this session_id). + worker2 = reg.get_or_create(sid, WORKSPACE) + assert worker2.task is not None + assert worker2 is not worker1, "get_or_create must have built a NEW worker" + + # HONESTY CHECK: genuinely fresh before restore. + assert worker2.services.data_layer_2.execution_start_ts is None + assert worker2.services.data_layer_2.iteration_count == 0 + + restored = await _drain_until( + lambda: worker2.services.data_layer_2.execution_start_ts is not None, + timeout=10.0, + ) + assert restored, "restore_cursor did not populate execution_start_ts in time" + assert worker2.services.data_layer_2.execution_start_ts == T1 + assert worker2.services.data_layer_2.iteration_count == 2 + + post_lines = _post_restart_lines(sid) + await _append_all(reg.queue_manager, sid, post_lines) + wrote2 = await _drain_until( + lambda: ( + reg.pipeline_counters()["written_total"] >= len(pre_lines) + len(post_lines) + ), + timeout=30.0, + ) + assert wrote2, "post-reap batch did not commit within the window" + + await _cancel_and_await(worker2) + + # Same duplicate=0 / run-scoped invariant as the crash-restart scenario. + iter_rows = await _run_query( + neo4j_container, + "MATCH (i:Iteration {session_id: $sid}) " + "RETURN i.node_id AS node_id, i.iteration_number AS n, " + "i.iteration_scope AS scope ORDER BY i.iteration_number", + sid=sid, + ) + node_ids = [r["node_id"] for r in iter_rows] + assert len(node_ids) == len(set(node_ids)), ( + f"duplicate Iteration node_ids for session {sid}: {node_ids}" + ) + bare = [nid for nid in node_ids if "::orch_run::" not in nid] + assert bare == [], f"bare (pre-fix-shape) Iteration node_ids found: {bare}" + assert len(node_ids) == 3, f"expected exactly 3 Iteration nodes, got {node_ids}" + + expected_prefix = f"{orch_run_id}::iteration::" + for r in iter_rows: + assert r["node_id"].startswith(expected_prefix), ( + f"Iteration {r['node_id']} is not scoped to {orch_run_id}" + ) + assert [r["n"] for r in iter_rows] == [1, 2, 3], ( + f"iteration_number must continue 1,2,3 across the reap, got " + f"{[r['n'] for r in iter_rows]}" + ) + scopes = [r["scope"] for r in iter_rows] + assert all(s is not None for s in scopes), f"missing iteration_scope: {scopes}" + assert scopes == ["run", "run", "run"], f"expected all-'run' scopes, got {scopes}" + + # E06 for the post-reap iteration -- proves execution_start_ts (not just + # iteration_count) survived the reap. + iter3_id = f"{orch_run_id}::iteration::3" + e06_rows = await _run_query( + neo4j_container, + "MATCH (o:OrchestratorRun {node_id: $orid})-[r:HAS_PART]->(i:Iteration) " + "WHERE i.node_id = $iter3 RETURN count(r) AS c", + orid=orch_run_id, + iter3=iter3_id, + ) + assert e06_rows[0]["c"] >= 1, "E06 HAS_PART edge missing across the reap boundary" + + # E09 across the reap boundary -- pending_tool_block_ids survival proof. + e09_rows = await _run_query( + neo4j_container, + "MATCH (b:ContentBlock {session_id: $sid})-[r:CAUSED]->(t:ToolCall) " + "WHERE t.tool_call_id = $tcid RETURN count(r) AS c", + sid=sid, + tcid=TOOL_CALL_ID, + ) + assert e09_rows[0]["c"] >= 1, ( + "E09 CAUSED edge missing across the reap boundary -- " + "pending_tool_block_ids did not survive" + ) diff --git a/tests/neo4j/test_incomplete_session_heal_forward.py b/tests/neo4j/test_incomplete_session_heal_forward.py new file mode 100644 index 00000000..7f7711fb --- /dev/null +++ b/tests/neo4j/test_incomplete_session_heal_forward.py @@ -0,0 +1,192 @@ +"""Tier 3 - Neo4j integration proof for IncompleteSession heal-forward (Part 1). + +Encodes the out-of-order race that produces the ~99% false-positive +IncompleteSession population: a forked sub-session's session:end drains +(independent per-session queue) BEFORE its session:fork/session:start. +SessionLabelStateMachine.classify() must strip the stale IncompleteSession +marker the moment the real start/fork is processed (see +docs/issues/incomplete-session-mislabeling.md and +docs/plans/2026-08-12-incomplete-session-relabel-spec.md, Part 1). + +This closes a real coverage gap: no prior real-Neo4j test exercised +set_labels() with a non-empty remove_labels list. It runs the real +SessionHandler (not just the pure classify() unit) against a live Neo4j +container, flushes, and reads back with a fresh Cypher query (bypassing the +in-memory buffer) to prove the label is PHYSICALLY removed from the node. + +It also proves non-interaction with the terminal-label lattice +normalization (_LATTICE_NORMALIZATION in neo4j_store.py): removing +IncompleteSession must not disturb the RootSession/SubSession/ForkedSession +convergence guarantee, since IncompleteSession is not a member of +_TERMINAL_LABELS. + +Run: uv run pytest tests/neo4j/test_incomplete_session_heal_forward.py -v -m neo4j +""" + +from __future__ import annotations + +from typing import Any + +import pytest + +from context_intelligence_server.handlers.data_layer_2.session import SessionHandler + +pytestmark = pytest.mark.neo4j + + +async def _neo4j_labels(services: Any, node_id: str) -> list[str]: + """Read labels directly from Neo4j (bypasses the in-memory buffer).""" + rows = await services.graph.execute_query( + "MATCH (n) WHERE n.node_id = $id AND n.workspace = $workspace " + "RETURN labels(n) AS lbls", + {"id": node_id, "workspace": services.graph.workspace}, + workspace="*", + ) + return list(rows[0]["lbls"]) if rows else [] + + +@pytest.mark.neo4j +class TestIncompleteSessionHealForward: + """Out-of-order end -> fork/start must physically strip IncompleteSession.""" + + async def test_out_of_order_end_then_fork_strips_incomplete_session_in_neo4j( + self, neo4j_services: Any + ) -> None: + """end (bare, stamps IncompleteSession) -> flush -> fork -> flush: + the real Neo4j node must end with ForkedSession only, no + IncompleteSession, proving REMOVE n:IncompleteSession actually ran. + """ + handler = SessionHandler(neo4j_services) + session_id = "child-heal-fork-001" + + # session:end drains first (out-of-order race) -- stamps IncompleteSession + await handler( + "session:end", + {"session_id": session_id, "timestamp": "2026-01-01T10:00:00Z"}, + ) + await neo4j_services.graph.flush() + + mid_labels = await _neo4j_labels(neo4j_services, session_id) + assert "IncompleteSession" in mid_labels, ( + f"precondition: out-of-order end must stamp IncompleteSession in " + f"Neo4j; got {mid_labels}" + ) + + # The real session:fork arrives late, in its own flush cycle. + await handler( + "session:fork", + { + "session_id": session_id, + "parent_id": "parent-heal-fork-001", + "timestamp": "2026-01-01T09:59:59Z", + }, + ) + await neo4j_services.graph.flush() + + final_labels = await _neo4j_labels(neo4j_services, session_id) + assert "ForkedSession" in final_labels, ( + f"expected ForkedSession in {final_labels}" + ) + assert "IncompleteSession" not in final_labels, ( + f"heal-forward must physically REMOVE n:IncompleteSession at " + f"flush; still present in Neo4j: {final_labels}" + ) + + async def test_out_of_order_end_then_start_strips_incomplete_session_in_neo4j( + self, neo4j_services: Any + ) -> None: + """end (bare, stamps IncompleteSession) -> flush -> start (no parent) + -> flush: the real Neo4j node must end with RootSession only. + """ + handler = SessionHandler(neo4j_services) + session_id = "root-heal-start-001" + + await handler( + "session:end", + {"session_id": session_id, "timestamp": "2026-01-01T10:00:00Z"}, + ) + await neo4j_services.graph.flush() + + mid_labels = await _neo4j_labels(neo4j_services, session_id) + assert "IncompleteSession" in mid_labels, ( + f"precondition: out-of-order end must stamp IncompleteSession in " + f"Neo4j; got {mid_labels}" + ) + + await handler( + "session:start", + {"session_id": session_id, "timestamp": "2026-01-01T09:59:59Z"}, + ) + await neo4j_services.graph.flush() + + final_labels = await _neo4j_labels(neo4j_services, session_id) + assert "RootSession" in final_labels, f"expected RootSession in {final_labels}" + assert "IncompleteSession" not in final_labels, ( + f"heal-forward must physically REMOVE n:IncompleteSession at " + f"flush; still present in Neo4j: {final_labels}" + ) + + async def test_heal_forward_does_not_disturb_terminal_lattice_normalization( + self, neo4j_services: Any + ) -> None: + """Healing IncompleteSession must not interact with, or break, the + RootSession/SubSession/ForkedSession lattice-normalization guarantee. + + Drives: end (bare, stamps IncompleteSession) -> flush -> start WITH a + parent (assigns SubSession) -> flush -> fork (reclassifies to + ForkedSession, the lattice's specificity ordering) -> flush. The node + must converge to exactly ONE terminal label (ForkedSession) with + IncompleteSession and the stale SubSession both absent -- proving the + IncompleteSession REMOVE and the terminal-lattice REMOVE/SET both ran + correctly and did not clobber each other. + """ + handler = SessionHandler(neo4j_services) + session_id = "lattice-heal-001" + parent_id = "lattice-heal-parent-001" + + await handler( + "session:end", + {"session_id": session_id, "timestamp": "2026-01-01T10:00:00Z"}, + ) + await neo4j_services.graph.flush() + + await handler( + "session:start", + { + "session_id": session_id, + "parent_id": parent_id, + "timestamp": "2026-01-01T09:59:58Z", + }, + ) + await neo4j_services.graph.flush() + + mid_labels = await _neo4j_labels(neo4j_services, session_id) + assert "SubSession" in mid_labels, f"expected SubSession in {mid_labels}" + assert "IncompleteSession" not in mid_labels, ( + f"IncompleteSession must already be healed after start: {mid_labels}" + ) + + await handler( + "session:fork", + { + "session_id": session_id, + "parent_id": parent_id, + "timestamp": "2026-01-01T09:59:59Z", + }, + ) + await neo4j_services.graph.flush() + + final_labels = await _neo4j_labels(neo4j_services, session_id) + terminals = [ + lbl + for lbl in final_labels + if lbl in ("RootSession", "SubSession", "ForkedSession") + ] + assert terminals == ["ForkedSession"], ( + f"lattice must converge to exactly one terminal (ForkedSession); " + f"got {terminals} in {final_labels}" + ) + assert "IncompleteSession" not in final_labels, ( + f"IncompleteSession must remain healed through the lattice " + f"reclassification: {final_labels}" + ) diff --git a/tests/neo4j/test_node_identity_migration.py b/tests/neo4j/test_node_identity_migration.py index 2358d0d2..ae5920d8 100644 --- a/tests/neo4j/test_node_identity_migration.py +++ b/tests/neo4j/test_node_identity_migration.py @@ -16,25 +16,34 @@ dead-weight O(graph-size) cost on an already-migrated graph). ``ensure_neo4j_schema`` now only creates cheap, idempotent indexes/constraints; if the graph still has untagged/duplicate legacy data, constraint creation (Step 6) either raises -(``fail_on_data_conflict=True``) or logs a WARNING and returns ``False`` (the -default). - -Cold start vs. mid-flight flush have OPPOSITE requirements for this function -(design decision, reversing the lifespan half of commit f4d8bab): - -- **Cold start** (``main.py``'s ``lifespan()``): FAILS LOUD. Nothing has been - written yet, so refusing to boot on an un-migrated graph (duplicate legacy - :Node-labeled nodes, caught by ``fail_on_data_conflict=True`` here; OR nodes - lacking the ``:Node`` label altogether, which the constraint can't see and - is instead caught by the separate O(1) ``count_untagged_nodes`` guard) loses - no data and surfaces the impossible state immediately. -- **Mid-flight flush** (``Neo4jGraphStore._ensure_schema``): must NEVER raise - -- a ``RuntimeError`` escaping there dead-letters real in-flight activity - records (reviewer Salil's PR #67 blocker, commit 14a6d30). Leaves the - default ``fail_on_data_conflict=False``: a data conflict is logged as a - WARNING and self-heals on the next flush once the graph is repaired. -- ``run_repair`` / ``doctor --fix`` also opts into ``fail_on_data_conflict=True`` - (a lingering conflict AFTER dedup+backfill is a genuine repair failure). +(``fail_on_data_conflict=True``) or logs a WARNING/ERROR and returns ``False`` +(the default) -- and, in the failure case, establishes a fallback +``idx_node_universal`` index so the write path keeps a ``NodeIndexSeek`` +(see the council amendment B2 note below and +``tests/neo4j/test_node_index_seek.py``). + +UPDATE (deploy-safe boot, council amendment 2026-08-12 -- +docs/plans/2026-08-12-deploy-safe-boot-spec.md, workspace root): cold start +(``main.py``'s ``lifespan()``) no longer fails closed on graph *data* state. +A deploy must never crash-loop against an un-migrated, unreachable, or +degraded graph, so lifespan now calls this function with the SAME +``fail_on_data_conflict=False`` default the mid-flight flush path always +used, and surfaces the result as a tri-state ``schema_health`` signal on +``GET /status`` instead of raising (see ``tests/test_main.py``'s +deploy-safe-boot test block). Only ``run_repair``/``doctor --fix`` still +opts into ``fail_on_data_conflict=True``: + +- **Cold start / mid-flight flush** (``main.py``'s ``lifespan()``; + ``Neo4jGraphStore._ensure_schema``): must NEVER raise due to graph *data* + state. A ``RuntimeError`` escaping the flush path dead-letters real + in-flight activity records (the PR #67 blocker, commit + 14a6d30); escaping the lifespan crash-loops the deploy (the 2026-08-12 + incident this amendment fixes). Both leave the default + ``fail_on_data_conflict=False``: a data conflict is logged and reported + via the return value, self-healing once the graph is repaired. +- ``run_repair`` / ``doctor --fix`` still opts into ``fail_on_data_conflict=True`` + (a lingering conflict AFTER dedup+backfill is a genuine repair failure -- + nothing is at risk of being lost or crash-looped at that point). See docs/node-identity-migration.md. @@ -50,6 +59,7 @@ import pytest from context_intelligence_server.neo4j_store import ( + _NODE_MERGE_CYPHER, Neo4jGraphStore, count_untagged_nodes, ensure_neo4j_schema, @@ -129,7 +139,7 @@ def _seed_node_constraint_conflict(container: dict[str, Any]) -> None: node_node_id_workspace_unique`` against this seed raises a genuine ``Neo.ClientError.Schema.ConstraintValidationFailed`` from a REAL Neo4j server (not a synthesized error) -- exactly the conflict the PR #67 - blocker fix (reviewer Salil) must survive without dead-lettering. + blocker fix must survive without dead-lettering. """ driver = GraphDatabase.driver( container["bolt_url"], @@ -289,8 +299,9 @@ async def _run_migration_assertions(neo4j_container: dict[str, Any]) -> None: async def test_cold_start_guard_detects_untagged_only_graph( neo4j_container: dict[str, Any], ) -> None: - """Reproduces main.py's lifespan cold-start guard, against a REAL Neo4j, - for the untagged-only shape the :Node constraint alone CANNOT see. + """Reproduces the primitives behind main.py's lifespan schema-health + computation, against a REAL Neo4j, for the untagged-only shape the + :Node constraint alone CANNOT see. Uses ``_seed_untagged_only_graph`` (NOT ``_seed_dirty_graph``, whose duplicate ``dup-1`` :Event nodes trip the separate, always fail-open @@ -298,22 +309,29 @@ async def test_cold_start_guard_detects_untagged_only_graph( ``test_run_repair_dedups_backfills_and_constrains``). Every node seeded here has a unique (node_id, workspace) and no label collision, so NO uniqueness constraint (Session/Event/Node) sees a conflict -- the ONLY - defect is the missing ``:Node`` label, which is exactly why the lifespan - guard needs its second, independent check (``count_untagged_nodes``): - the constraint step provides no signal for this case on its own. - - Reproduces the lifespan's two ordered steps directly against the live - container (the guard logic is inline in ``main.py``'s ``lifespan()``, - not its own importable function): - - 1. ``ensure_neo4j_schema(driver, fail_on_data_conflict=True)`` -- - succeeds (``True``), no constraint conflict to raise on. + defect is the missing ``:Node`` label, which is exactly why lifespan's + schema-health computation needs its second, independent check + (``count_untagged_nodes``): the constraint step provides no signal for + this case on its own. + + UPDATE (deploy-safe boot, council amendment 2026-08-12): lifespan no + longer raises on this condition -- it now calls this function with + ``fail_on_data_conflict=False`` (the same default used here for + parity/documentation purposes; the constraint would succeed either way, + since there is no conflict for it to see) and feeds the two results + below into a tri-state ``schema_health`` signal on ``GET /status`` + (see ``tests/test_main.py::test_lifespan_does_not_raise_on_untagged_nodes`` + for the unit-level contract). This test still exercises the exact two + primitives lifespan calls, directly against a live Neo4j: + + 1. ``ensure_neo4j_schema(driver, fail_on_data_conflict=False)`` -- + succeeds (``True``), no constraint conflict for it to see. 2. ``count_untagged_nodes(driver)`` -- reports > 0. Together, (1) succeeding and (2) being > 0 is precisely the condition - under which ``lifespan()`` raises ``RuntimeError`` naming - ``doctor --fix`` -- i.e. the un-migrated (untagged-only) graph IS - detected and cold start WOULD refuse to boot. + under which lifespan now reports ``schema_health="degraded"`` (never a + boot refusal) -- i.e. the un-migrated (untagged-only) graph IS detected, + surfaced on ``/status``, and the server boots and serves regardless. """ _wipe(neo4j_container) try: @@ -324,15 +342,17 @@ async def test_cold_start_guard_detects_untagged_only_graph( auth=(neo4j_container["user"], neo4j_container["password"]), ) try: - # Step 1 (lifespan): fail-loud schema init does NOT raise here -- - # none of the seeded nodes carry :Node (or collide under any - # OTHER constraint), so no constraint sees a conflict. This is - # the case the constraint check alone misses. - established = await ensure_neo4j_schema(driver, fail_on_data_conflict=True) + # Step 1 (lifespan): schema init does NOT fail here -- none of + # the seeded nodes carry :Node (or collide under any OTHER + # constraint), so no constraint sees a conflict. This is the + # case the constraint check alone misses. + established = await ensure_neo4j_schema( + driver, fail_on_data_conflict=False + ) assert established is True, ( - "ensure_neo4j_schema(fail_on_data_conflict=True) must succeed " - "on an untagged-only (no :Node label anywhere, no duplicates) " - "dirty graph -- there is no constraint conflict to raise on." + "ensure_neo4j_schema must succeed on an untagged-only (no " + ":Node label anywhere, no duplicates) dirty graph -- there " + "is no constraint conflict for it to see." ) # Step 2 (lifespan): the O(1) untagged guard DOES catch it. @@ -340,8 +360,8 @@ async def test_cold_start_guard_detects_untagged_only_graph( assert untagged > 0, ( "count_untagged_nodes must report the seeded untagged legacy " "nodes (legacy-sess-only + legacy-bare) -- this is the exact " - "signal main.py's lifespan() uses to raise RuntimeError and " - "refuse to boot on an un-migrated graph." + "signal main.py's lifespan() feeds into schema_health=" + "\"degraded\" on GET /status for an un-migrated graph." ) finally: await driver.close() @@ -353,7 +373,7 @@ async def test_flush_path_self_heals_and_repair_converges_e2e( neo4j_container: dict[str, Any], caplog: pytest.LogCaptureFixture, ) -> None: - """Live E2E regression for the PR #67 merge-blocker (reviewer Salil, commit + """Live E2E regression for the PR #67 merge-blocker (commit 14a6d30): a genuine :Node constraint data conflict against a REAL Neo4j must NOT dead-letter in-flight activity records on the flush path, must self-heal once ``doctor --fix`` (``run_repair``) repairs the graph, and the @@ -511,3 +531,253 @@ async def _phase_e_doctor_contract_fails_closed( await ensure_neo4j_schema(driver, fail_on_data_conflict=True) finally: await driver.close() + + +# --------------------------------------------------------------------------- +# Council amendment B2 (deploy-safe boot, 2026-08-12): degraded mode must +# never lose the write-path index seek, only atomicity. These tests EXPLAIN +# the real production node-MERGE query against a live Neo4j to prove the +# fallback idx_node_universal index keeps a NodeIndexSeek even when the +# :Node uniqueness constraint cannot be established. +# --------------------------------------------------------------------------- + +def _collect_plan_operators(plan: dict[str, Any]) -> list[str]: + """Recursively collect every operatorType in a Neo4j EXPLAIN/PROFILE plan.""" + ops: list[str] = [] + if not plan: + return ops + op = plan.get("operatorType") or plan.get("operator_type") + if op: + ops.append(op) + for child in plan.get("children", []) or []: + ops.extend(_collect_plan_operators(child)) + return ops + + +def _explain_node_merge(container: dict[str, Any]) -> list[str]: + """EXPLAIN the production node MERGE query and return its plan operators.""" + driver = GraphDatabase.driver( + container["bolt_url"], + auth=(container["user"], container["password"]), + ) + try: + with driver.session() as s: + result = s.run( + "EXPLAIN " + _NODE_MERGE_CYPHER, + rows=[{"node_id": "b2-plan-node", "props": {"workspace": _WS}}], + ) + summary = result.consume() + plan = summary.plan or {} + return _collect_plan_operators(plan) + finally: + driver.close() + + +async def test_degraded_graph_fallback_index_keeps_node_index_seek( + neo4j_container: dict[str, Any], +) -> None: + """On a duplicate-blocked-constraint graph (the :Node constraint CANNOT + be established), ensure_neo4j_schema must create the fallback + idx_node_universal index -- so the production node-MERGE query plans as + a NodeIndexSeek, NEVER a NodeByLabelScan/AllNodesScan (the 25-30s stall + PR #67 removed from the write path). Degraded mode costs atomicity + only, never the seek. + """ + _wipe(neo4j_container) + try: + _seed_node_constraint_conflict(neo4j_container) + + driver = AsyncGraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + try: + established = await ensure_neo4j_schema( + driver, fail_on_data_conflict=False + ) + assert established is False, ( + "the :Node constraint must NOT be established against a " + "genuine duplicate-node conflict" + ) + finally: + await driver.close() + + # The fallback idx_node_universal must exist despite the constraint + # failure (B2: degraded mode never loses the index). + sync_driver = GraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + try: + with sync_driver.session() as s: + idx_count = s.run( + "SHOW INDEXES YIELD name WHERE name = 'idx_node_universal' " + "RETURN count(*) AS c" + ).single()["c"] + finally: + sync_driver.close() + assert idx_count == 1, ( + "Fallback idx_node_universal must be created when the :Node " + "constraint cannot be established (degraded mode, B2)." + ) + + ops = _explain_node_merge(neo4j_container) + assert ops, f"EXPLAIN returned no plan operators (ops={ops!r})" + assert not any("AllNodesScan" in op for op in ops), ( + "Degraded-mode node MERGE regressed to a full-graph AllNodesScan " + f"-- the fallback index did not back the seek. Plan operators: {ops}" + ) + assert not any("NodeByLabelScan" in op for op in ops), ( + f"Degraded-mode node MERGE did a NodeByLabelScan instead of an " + f"index seek. Plan operators: {ops}" + ) + assert any("IndexSeek" in op for op in ops), ( + "Degraded-mode node MERGE is not index-backed -- expected a " + f"NodeIndexSeek from the fallback idx_node_universal. Plan " + f"operators: {ops}" + ) + finally: + _wipe(neo4j_container) + + +async def test_healthy_graph_keeps_constraint_backed_index_seek( + neo4j_container: dict[str, Any], +) -> None: + """On a healthy graph (no data conflict), the :Node uniqueness + constraint is established and the node-MERGE query plans as a + NodeIndexSeek backed by the constraint's own index -- zero behavior + change from before the B2 reorder. + """ + _wipe(neo4j_container) + try: + driver = AsyncGraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + try: + established = await ensure_neo4j_schema( + driver, fail_on_data_conflict=False + ) + assert established is True, ( + "ensure_neo4j_schema must fully establish the schema on a " + "healthy graph with no data conflict." + ) + finally: + await driver.close() + + ops = _explain_node_merge(neo4j_container) + assert ops, f"EXPLAIN returned no plan operators (ops={ops!r})" + assert not any("AllNodesScan" in op for op in ops), ( + f"Healthy-graph node MERGE regressed to an AllNodesScan. Plan " + f"operators: {ops}" + ) + assert any("IndexSeek" in op for op in ops), ( + "Healthy-graph node MERGE is not index-backed -- expected a " + f"NodeIndexSeek from the :Node uniqueness constraint. Plan " + f"operators: {ops}" + ) + finally: + _wipe(neo4j_container) + + +# --------------------------------------------------------------------------- +# Review-remediation gap: idx_node_universal must be DROPPED once a +# successful run_repair establishes the :Node constraint (Step 6 comment +# block above: "Constraint carries its own backing index; a standalone +# idx_node_universal ... is now redundant. Drop it ONLY after the +# constraint is confirmed established"). No prior test asserted the DROP +# side of that contract -- test_degraded_graph_fallback_index_keeps_node_ +# index_seek above only proves the index is CREATED/kept while degraded. +# --------------------------------------------------------------------------- + + +async def test_run_repair_drops_redundant_idx_node_universal_after_success( + neo4j_container: dict[str, Any], +) -> None: + """A standalone ``idx_node_universal`` (e.g. left behind by a PRIOR + degraded boot, or the pre-amendment code) becomes redundant dead weight + the moment the ``:Node`` uniqueness constraint exists -- the constraint + carries its own backing range index. This test seeds that exact + leftover-index shape on a graph with NO ``:Node`` data conflicts (a + single, uniquely-keyed node), so ``run_repair``'s constraint-creation + step succeeds cleanly, and proves: + + 1. Non-vacuity -- ``idx_node_universal`` genuinely exists BEFORE + ``run_repair`` is called (proves the test seed is meaningful, not a + vacuous pass against an index that was never there). + 2. After ``run_repair``: the ``:Node`` uniqueness constraint IS present. + 3. After ``run_repair``: the now-redundant standalone + ``idx_node_universal`` is ABSENT (dropped). + """ + _wipe(neo4j_container) + try: + driver = GraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + try: + with driver.session() as s: + # A single, uniquely-keyed :Node -- no duplicate (node_id, + # workspace) anywhere, so the :Node constraint creation + # (Step 6) succeeds cleanly (no data conflict for it to see). + s.run( + "CREATE (:Node:Event {node_id: 'evt-clean', workspace: $ws})", + ws=_WS, + ) + # Seed the standalone fallback index directly -- the exact + # leftover shape a PRIOR degraded boot (or the + # pre-amendment code) leaves behind, which must be cleaned + # up once the constraint can finally be established. + s.run( + "CREATE INDEX idx_node_universal IF NOT EXISTS " + "FOR (n:Node) ON (n.node_id, n.workspace)" + ) + + # Non-vacuity: prove the index is genuinely present BEFORE repair. + with driver.session() as s: + pre_count = s.run( + "SHOW INDEXES YIELD name " + "WHERE name = 'idx_node_universal' " + "RETURN count(*) AS c" + ).single()["c"] + assert pre_count == 1, ( + "test setup failed: idx_node_universal must exist BEFORE " + f"run_repair for this test to be meaningful, found {pre_count}" + ) + finally: + driver.close() + + store = Neo4jGraphStore( + uri=neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + workspace=_WS, + ) + try: + await run_repair(store._driver, store._database) + + constraint_rows = await store.execute_query( + "SHOW CONSTRAINTS YIELD name " + "WHERE name = 'node_node_id_workspace_unique' " + "RETURN count(*) AS c", + workspace="*", + ) + idx_rows_after = await store.execute_query( + "SHOW INDEXES YIELD name " + "WHERE name = 'idx_node_universal' " + "RETURN count(*) AS c", + workspace="*", + ) + finally: + await store.close() + + assert constraint_rows[0]["c"] == 1, ( + "the :Node uniqueness constraint was not established by " + "run_repair on a clean, conflict-free graph" + ) + assert idx_rows_after[0]["c"] == 0, ( + "the redundant standalone idx_node_universal must be DROPPED " + f"once the :Node constraint is established, found " + f"{idx_rows_after[0]['c']} still present" + ) + finally: + _wipe(neo4j_container) diff --git a/tests/neo4j/test_relabel_incomplete_sessions.py b/tests/neo4j/test_relabel_incomplete_sessions.py new file mode 100644 index 00000000..f232f27c --- /dev/null +++ b/tests/neo4j/test_relabel_incomplete_sessions.py @@ -0,0 +1,487 @@ +"""Tier 3 Neo4j integration test module -- IncompleteSession backfill (Part 2). + +Seeds real Session/SessionStartEvent/SessionForkEvent nodes and SOURCED_FROM +edges to exercise scripts/relabel_incomplete_sessions.py against a live +Neo4j container. Covers the six scenarios from +docs/plans/2026-08-12-incomplete-session-relabel-spec.md, Part 2 "Tests": + +(a) forked node with a real terminal label + stale marker -> cleared by the + gated --apply path (run_apply). +(b) node with a linked SessionStartEvent (no terminal label yet) + stale + marker -> cleared by the raw selector/write function (apply_relabel), + proving B1's selector mechanics work on this node shape in isolation. +(c) genuine no-start/no-fork node -> RETAINED. +(d) idempotence: a second apply_relabel() call matches zero rows. +(e) the B2 reconciliation gate: seed a linked_but_untyped node and assert + the gated --apply path (run_apply) REFUSES and makes NO write. +(f) --restore re-adds :IncompleteSession to exactly the touched ids. + +Note: (b) and (e) seed the SAME node shape (IncompleteSession + a linked +start/fork event + no terminal label -- exactly "linked_but_untyped"). They +are deliberately not a contradiction: (b) proves the underlying B1 selector +mechanically heals that shape when invoked directly; (e) proves the B2 +safety gate refuses the FULL --apply workflow when such a node exists +anywhere in the graph (the gate exists precisely because the assumption is +unverified on historical data, independent of whether the mechanics work). + +Run: uv run pytest tests/neo4j/test_relabel_incomplete_sessions.py -v -m neo4j +""" + +from __future__ import annotations + +from typing import Any + +import pytest +from neo4j import GraphDatabase +from scripts import relabel_incomplete_sessions as relabel + +WORKSPACE = "test" + +pytestmark = pytest.mark.neo4j + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _driver(neo4j_container: dict) -> GraphDatabase: + """Return a synchronous Neo4j driver for the test container.""" + return GraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + + +def _seed_session(session, node_id: str, extra_labels: list[str] | None = None) -> None: + """Seed a bare :Session:IncompleteSession node, optionally with extra labels.""" + labels = "".join(f":{lbl}" for lbl in (extra_labels or [])) + session.run( + f"MERGE (s:Session:IncompleteSession{labels} " + "{node_id: $node_id, workspace: $workspace})", + node_id=node_id, + workspace=WORKSPACE, + ) + + +def _seed_linked_event( + session, session_node_id: str, event_label: str, event_node_id: str +) -> None: + """Seed a SessionStartEvent/SessionForkEvent node and the real + (Session)-[:SOURCED_FROM]->(Event) edge direction (see the module + docstring's "Selector direction correction" for why this direction, not + the reverse, is what the shipping write path actually produces). + """ + session.run( + f"MATCH (s:Session {{node_id: $session_node_id, workspace: $workspace}}) " + f"MERGE (e:{event_label} {{node_id: $event_node_id, workspace: $workspace}}) " + "MERGE (s)-[:SOURCED_FROM]->(e)", + session_node_id=session_node_id, + event_node_id=event_node_id, + workspace=WORKSPACE, + ) + + +def _labels(session, node_id: str) -> list[str]: + """Return the labels of a node matched by node_id + workspace, or [].""" + result = session.run( + "MATCH (n {node_id: $node_id, workspace: $workspace}) RETURN labels(n) AS lbls", + node_id=node_id, + workspace=WORKSPACE, + ) + record = result.single() + if record is None: + return [] + return list(record["lbls"]) + + +# --------------------------------------------------------------------------- +# Per-test isolation +# --------------------------------------------------------------------------- + + +@pytest.fixture(autouse=True) +def _clear_neo4j(neo4j_container: dict) -> None: # type: ignore[return] + """Wipe the container clean before each test for complete isolation.""" + driver = GraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + try: + with driver.session() as s: + s.run("MATCH (n) DETACH DELETE n") + finally: + driver.close() + + +# --------------------------------------------------------------------------- +# (a) forked node with real terminal + stale marker -> cleared by --apply +# --------------------------------------------------------------------------- + + +@pytest.mark.neo4j +class TestApplyClearsTerminalLabeledNode: + def test_forked_node_with_terminal_label_is_healed_by_gated_apply( + self, neo4j_container: dict[str, Any], tmp_path: Any + ) -> None: + driver = _driver(neo4j_container) + try: + with driver.session() as s: + _seed_session(s, "sess-a", extra_labels=["ForkedSession"]) + + exit_code = relabel.run_apply( + s, + batch_size=relabel.DEFAULT_BATCH_SIZE, + undo_log_path=str(tmp_path / "undo-a.json"), + neo4j_url=neo4j_container["bolt_url"], + ) + + assert exit_code == 0 + labels = _labels(s, "sess-a") + assert "ForkedSession" in labels + assert "IncompleteSession" not in labels + finally: + driver.close() + + +# --------------------------------------------------------------------------- +# (b) linked SessionStartEvent, no terminal label -> cleared by the raw +# selector/write function (apply_relabel), proving B1 mechanics. +# --------------------------------------------------------------------------- + + +@pytest.mark.neo4j +class TestApplyRelabelClearsLinkedUntypedNode: + def test_linked_start_event_no_terminal_label_healed_by_raw_selector( + self, neo4j_container: dict[str, Any] + ) -> None: + driver = _driver(neo4j_container) + try: + with driver.session() as s: + _seed_session(s, "sess-b") + _seed_linked_event(s, "sess-b", "SessionStartEvent", "sess-b::start") + + touched = relabel.apply_relabel( + s, batch_size=relabel.DEFAULT_BATCH_SIZE + ) + + assert touched == ["sess-b"] + labels = _labels(s, "sess-b") + assert "IncompleteSession" not in labels + assert "Session" in labels + finally: + driver.close() + + +# --------------------------------------------------------------------------- +# (c) genuine no-start/no-fork node -> RETAINED +# --------------------------------------------------------------------------- + + +@pytest.mark.neo4j +class TestGenuineIncompleteRetained: + def test_genuine_node_with_no_linked_event_and_no_terminal_is_retained( + self, neo4j_container: dict[str, Any], tmp_path: Any + ) -> None: + driver = _driver(neo4j_container) + try: + with driver.session() as s: + _seed_session(s, "sess-c") + + exit_code = relabel.run_apply( + s, + batch_size=relabel.DEFAULT_BATCH_SIZE, + undo_log_path=str(tmp_path / "undo-c.json"), + neo4j_url=neo4j_container["bolt_url"], + ) + + assert exit_code == 0 + labels = _labels(s, "sess-c") + assert "IncompleteSession" in labels, ( + "genuine incomplete session (no linked start/fork event) " + f"must be retained; got {labels}" + ) + finally: + driver.close() + + +# --------------------------------------------------------------------------- +# (d) idempotence: second apply_relabel() matches zero rows +# --------------------------------------------------------------------------- + + +@pytest.mark.neo4j +class TestIdempotence: + def test_second_apply_relabel_touches_zero_rows( + self, neo4j_container: dict[str, Any] + ) -> None: + driver = _driver(neo4j_container) + try: + with driver.session() as s: + _seed_session(s, "sess-d", extra_labels=["ForkedSession"]) + + first = relabel.apply_relabel(s, batch_size=relabel.DEFAULT_BATCH_SIZE) + second = relabel.apply_relabel(s, batch_size=relabel.DEFAULT_BATCH_SIZE) + + assert first == ["sess-d"] + assert second == [] + labels = _labels(s, "sess-d") + assert "IncompleteSession" not in labels + assert "ForkedSession" in labels + finally: + driver.close() + + +# --------------------------------------------------------------------------- +# (e) B2 reconciliation gate: linked_but_untyped node -> --apply REFUSES +# --------------------------------------------------------------------------- + + +@pytest.mark.neo4j +class TestReconciliationGateRefusesApply: + def test_linked_but_untyped_node_causes_gated_apply_to_refuse( + self, neo4j_container: dict[str, Any], tmp_path: Any + ) -> None: + driver = _driver(neo4j_container) + try: + with driver.session() as s: + # Same shape as (b) -- IncompleteSession + linked start event, + # no terminal label -- but exercised through the GATED + # run_apply() path instead of the raw apply_relabel(). + _seed_session(s, "sess-e") + _seed_linked_event(s, "sess-e", "SessionStartEvent", "sess-e::start") + + diag = relabel.diagnostic_report(s) + assert diag["linked_but_untyped"] == 1 + assert diag["linked_but_untyped_samples"] == ["sess-e"] + + exit_code = relabel.run_apply( + s, + batch_size=relabel.DEFAULT_BATCH_SIZE, + undo_log_path=str(tmp_path / "undo-e.json"), + neo4j_url=neo4j_container["bolt_url"], + ) + + assert exit_code == 1, ( + "run_apply must REFUSE (non-zero exit) when the B2 " + "diagnostic finds a linked_but_untyped node" + ) + # No write must have occurred: the marker is still present. + labels = _labels(s, "sess-e") + assert "IncompleteSession" in labels, ( + "gate refusal must not mutate the graph; " + f"IncompleteSession missing from {labels}" + ) + # The undo-log must not have been written either. + assert not (tmp_path / "undo-e.json").exists() + finally: + driver.close() + + +# --------------------------------------------------------------------------- +# (f) --restore re-adds :IncompleteSession to exactly the touched ids +# --------------------------------------------------------------------------- + + +@pytest.mark.neo4j +class TestRestore: + def test_restore_re_adds_marker_to_exactly_the_touched_ids( + self, neo4j_container: dict[str, Any], tmp_path: Any + ) -> None: + driver = _driver(neo4j_container) + try: + with driver.session() as s: + _seed_session(s, "sess-f", extra_labels=["ForkedSession"]) + undo_log_path = tmp_path / "undo-f.json" + + exit_code = relabel.run_apply( + s, + batch_size=relabel.DEFAULT_BATCH_SIZE, + undo_log_path=str(undo_log_path), + neo4j_url=neo4j_container["bolt_url"], + ) + assert exit_code == 0 + assert undo_log_path.exists() + + healed_labels = _labels(s, "sess-f") + assert "IncompleteSession" not in healed_labels + assert "ForkedSession" in healed_labels + + snap = relabel.read_undo_log(str(undo_log_path)) + assert snap["node_ids"] == ["sess-f"] + assert "generated_at" in snap + assert "neo4j_host" in snap + + restore_exit_code = relabel.run_restore( + s, str(undo_log_path), batch_size=relabel.DEFAULT_BATCH_SIZE + ) + assert restore_exit_code == 0 + + restored_labels = _labels(s, "sess-f") + assert "IncompleteSession" in restored_labels + # The terminal label must never have been touched. + assert "ForkedSession" in restored_labels + finally: + driver.close() + + +# --------------------------------------------------------------------------- +# (g) W-1: undo-log is written from a read-only PRE-MUTATION candidate +# collection, EXACTLY matching the full false-positive set, and restore +# from it fully re-adds the label to every candidate. +# --------------------------------------------------------------------------- + + +@pytest.mark.neo4j +class TestUndoLogCapturesFullPreMutationCandidateSet: + """W-1: apply_relabel()'s batched CALL {...} IN TRANSACTIONS OF N ROWS + commits per batch, so the undo-log must exist -- complete -- the moment + any batch has committed, not only after apply_relabel() returns. This + test seeds a NON-VACUOUS candidate set (two distinct false-positive + shapes that both clear the B2 gate: a terminal-labeled node, and a + terminal-labeled node that ALSO has a linked start event) and asserts + the undo-log written by the gated --apply path (run_apply) contains + EXACTLY that candidate set, and that restoring from it fully re-adds + :IncompleteSession to both. + """ + + def test_undo_log_exactly_matches_candidates_and_restore_heals_all( + self, neo4j_container: dict[str, Any], tmp_path: Any + ) -> None: + driver = _driver(neo4j_container) + try: + with driver.session() as s: + # Non-vacuity: two distinct false-positive shapes, so the + # candidate set has more than one member and is not a + # degenerate single-node case. Both carry a terminal label + # (so NEITHER trips the B2 linked_but_untyped gate); the + # second additionally has a linked start event, to prove the + # selector's OR-shape doesn't affect completeness. + _seed_session(s, "sess-g1", extra_labels=["ForkedSession"]) + _seed_session(s, "sess-g2", extra_labels=["SubSession"]) + _seed_linked_event(s, "sess-g2", "SessionStartEvent", "sess-g2::start") + + # Ground truth: the pre-mutation candidate set, read + # independently via the same read-only helper run_apply + # uses internally, BEFORE run_apply does anything. + expected_candidates = sorted(relabel.collect_false_positive_ids(s)) + assert expected_candidates == ["sess-g1", "sess-g2"], ( + "test setup must be non-vacuous: exactly two " + f"false-positive candidates expected, got {expected_candidates}" + ) + + undo_log_path = tmp_path / "undo-g.json" + exit_code = relabel.run_apply( + s, + batch_size=relabel.DEFAULT_BATCH_SIZE, + undo_log_path=str(undo_log_path), + neo4j_url=neo4j_container["bolt_url"], + ) + assert exit_code == 0 + assert undo_log_path.exists(), "undo-log file must exist after --apply" + + snap = relabel.read_undo_log(str(undo_log_path)) + logged_ids = sorted(snap["node_ids"]) + # (a) The undo log holds EXACTLY the candidate set -- no + # more, no less. + assert logged_ids == expected_candidates == ["sess-g1", "sess-g2"] + + # Both were actually mutated. + assert "IncompleteSession" not in _labels(s, "sess-g1") + assert "IncompleteSession" not in _labels(s, "sess-g2") + + # (b) Restoring from the undo log fully re-adds the marker + # to every logged node_id. + restore_exit_code = relabel.run_restore( + s, str(undo_log_path), batch_size=relabel.DEFAULT_BATCH_SIZE + ) + assert restore_exit_code == 0 + assert "IncompleteSession" in _labels(s, "sess-g1") + assert "IncompleteSession" in _labels(s, "sess-g2") + finally: + driver.close() + + +@pytest.mark.neo4j +class TestUndoLogSourcedFromPreMutationReadNotApplyReturnValue: + """W-1: the undo-log must be sourced from a read-only, pre-mutation + collection of the candidate set (``collect_false_positive_ids``) -- NOT + derived from whatever ``apply_relabel()`` happens to report as touched. + + Proven by monkeypatching ``apply_relabel`` to still perform the REAL + mutation (so the graph state and restore path stay meaningful) but + report a deliberately WRONG, truncated touched set. Under the pre-fix + ordering (log written from apply_relabel's return value, AFTER the + mutation), the undo-log would contain only the truncated set. Under the + fix, run_apply captures the full candidate set via a separate read + BEFORE calling apply_relabel at all, so the log is unaffected by + whatever apply_relabel reports. + """ + + def test_undo_log_unaffected_by_apply_relabels_return_value( + self, + neo4j_container: dict[str, Any], + tmp_path: Any, + monkeypatch: pytest.MonkeyPatch, + ) -> None: + driver = _driver(neo4j_container) + try: + with driver.session() as s: + _seed_session(s, "sess-h1", extra_labels=["ForkedSession"]) + _seed_session(s, "sess-h2", extra_labels=["RootSession"]) + + expected_candidates = sorted(relabel.collect_false_positive_ids(s)) + assert expected_candidates == ["sess-h1", "sess-h2"], ( + "test setup must be non-vacuous: exactly two " + f"false-positive candidates expected, got {expected_candidates}" + ) + + real_apply_relabel = relabel.apply_relabel + + def _fake_apply_relabel(session, batch_size): + # Perform the REAL mutation (so the restore-path + # assertions below stay meaningful) but report a + # deliberately WRONG, truncated touched set -- the + # shape of divergence a mid-run crash would produce if + # the undo-log were sourced from this return value. + real_apply_relabel(session, batch_size) + return ["sess-h1"] # deliberately omits sess-h2 + + monkeypatch.setattr(relabel, "apply_relabel", _fake_apply_relabel) + + undo_log_path = tmp_path / "undo-h.json" + exit_code = relabel.run_apply( + s, + batch_size=relabel.DEFAULT_BATCH_SIZE, + undo_log_path=str(undo_log_path), + neo4j_url=neo4j_container["bolt_url"], + ) + assert exit_code == 0 + assert undo_log_path.exists() + + snap = relabel.read_undo_log(str(undo_log_path)) + logged_ids = sorted(snap["node_ids"]) + + # THE core assertion: the log holds the FULL pre-mutation + # candidate set, not apply_relabel's (faked, truncated) + # return value. If run_apply sourced the log from + # touched_ids (the pre-fix ordering), logged_ids would + # equal ["sess-h1"] here. + assert logged_ids == expected_candidates == ["sess-h1", "sess-h2"] + assert logged_ids != ["sess-h1"], ( + "undo log must not be derived from apply_relabel's " + "reported touched set" + ) + + # Restoring from the (correctly complete) log heals BOTH + # nodes, including "sess-h2" which the faked touched-set + # omitted -- proving the log's completeness has real + # recovery value, not just structural equality. + restore_exit_code = relabel.run_restore( + s, str(undo_log_path), batch_size=relabel.DEFAULT_BATCH_SIZE + ) + assert restore_exit_code == 0 + assert "IncompleteSession" in _labels(s, "sess-h1") + assert "IncompleteSession" in _labels(s, "sess-h2") + finally: + driver.close() diff --git a/tests/neo4j/test_schema_version_baseline.py b/tests/neo4j/test_schema_version_baseline.py new file mode 100644 index 00000000..3ebd2a4c --- /dev/null +++ b/tests/neo4j/test_schema_version_baseline.py @@ -0,0 +1,198 @@ +"""Tier 3 - Neo4j integration proof for the SchemaMeta baseline singleton (§10.2). + +BASELINE DATA POINTS ONLY: this proves ``ensure_schema_version_baseline``'s +``:SchemaMeta {id: 'singleton'}`` write is create-if-absent (``ON CREATE SET`` +only, no ``ON MATCH SET``) and O(1) -- calling it twice must leave exactly one +node in place with an UNCHANGED `last_updated`, proving the second call did +not clobber it. It also proves the uniqueness constraint on +``(:SchemaMeta).id`` is actually created, and that the constraint is what +makes the singleton race-free under real concurrency (N concurrent callers +still leave exactly one node) -- the whole point of BLOCKER-3's hardening +follow-up: moving the write off the per-worker-flush path onto a +single-writer startup call, backed by a uniqueness constraint so even a +violation of that single-writer invariant could never create a duplicate. + +Also proves ``ensure_neo4j_schema`` itself no longer writes SchemaMeta at +all -- that responsibility now belongs exclusively to +``ensure_schema_version_baseline``, called once from the lifespan startup +handler, never from the per-flush / doctor-repair paths that call +``ensure_neo4j_schema``. + +No comparison/upgrade/migration logic is exercised here; there is none to +exercise -- that is the point of this test. + +Run: uv run pytest tests/neo4j/test_schema_version_baseline.py -v -m neo4j +""" + +from __future__ import annotations + +import asyncio +from typing import Any + +import pytest +from context_intelligence_server.neo4j_store import ( + ensure_neo4j_schema, + ensure_schema_version_baseline, +) +from context_intelligence_server.status import SCHEMA_VERSION +from neo4j import AsyncGraphDatabase + + +async def _schema_meta_rows(driver: Any) -> list[Any]: + """Return all :SchemaMeta{id:'singleton'} rows (schema_version, last_updated).""" + async with driver.session() as session: + result = await session.run( + "MATCH (m:SchemaMeta {id: 'singleton'}) " + "RETURN m.schema_version AS schema_version, " + "m.last_updated AS last_updated" + ) + return [record async for record in result] + + +@pytest.mark.neo4j +class TestSchemaMetaBaselineIdempotence: + """``ensure_schema_version_baseline`` writes the singleton create-if-absent only.""" + + async def test_second_call_does_not_clobber_first( + self, neo4j_container: dict[str, Any] + ) -> None: + auth = (neo4j_container["user"], neo4j_container["password"]) + bolt = neo4j_container["bolt_url"] + + driver = AsyncGraphDatabase.driver(bolt, auth=auth) + try: + # First call: creates the singleton (and the uniqueness constraint). + await ensure_schema_version_baseline(driver) + + rows = await _schema_meta_rows(driver) + assert len(rows) == 1, ( + f"expected exactly one :SchemaMeta singleton after first call, " + f"got {len(rows)}" + ) + assert rows[0]["schema_version"] == SCHEMA_VERSION + first_last_updated = rows[0]["last_updated"] + assert first_last_updated is not None + + # Second call: must be a no-op on the existing node (ON CREATE only). + await ensure_schema_version_baseline(driver) + + rows_after = await _schema_meta_rows(driver) + assert len(rows_after) == 1, ( + f"expected exactly one :SchemaMeta node after second call " + f"(no duplicate created), got {len(rows_after)}" + ) + assert rows_after[0]["last_updated"] == first_last_updated, ( + "last_updated changed on the second call -- ON MATCH SET must " + "not be present; the singleton must be left untouched once it " + "exists" + ) + assert rows_after[0]["schema_version"] == SCHEMA_VERSION + finally: + await driver.close() + + async def test_uniqueness_constraint_exists( + self, neo4j_container: dict[str, Any] + ) -> None: + """The (:SchemaMeta).id uniqueness constraint is created and enforced.""" + auth = (neo4j_container["user"], neo4j_container["password"]) + bolt = neo4j_container["bolt_url"] + + driver = AsyncGraphDatabase.driver(bolt, auth=auth) + try: + await ensure_schema_version_baseline(driver) + + async with driver.session() as session: + result = await session.run("SHOW CONSTRAINTS") + constraints = [record async for record in result] + + schema_meta_constraints = [ + c + for c in constraints + if "SchemaMeta" in (c.get("labelsOrTypes") or []) + and "id" in (c.get("properties") or []) + ] + assert schema_meta_constraints, ( + "expected a uniqueness constraint on (:SchemaMeta).id to exist " + f"after ensure_schema_version_baseline; SHOW CONSTRAINTS returned: " + f"{constraints}" + ) + + # Belt-and-suspenders: attempting to create a second singleton node + # directly (bypassing MERGE) must be rejected by the constraint. + with pytest.raises(Exception): # noqa: B017 - Neo4jError subtype + async with driver.session() as session: + await session.run("CREATE (m:SchemaMeta {id: 'singleton'})") + finally: + await driver.close() + + async def test_concurrent_calls_create_exactly_one_node( + self, neo4j_container: dict[str, Any] + ) -> None: + """N concurrent baseline calls against the same DB leave exactly one node. + + This is the whole point of the uniqueness-constraint hardening: without + it, concurrent MERGEs on a fresh singleton key can each pass the + existence check and create divergent duplicate nodes. Fire many + concurrent calls (each on its own driver, mirroring independent + SessionWorker stores) and assert the constraint prevents any + duplication. + """ + auth = (neo4j_container["user"], neo4j_container["password"]) + bolt = neo4j_container["bolt_url"] + + n_concurrent = 20 + drivers = [ + AsyncGraphDatabase.driver(bolt, auth=auth) for _ in range(n_concurrent) + ] + try: + await asyncio.gather(*(ensure_schema_version_baseline(d) for d in drivers)) + + rows = await _schema_meta_rows(drivers[0]) + assert len(rows) == 1, ( + f"expected exactly one :SchemaMeta singleton after " + f"{n_concurrent} concurrent calls, got {len(rows)} -- the " + "uniqueness constraint should make concurrent creation race-free" + ) + assert rows[0]["schema_version"] == SCHEMA_VERSION + finally: + for d in drivers: + await d.close() + + +@pytest.mark.neo4j +class TestEnsureNeo4jSchemaNoLongerWritesSchemaMeta: + """``ensure_neo4j_schema`` must not touch :SchemaMeta at all (BLOCKER-3 follow-up). + + That responsibility moved exclusively to ``ensure_schema_version_baseline``, + called once from the lifespan startup handler -- NOT from the per-flush / + doctor-repair paths that call ``ensure_neo4j_schema``. If ``ensure_neo4j_schema`` + still created the singleton, it would fire redundantly (and concurrently) on + every SessionWorker's first flush. + """ + + async def test_ensure_neo4j_schema_does_not_create_schema_meta( + self, neo4j_container: dict[str, Any] + ) -> None: + auth = (neo4j_container["user"], neo4j_container["password"]) + bolt = neo4j_container["bolt_url"] + + driver = AsyncGraphDatabase.driver(bolt, auth=auth) + try: + # Remove any pre-existing singleton so this test is unambiguous + # regardless of what earlier tests in this (session-scoped) + # container have already written. + async with driver.session() as session: + await session.run( + "MATCH (m:SchemaMeta {id: 'singleton'}) DETACH DELETE m" + ) + + await ensure_neo4j_schema(driver) + + rows = await _schema_meta_rows(driver) + assert rows == [], ( + "ensure_neo4j_schema must not write the :SchemaMeta singleton " + f"-- that is ensure_schema_version_baseline's job now, but " + f"found {len(rows)} node(s)" + ) + finally: + await driver.close() diff --git a/tests/neo4j/test_tag_legacy_pooled_iterations.py b/tests/neo4j/test_tag_legacy_pooled_iterations.py new file mode 100644 index 00000000..64959b17 --- /dev/null +++ b/tests/neo4j/test_tag_legacy_pooled_iterations.py @@ -0,0 +1,254 @@ +"""Live Neo4j test coverage for scripts/tag_legacy_pooled_iterations.py. + +The script's own module docstring states it has **NO test coverage** ("NOT +product code -- no unit tests"), despite mutating a live graph (it ``SET``s a +non-destructive marker property on a confirmed-corrupt subset of :Iteration +nodes). This module closes that gap against a REAL Neo4j test container (see +tests/neo4j/conftest.py), mirroring the direct-function-import pattern used +by ``test_relabel_incomplete_sessions.py`` -- both scripts share the same +``CALL { ... } IN TRANSACTIONS OF N ROWS`` batching shape and both expose +their session-taking functions (``run_dry_run``, ``run_apply``, +``tag_confirmed_corrupt``) as directly importable/testable seams, so no +subprocess/CLI invocation is needed. + +Covers: + +(a) Selector precision -- an :Iteration node reached by ``HAS_PART`` from + ``>=2`` distinct :OrchestratorRun nodes (the confirmed-corrupt / legacy + pooled shape) IS tagged, and a node with exactly ONE run parent (the + clean, single-run shape that is the large majority of bare-id nodes in + live data per the script's own docstring) is NEVER tagged. Both shapes + are seeded in the SAME graph so the selector's precision is proven, not + just its recall. +(b) The --apply gate -- ``run_dry_run()`` (the default / non-apply + invocation) performs ZERO writes: the pooled node is left untagged. + Only ``run_apply()`` (the --apply path) sets the marker. +(c) Idempotence -- a second ``tag_confirmed_corrupt``/``run_apply`` call + against an already-tagged graph tags/matches zero additional rows, and + a subsequent ``run_apply()`` still reports success (exit 0). + +Run: uv run pytest tests/neo4j/test_tag_legacy_pooled_iterations.py -v -m neo4j +""" + +from __future__ import annotations + +from typing import Any + +import pytest +from neo4j import GraphDatabase +from scripts import tag_legacy_pooled_iterations as tagger + +WORKSPACE = "test" + +pytestmark = pytest.mark.neo4j + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _driver(neo4j_container: dict) -> GraphDatabase: + """Return a synchronous Neo4j driver for the test container.""" + return GraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + + +def _seed_run(session, run_id: str) -> None: + """Seed a bare :OrchestratorRun node.""" + session.run( + "MERGE (r:OrchestratorRun {node_id: $run_id, workspace: $workspace})", + run_id=run_id, + workspace=WORKSPACE, + ) + + +def _seed_bare_iteration(session, iter_id: str) -> None: + """Seed a bare-id :Iteration node. + + ``iter_id`` must NOT contain ``'::orch_run::'`` -- that is the pre-I5, + per-session-counter shape the script's ``_CONFIRMED_CORRUPT_MATCH`` + selector considers at all. Run-scoped (post-I5) node_ids are excluded by + the selector's ``WHERE NOT i.node_id CONTAINS '::orch_run::'`` clause. + """ + session.run( + "MERGE (i:Iteration {node_id: $iter_id, workspace: $workspace})", + iter_id=iter_id, + workspace=WORKSPACE, + ) + + +def _link(session, run_id: str, iter_id: str) -> None: + """Seed the real ``(OrchestratorRun)-[:HAS_PART]->(Iteration)`` edge + shape the script's selector matches (``MATCH (run:OrchestratorRun) + -[:HAS_PART]->(i:Iteration)``).""" + session.run( + "MATCH (r:OrchestratorRun {node_id: $run_id, workspace: $workspace}) " + "MATCH (i:Iteration {node_id: $iter_id, workspace: $workspace}) " + "MERGE (r)-[:HAS_PART]->(i)", + run_id=run_id, + iter_id=iter_id, + workspace=WORKSPACE, + ) + + +def _data_quality(session, iter_id: str) -> str | None: + """Return the ``data_quality`` property of an :Iteration node, or None.""" + result = session.run( + "MATCH (i:Iteration {node_id: $iter_id, workspace: $workspace}) " + "RETURN i.data_quality AS dq", + iter_id=iter_id, + workspace=WORKSPACE, + ) + record = result.single() + if record is None: + return None + return record["dq"] + + +def _seed_pooled_pair(session) -> None: + """Seed one pooled (>=2 run parents) and one clean (1 run parent) + bare-id Iteration node -- the shared fixture used by every test below.""" + _seed_run(session, "run-1") + _seed_run(session, "run-2") + _seed_bare_iteration(session, "sess-pooled::iteration::1") + _link(session, "run-1", "sess-pooled::iteration::1") + _link(session, "run-2", "sess-pooled::iteration::1") + + _seed_run(session, "run-3") + _seed_bare_iteration(session, "sess-clean::iteration::1") + _link(session, "run-3", "sess-clean::iteration::1") + + +# --------------------------------------------------------------------------- +# Per-test isolation +# --------------------------------------------------------------------------- + + +@pytest.fixture(autouse=True) +def _clear_neo4j(neo4j_container: dict) -> None: # type: ignore[return] + """Wipe the container clean before each test for complete isolation.""" + driver = GraphDatabase.driver( + neo4j_container["bolt_url"], + auth=(neo4j_container["user"], neo4j_container["password"]), + ) + try: + with driver.session() as s: + s.run("MATCH (n) DETACH DELETE n") + finally: + driver.close() + + +# --------------------------------------------------------------------------- +# (a) selector precision: >=2 parents tagged, exactly-1 parent NEVER tagged +# --------------------------------------------------------------------------- + + +@pytest.mark.neo4j +class TestSelectorPrecision: + def test_pooled_node_tagged_clean_node_untouched( + self, neo4j_container: dict[str, Any] + ) -> None: + driver = _driver(neo4j_container) + try: + with driver.session() as s: + _seed_pooled_pair(s) + + exit_code = tagger.run_apply(s, batch_size=tagger.DEFAULT_BATCH_SIZE) + + assert exit_code == 0 + assert ( + _data_quality(s, "sess-pooled::iteration::1") == tagger.TAG_VALUE + ), "the >=2-distinct-run-parent node must be tagged" + assert _data_quality(s, "sess-clean::iteration::1") is None, ( + "the exactly-1-run-parent (clean, single-run) node must " + "NEVER be tagged -- this is exactly the shape the " + "script's own docstring says is the majority of live " + "bare-id nodes and must be left untouched" + ) + finally: + driver.close() + + +# --------------------------------------------------------------------------- +# (b) --apply gate: dry-run makes NO mutation; only --apply writes +# --------------------------------------------------------------------------- + + +@pytest.mark.neo4j +class TestApplyGate: + def test_dry_run_makes_no_mutation_only_apply_writes( + self, neo4j_container: dict[str, Any] + ) -> None: + driver = _driver(neo4j_container) + try: + with driver.session() as s: + _seed_pooled_pair(s) + + # The default / non-apply invocation: run_dry_run() only + # calls classify() (read-only counts, no SET anywhere in its + # call graph). It must leave the confirmed-corrupt node + # completely untagged. + dry_exit_code = tagger.run_dry_run(s) + assert dry_exit_code == 0 + assert _data_quality(s, "sess-pooled::iteration::1") is None, ( + "run_dry_run must make ZERO writes -- the confirmed-" + "corrupt pooled node must remain untagged after a " + "dry-run invocation" + ) + assert _data_quality(s, "sess-clean::iteration::1") is None + + # Only --apply (run_apply) is permitted to write the tag. + apply_exit_code = tagger.run_apply( + s, batch_size=tagger.DEFAULT_BATCH_SIZE + ) + assert apply_exit_code == 0 + assert ( + _data_quality(s, "sess-pooled::iteration::1") == tagger.TAG_VALUE + ), "run_apply must tag the pooled node once explicitly invoked" + finally: + driver.close() + + +# --------------------------------------------------------------------------- +# (c) idempotence: second apply tags/matches zero additional rows +# --------------------------------------------------------------------------- + + +@pytest.mark.neo4j +class TestIdempotence: + def test_second_apply_tags_zero_additional_rows( + self, neo4j_container: dict[str, Any] + ) -> None: + driver = _driver(neo4j_container) + try: + with driver.session() as s: + _seed_pooled_pair(s) + + first_tagged = tagger.tag_confirmed_corrupt( + s, batch_size=tagger.DEFAULT_BATCH_SIZE + ) + second_tagged = tagger.tag_confirmed_corrupt( + s, batch_size=tagger.DEFAULT_BATCH_SIZE + ) + + assert first_tagged == 1, ( + "expected exactly 1 node tagged on the first apply " + f"(the pooled node only), got {first_tagged}" + ) + assert second_tagged == 0, ( + "a second apply against an already-tagged graph must " + f"match and write zero rows (idempotent), got {second_tagged}" + ) + assert _data_quality(s, "sess-pooled::iteration::1") == tagger.TAG_VALUE + assert _data_quality(s, "sess-clean::iteration::1") is None + + # The full run_apply() gate (not just the raw write helper) + # must also report success and find nothing outstanding on + # a third, still-idempotent re-run -- no error, no double-tag. + exit_code = tagger.run_apply(s, batch_size=tagger.DEFAULT_BATCH_SIZE) + assert exit_code == 0 + finally: + driver.close() diff --git a/tests/neo4j/test_working_dir_non_overwrite.py b/tests/neo4j/test_working_dir_non_overwrite.py new file mode 100644 index 00000000..72a0f20e --- /dev/null +++ b/tests/neo4j/test_working_dir_non_overwrite.py @@ -0,0 +1,158 @@ +"""Live E2E tests: working_dir is never clobbered at the DB level (W-3 defect 2). + +Root cause being guarded here: prior to this fix, the ONLY guarantee that an +already-populated ``working_dir`` is never overwritten lived in the Python layer +(``services.py``'s ``if data.get("working_dir") and not existing.get("working_dir")`` +populate-if-missing check). That check reads the buffered/graph node BEFORE the +write is issued, so it cannot protect against a cross-writer or replica race: a +second concurrent flush that read the node before the first write committed would +still see no working_dir, and its ``SET n += row.props`` would blindly overwrite +whatever the first writer just set. + +The fix adds a genuine DB-level guarantee: the Session-node MERGE in +``_write_batch`` excludes working_dir from the blind ``+=`` merge and instead +applies ``SET n.working_dir = coalesce(n.working_dir, row.working_dir)`` -- a +non-overwrite rule enforced by Neo4j itself, at the same MERGE lock hold, immune +to read-then-write races between writers. + +Requires Docker and the docker Python package. Skip-if-absent via the +``neo4j_container`` fixture in tests/neo4j/conftest.py. + +Run explicitly: + cd amplifier-context-intelligence + uv run pytest tests/neo4j/test_working_dir_non_overwrite.py -v -m neo4j +""" + +from __future__ import annotations + +from typing import Any + +import pytest +from context_intelligence_server.neo4j_store import Neo4jGraphStore +from neo4j import GraphDatabase + +pytestmark = pytest.mark.neo4j + + +async def _flush_session_node( + container: dict[str, Any], node_id: str, data: dict[str, Any] +) -> None: + """Drive a single Session node through the real flush path via a FRESH store. + + A fresh ``Neo4jGraphStore`` per call simulates independent writers (e.g. two + drainer workers, or a writer racing a replica) rather than two writes queued + through the same in-process buffer -- the scenario the Python-layer + populate-if-missing check cannot see. + """ + store = Neo4jGraphStore( + uri=container["bolt_url"], + auth=(container["user"], container["password"]), + workspace="test", + ) + try: + await store.upsert_node(node_id, {"labels": ["Session"], **data}) + await store.flush() + finally: + await store.close() + + +def _read_working_dir(container: dict[str, Any], node_id: str) -> Any: + driver = GraphDatabase.driver( + container["bolt_url"], auth=(container["user"], container["password"]) + ) + try: + with driver.session() as session: + rec = session.run( + "MATCH (n:Session {node_id: $nid, workspace: $ws}) " + "RETURN n.working_dir AS wd", + nid=node_id, + ws="test", + ).single() + assert rec is not None, f"Session node {node_id!r} was not written" + return rec["wd"] + finally: + driver.close() + + +async def test_existing_working_dir_survives_conflicting_later_write( + neo4j_container: dict[str, Any], +) -> None: + """DB-level guarantee: an already-set working_dir is never overwritten, + even by a second independent writer (simulating a cross-writer/replica race). + + RED (unfixed): ``SET n += row.props`` blindly overwrites -> working_dir + becomes "/y" -> this assertion fails. + GREEN (fixed): ``coalesce(n.working_dir, row.working_dir)`` keeps the + existing value -> working_dir stays "/x". + """ + node_id = "sess-wd-no-clobber-live" + + # Writer 1: establishes working_dir="/x". + await _flush_session_node( + neo4j_container, + node_id, + {"status": "running", "working_dir": "/x"}, + ) + assert _read_working_dir(neo4j_container, node_id) == "/x" + + # Writer 2 (independent store instance): tries to write a DIFFERENT + # working_dir for the SAME node -- must be rejected at the DB level. + await _flush_session_node( + neo4j_container, + node_id, + {"status": "running", "working_dir": "/y"}, + ) + + assert _read_working_dir(neo4j_container, node_id) == "/x", ( + "An already-populated working_dir must never be overwritten by a " + "later/concurrent writer -- DB-level coalesce guarantee failed" + ) + + +async def test_working_dir_fills_gap_at_db_level_when_previously_absent( + neo4j_container: dict[str, Any], +) -> None: + """DB-level populate-if-missing: a node with no working_dir gets filled in + by a later write that supplies one (coalesce(null, value) -> value). + + Mirrors the Python-layer guarantee (services.py) but proves it also holds + purely at the DB level, independent of the in-process buffer. + """ + node_id = "sess-wd-fill-gap-live" + + # Writer 1: no working_dir supplied. + await _flush_session_node(neo4j_container, node_id, {"status": "running"}) + assert _read_working_dir(neo4j_container, node_id) is None + + # Writer 2: supplies working_dir for the first time. + await _flush_session_node( + neo4j_container, + node_id, + {"status": "running", "working_dir": "/first-value"}, + ) + + assert _read_working_dir(neo4j_container, node_id) == "/first-value", ( + "working_dir must be filled in at the DB level once a writer supplies " + "a value for a node that previously had none" + ) + + +async def test_working_dir_absent_write_does_not_clear_existing_value( + neo4j_container: dict[str, Any], +) -> None: + """A later write that omits working_dir entirely must not null out an + already-set value (coalesce(n.working_dir, null) -> unchanged). + """ + node_id = "sess-wd-absent-no-clear-live" + + await _flush_session_node( + neo4j_container, node_id, {"status": "running", "working_dir": "/keep-me"} + ) + assert _read_working_dir(neo4j_container, node_id) == "/keep-me" + + # Second write carries no working_dir key at all (e.g. a touch/heartbeat event). + await _flush_session_node(neo4j_container, node_id, {"status": "still-running"}) + + assert _read_working_dir(neo4j_container, node_id) == "/keep-me", ( + "A write that omits working_dir must never null out an already-set value" + ) diff --git a/tests/routers/test_admin_maintenance.py b/tests/routers/test_admin_maintenance.py new file mode 100644 index 00000000..bd9d237e --- /dev/null +++ b/tests/routers/test_admin_maintenance.py @@ -0,0 +1,336 @@ +"""Tests for POST/GET /admin/maintenance (WS-3c) and the small WS-3b op +wiring (maintenance_ops.run_maintenance_operation). + +Covers the WS-3 spec's unit test plan (sec 7c), items C1-C6: +single-flight 409/202 (C1, C2), prompt-return (C3), honest idempotent +re-scan (C4), failure recording (C5), and admin-auth enforcement (C6). C7-C9 +are DTU-only (real Neo4j + real restart) and are out of scope here. + +None of these tests touch a real Neo4j: ``run_maintenance_operation`` (or, +for C5, the underlying ``neo4j_store.run_repair`` it wraps) is monkeypatched +per-test so the CAS/HTTP/bookkeeping logic under test is exercised without a +live driver. ``reset_maintenance_coordinator`` (conftest.py, autouse) resets +the process-wide coordinator singleton before and after every test. +""" + +from __future__ import annotations + +import asyncio +import hashlib +from collections.abc import AsyncGenerator +from pathlib import Path +from typing import Any + +import httpx +import pytest +from context_intelligence_server.maintenance import coordinator + +# --------------------------------------------------------------------------- +# Fixtures -- admin-override client (no real auth; require_admin no-op'd) +# --------------------------------------------------------------------------- + + +@pytest.fixture +async def admin_client() -> AsyncGenerator[httpx.AsyncClient, None]: + """Client with require_admin overridden to a no-op (T4-style override). + + Used for C1-C5: these tests exercise the endpoint's OWN logic (CAS, + prompt-return, status reporting), not the auth layer -- that is C6's job, + which uses a real, non-overridden auth client below. + """ + from context_intelligence_server.main import app + from context_intelligence_server.routers.admin import require_admin + + app.dependency_overrides[require_admin] = lambda: None + try: + async with httpx.AsyncClient( + transport=httpx.ASGITransport(app=app), base_url="http://test" + ) as c: + yield c + finally: + app.dependency_overrides.pop(require_admin, None) + + +def _stub_op(*, sleep_seconds: float = 0.0, records_affected: int = 0) -> Any: + """Build a fake ``run_maintenance_operation`` replacement. + + Signature-compatible with the real function: ``(driver, run_id, *, + quiesce_seconds)``. Reports success via ``coordinator.finish_op`` after + an optional sleep, exactly like the real function would after its own + quiesce + run_repair call -- but without touching Neo4j. + """ + + async def _fake(driver: Any, run_id: str, *, quiesce_seconds: float) -> None: + if sleep_seconds: + await asyncio.sleep(sleep_seconds) + coordinator.finish_op(run_id, records_affected=records_affected, error=None) + + return _fake + + +# --------------------------------------------------------------------------- +# C1 -- POST while an op runs -> 409 with the current run_id +# --------------------------------------------------------------------------- + + +class TestSingleFlight409: + async def test_post_while_running_returns_409_with_current_run_id( + self, admin_client: httpx.AsyncClient, monkeypatch: pytest.MonkeyPatch + ) -> None: + # Never-finishing stub -- the op stays "running" for the whole test. + monkeypatch.setattr( + "context_intelligence_server.routers.admin.run_maintenance_operation", + _stub_op(sleep_seconds=10.0), + ) + + first = await admin_client.post("/admin/maintenance") + assert first.status_code == 202 + first_run_id = first.json()["run_id"] + assert first.json()["state"] == "running" + + second = await admin_client.post("/admin/maintenance") + assert second.status_code == 409 + body = second.json() + assert body["run_id"] == first_run_id + assert body["state"] == "running" + assert "already running" in body["detail"] + + # -- C2: 20 concurrent POSTs -> exactly one 202, nineteen 409 ----------- + + async def test_twenty_concurrent_posts_yield_exactly_one_202( + self, admin_client: httpx.AsyncClient, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr( + "context_intelligence_server.routers.admin.run_maintenance_operation", + _stub_op(sleep_seconds=0.05), + ) + + responses = await asyncio.gather( + *[admin_client.post("/admin/maintenance") for _ in range(20)] + ) + statuses = [r.status_code for r in responses] + assert statuses.count(202) == 1, ( + f"expected exactly one 202 among 20 concurrent POSTs, got: {statuses}" + ) + assert statuses.count(409) == 19 + + +# --------------------------------------------------------------------------- +# C3 -- POST returns promptly; does NOT await the op's full duration +# --------------------------------------------------------------------------- + + +class TestReturnsPromptly: + async def test_post_returns_before_op_completes( + self, admin_client: httpx.AsyncClient, monkeypatch: pytest.MonkeyPatch + ) -> None: + """MUST-FIX #5a: the handler must not block for the op's duration. + + The stub sleeps 0.5s before finishing; the POST itself must return in + well under that, and an immediate GET must still observe "running". + """ + monkeypatch.setattr( + "context_intelligence_server.routers.admin.run_maintenance_operation", + _stub_op(sleep_seconds=0.5), + ) + + loop = asyncio.get_event_loop() + start = loop.time() + resp = await admin_client.post("/admin/maintenance") + elapsed = loop.time() - start + + assert resp.status_code == 202 + assert elapsed < 0.3, ( + f"POST took {elapsed:.3f}s -- should return well before the " + f"0.5s op completes (MUST-FIX #5a: prompt return)" + ) + + get_resp = await admin_client.get("/admin/maintenance") + assert get_resp.json()["state"] == "running" + + +# --------------------------------------------------------------------------- +# C4 -- POST on a clean graph: honest re-scan, fresh run_id + completed_at +# --------------------------------------------------------------------------- + + +class TestCleanGraphHonestRescan: + async def test_post_on_clean_graph_reports_succeeded_zero_affected( + self, admin_client: httpx.AsyncClient, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr( + "context_intelligence_server.routers.admin.run_maintenance_operation", + _stub_op(sleep_seconds=0.0, records_affected=0), + ) + + resp = await admin_client.post("/admin/maintenance") + assert resp.status_code == 202 + run_id = resp.json()["run_id"] + assert resp.json()["started_at"] is not None + + # Let the (instant) stub task actually run before polling GET. + await asyncio.sleep(0.02) + + get_resp = await admin_client.get("/admin/maintenance") + body = get_resp.json() + assert body["state"] == "succeeded" + assert body["run_id"] == run_id + assert body["records_affected"] == 0 + assert body["completed_at"] is not None + assert body["error"] is None + + async def test_second_post_after_completion_gets_a_fresh_run_id( + self, admin_client: httpx.AsyncClient, monkeypatch: pytest.MonkeyPatch + ) -> None: + """D-H: re-running on an already-clean graph is a genuine re-scan -- + not a short-circuit -- proved by a NEW run_id/completed_at each time. + """ + monkeypatch.setattr( + "context_intelligence_server.routers.admin.run_maintenance_operation", + _stub_op(sleep_seconds=0.0, records_affected=0), + ) + + first = await admin_client.post("/admin/maintenance") + assert first.status_code == 202 + first_run_id = first.json()["run_id"] + await asyncio.sleep(0.02) + + second = await admin_client.post("/admin/maintenance") + assert second.status_code == 202 + second_run_id = second.json()["run_id"] + await asyncio.sleep(0.02) + + assert second_run_id != first_run_id + get_resp = await admin_client.get("/admin/maintenance") + assert get_resp.json()["run_id"] == second_run_id + assert get_resp.json()["state"] == "succeeded" + + +# --------------------------------------------------------------------------- +# C5 -- op raises -> state "failed", error populated and persists +# --------------------------------------------------------------------------- + + +class TestOpFailure: + async def test_run_repair_exception_is_recorded_as_failed( + self, admin_client: httpx.AsyncClient, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Exercises the REAL run_maintenance_operation's except-clause (only + the underlying run_repair call is stubbed to raise) -- proving the + actual exception -> finish_op(error=...) wiring, not a re-implemented + fake of it. + """ + + async def _raising_run_repair(driver: Any, database: str = "neo4j") -> Any: + raise RuntimeError("simulated repair failure") + + monkeypatch.setattr( + "context_intelligence_server.maintenance_ops.run_repair", + _raising_run_repair, + ) + # Skip the quiesce sleep so the test doesn't wait on the 2.0s default. + monkeypatch.setattr( + "context_intelligence_server.routers.admin.get_settings", + lambda: _FakeSettingsZeroQuiesce(), + ) + + resp = await admin_client.post("/admin/maintenance") + assert resp.status_code == 202 + run_id = resp.json()["run_id"] + + # No quiesce sleep configured -- give the task one scheduling slot. + await asyncio.sleep(0.02) + + get_resp = await admin_client.get("/admin/maintenance") + body = get_resp.json() + assert body["state"] == "failed" + assert body["run_id"] == run_id + assert body["error"] is not None + assert "simulated repair failure" in body["error"] + assert body["completed_at"] is not None + + # Persistence: a second GET still shows the same failed record. + get_resp_2 = await admin_client.get("/admin/maintenance") + assert get_resp_2.json()["state"] == "failed" + assert get_resp_2.json()["error"] == body["error"] + + +class _FakeSettingsZeroQuiesce: + """Minimal settings stand-in: only the one attribute the endpoint reads.""" + + maintenance_quiesce_seconds = 0.0 + + +# --------------------------------------------------------------------------- +# C6 -- GET/POST without admin auth -> 401/403 (inherited require_admin) +# --------------------------------------------------------------------------- + +FAKE_ADMIN_RAW_KEY = "maint-test-admin-key-do-not-use" +FAKE_ADMIN_KEY_DIGEST = hashlib.sha256(FAKE_ADMIN_RAW_KEY.encode()).hexdigest() +FAKE_DATA_RAW_KEY = "maint-test-data-key-ordinary-user" +FAKE_DATA_KEY_DIGEST = hashlib.sha256(FAKE_DATA_RAW_KEY.encode()).hexdigest() +FAKE_CONTRIBUTOR = "maint-tester" + + +def _make_static_settings_with_admin(tmp_path: Path) -> Any: + from context_intelligence_server.config import Settings + + return Settings( + auth_mode="static", + api_keys={FAKE_DATA_KEY_DIGEST: {"id": FAKE_CONTRIBUTOR}}, + admin_api_key=FAKE_ADMIN_RAW_KEY, + api_keys_store_path=str(tmp_path / "api-keys.json"), + entra_identities_store_path=str(tmp_path / "entra-identities.json"), + ) + + +@pytest.fixture +async def real_auth_client(tmp_path: Path) -> AsyncGenerator[httpx.AsyncClient, None]: + """Client routed through the REAL auth middleware (no override) -- for + C6, proving /admin/maintenance inherits require_admin's fail-closed + 401/403 matrix exactly like every other /admin/* route.""" + from context_intelligence_server.main import create_asgi_app + + settings = _make_static_settings_with_admin(tmp_path) + middleware = create_asgi_app(settings=settings) + async with httpx.AsyncClient( + transport=httpx.ASGITransport(app=middleware), base_url="http://test" + ) as c: + yield c + + +class TestAdminAuthEnforced: + async def test_post_no_token_401(self, real_auth_client: httpx.AsyncClient) -> None: + resp = await real_auth_client.post("/admin/maintenance") + assert resp.status_code == 401 + + async def test_get_no_token_401(self, real_auth_client: httpx.AsyncClient) -> None: + resp = await real_auth_client.get("/admin/maintenance") + assert resp.status_code == 401 + + async def test_post_data_key_403(self, real_auth_client: httpx.AsyncClient) -> None: + resp = await real_auth_client.post( + "/admin/maintenance", + headers={"Authorization": f"Bearer {FAKE_DATA_RAW_KEY}"}, + ) + assert resp.status_code == 403 + + async def test_get_data_key_403(self, real_auth_client: httpx.AsyncClient) -> None: + resp = await real_auth_client.get( + "/admin/maintenance", + headers={"Authorization": f"Bearer {FAKE_DATA_RAW_KEY}"}, + ) + assert resp.status_code == 403 + + async def test_get_admin_key_200( + self, real_auth_client: httpx.AsyncClient, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Sanity: the admin key itself DOES authenticate + authorize (no + false-positive 401/403 for the correct credential).""" + resp = await real_auth_client.get( + "/admin/maintenance", + headers={"Authorization": f"Bearer {FAKE_ADMIN_RAW_KEY}"}, + ) + assert resp.status_code == 200 + body = resp.json() + assert body["state"] == "unknown" # never ran in this fresh coordinator diff --git a/tests/routers/test_blob_reclaim_auth.py b/tests/routers/test_blob_reclaim_auth.py new file mode 100644 index 00000000..2e1c4c75 --- /dev/null +++ b/tests/routers/test_blob_reclaim_auth.py @@ -0,0 +1,152 @@ +"""Non-neo4j auth test for POST /admin/blobs/reclaim. + +Mirrors the real-auth-enforcement pattern in ``test_admin_auth.py`` (static +mode, no ``require_admin`` dependency override): a data key authenticates via +BearerTokenMiddleware but is not the admin key, so ``require_admin`` must +reject it with 403 before the handler ever runs. This test never reaches +Neo4j or the filesystem selection logic -- it only proves the auth gate. + +Fake constants only -- never real credentials (see repo AGENTS.md / design +doc §0.3 convention followed by test_admin_auth.py). +""" + +from __future__ import annotations + +import hashlib +from collections.abc import AsyncGenerator +from pathlib import Path +from typing import Any + +import httpx +import pytest + +FAKE_ADMIN_RAW_KEY = "blob-reclaim-admin-key-do-not-use" +FAKE_ADMIN_KEY_DIGEST = hashlib.sha256(FAKE_ADMIN_RAW_KEY.encode()).hexdigest() + +FAKE_DATA_RAW_KEY = "blob-reclaim-data-key-ordinary-user" +FAKE_DATA_KEY_DIGEST = hashlib.sha256(FAKE_DATA_RAW_KEY.encode()).hexdigest() +FAKE_CONTRIBUTOR = "carol" + + +def _make_static_settings(tmp_path: Path) -> Any: + from context_intelligence_server.config import Settings + + return Settings( + auth_mode="static", + api_keys={FAKE_DATA_KEY_DIGEST: {"id": FAKE_CONTRIBUTOR}}, + admin_api_key=FAKE_ADMIN_RAW_KEY, + api_keys_store_path=str(tmp_path / "api-keys.json"), + entra_identities_store_path=str(tmp_path / "entra-identities.json"), + ) + + +@pytest.fixture +async def static_auth_client(tmp_path: Path) -> AsyncGenerator[httpx.AsyncClient, None]: + """Client routed through the REAL auth middleware (no require_admin override).""" + from context_intelligence_server.main import create_asgi_app + + settings = _make_static_settings(tmp_path) + middleware = create_asgi_app(settings=settings) + async with httpx.AsyncClient( + transport=httpx.ASGITransport(app=middleware), base_url="http://test" + ) as c: + yield c + + +@pytest.mark.anyio +async def test_data_key_403_on_blob_reclaim( + static_auth_client: httpx.AsyncClient, +) -> None: + """A data (non-admin) key authenticates but must get 403 on the reclaim route.""" + resp = await static_auth_client.post( + "/admin/blobs/reclaim", + json={"dry_run": True}, + headers={"Authorization": f"Bearer {FAKE_DATA_RAW_KEY}"}, + ) + assert resp.status_code == 403 + + +@pytest.mark.anyio +async def test_no_token_401_on_blob_reclaim( + static_auth_client: httpx.AsyncClient, +) -> None: + """No bearer token at all must 401 before reaching require_admin (TB-07).""" + resp = await static_auth_client.post("/admin/blobs/reclaim", json={"dry_run": True}) + assert resp.status_code == 401 + + +@pytest.mark.anyio +async def test_admin_key_reaches_handler_and_validates_body( + static_auth_client: httpx.AsyncClient, +) -> None: + """The admin key passes require_admin; a bad body (max_delete missing) still 422s. + + Proves auth is satisfied (not 401/403) and the handler's own body + validation runs -- without touching Neo4j or the filesystem scan. + """ + resp = await static_auth_client.post( + "/admin/blobs/reclaim", + json={"dry_run": False}, + headers={"Authorization": f"Bearer {FAKE_ADMIN_RAW_KEY}"}, + ) + assert resp.status_code == 422 + + +@pytest.mark.anyio +async def test_admin_key_min_age_below_floor_422( + static_auth_client: httpx.AsyncClient, +) -> None: + """min_age_minutes below the hard safety floor is rejected with 422.""" + resp = await static_auth_client.post( + "/admin/blobs/reclaim", + json={"dry_run": True, "min_age_minutes": 0}, + headers={"Authorization": f"Bearer {FAKE_ADMIN_RAW_KEY}"}, + ) + assert resp.status_code == 422 + + +@pytest.mark.anyio +async def test_admin_key_max_delete_zero_422( + static_auth_client: httpx.AsyncClient, +) -> None: + """max_delete=0 is rejected with 422 (Field(ge=1) schema boundary). + + Schema validation runs before the handler, so this never reaches + ``_select_orphans`` / Neo4j. + """ + resp = await static_auth_client.post( + "/admin/blobs/reclaim", + json={"dry_run": True, "max_delete": 0}, + headers={"Authorization": f"Bearer {FAKE_ADMIN_RAW_KEY}"}, + ) + assert resp.status_code == 422 + + +@pytest.mark.anyio +async def test_admin_key_max_delete_negative_422( + static_auth_client: httpx.AsyncClient, +) -> None: + """max_delete=-1 is rejected with 422. + + Regression test: without the ``ge=1`` floor, ``candidates[:-1]`` silently + inverts the cap into "delete all but one" instead of raising. + """ + resp = await static_auth_client.post( + "/admin/blobs/reclaim", + json={"dry_run": True, "max_delete": -1}, + headers={"Authorization": f"Bearer {FAKE_ADMIN_RAW_KEY}"}, + ) + assert resp.status_code == 422 + + +def test_max_delete_one_is_accepted_by_schema() -> None: + """max_delete=1 (the floor) passes schema validation, not rejected. + + Exercised at the model level (not via HTTP) because a valid body + proceeds into ``_select_orphans``, which requires a live Neo4j driver + on ``request.app.state`` -- out of scope for this auth/router test file. + """ + from context_intelligence_server.routers.admin import BlobReclaimBody + + body = BlobReclaimBody(dry_run=False, max_delete=1) + assert body.max_delete == 1 diff --git a/tests/routers/test_version.py b/tests/routers/test_version.py index 639941b6..be74645c 100644 --- a/tests/routers/test_version.py +++ b/tests/routers/test_version.py @@ -5,7 +5,8 @@ import httpx import pytest -from context_intelligence_server.status import SERVER_VERSION +import context_intelligence_server.main as main_module +from context_intelligence_server.status import SCHEMA_VERSION, SERVER_VERSION class TestGetVersion200: @@ -37,6 +38,63 @@ async def test_version_matches_server_version_constant( data = response.json() assert data["version"] == SERVER_VERSION + @pytest.mark.anyio + async def test_returns_schema_version_field( + self, client: httpx.AsyncClient + ) -> None: + response = await client.get("/version") + data = response.json() + assert "schema_version" in data + + @pytest.mark.anyio + async def test_schema_version_matches_constant( + self, client: httpx.AsyncClient + ) -> None: + response = await client.get("/version") + data = response.json() + assert data["schema_version"] == SCHEMA_VERSION + + +class TestSchemaVersionConstant: + """SCHEMA_VERSION is a plain integer baseline data point (§10.2).""" + + def test_schema_version_is_int(self) -> None: + assert isinstance(SCHEMA_VERSION, int) + + def test_schema_version_initial_value(self) -> None: + assert SCHEMA_VERSION == 1 + + +class TestGetVersionUnchangedByW4: + """W-4 added an advisory graph_schema_version signal to GET /status only. + + GET /version must stay exactly the cheap compiled-in-constant return it + was before -- no graph read, no new fields -- per the deliberate + read/write separation documented on ``ensure_schema_version_baseline`` + (neo4j_store.py) and ``read_graph_schema_version``'s docstring. + """ + + @pytest.mark.anyio + async def test_response_shape_unchanged(self, client: httpx.AsyncClient) -> None: + response = await client.get("/version") + data = response.json() + assert set(data.keys()) == {"version", "schema_version"} + assert "graph_schema_version" not in data + assert "schema_version_current" not in data + + @pytest.mark.anyio + async def test_works_with_no_neo4j_driver_configured( + self, client: httpx.AsyncClient, monkeypatch: pytest.MonkeyPatch + ) -> None: + """/version never touches app.state.neo4j_driver -- it must return + 200 even when no driver has been configured at all.""" + if hasattr(main_module.app.state, "neo4j_driver"): + monkeypatch.delattr(main_module.app.state, "neo4j_driver", raising=False) + + response = await client.get("/version") + assert response.status_code == 200 + assert response.json()["schema_version"] == SCHEMA_VERSION + class TestGetVersionNoAuth: """GET /version is accessible without credentials even when auth is enabled.""" diff --git a/tests/test_blob_carrier_allowlist.py b/tests/test_blob_carrier_allowlist.py new file mode 100644 index 00000000..43caebdc --- /dev/null +++ b/tests/test_blob_carrier_allowlist.py @@ -0,0 +1,151 @@ +"""Tests for the WS-5 blob-carrier allowlist runtime tripwire. + +Covers the single source of truth shared by the mint path +(``blob_processor.BLOB_REF_CARRIER_PROPERTIES`` / +``blob_processor.assert_carrier_registered``) and the reclaim-scan path +(``routers.admin._BLOB_REF_CARRIER_PROPERTIES`` / ``_BLOB_REF_SCAN_QUERY``): + +1. The allowlist is exactly the current 4-item tuple (regression lock). +2. ``routers.admin`` imports the SAME object -- no local re-declaration to + drift out of sync. +3. The generated Cypher scan query references exactly the allowlist's + properties -- no more, no less (mint/scan agreement, structurally). +4. ``assert_carrier_registered`` is non-vacuous: it raises for an + unregistered property and is a no-op for a registered one. +5. ``process_event_data`` (the real mint call path) propagates the + tripwire's exception -- fail-closed, BEFORE any blob is written -- when + its destination carrier ("data") is not registered, and completes + normally when it is. +""" + +from __future__ import annotations + +import re +from typing import Any +from unittest.mock import AsyncMock + +import pytest +from context_intelligence_server.blob_processor import ( + BLOB_REF_CARRIER_PROPERTIES, + UnregisteredBlobCarrierError, + assert_carrier_registered, + process_event_data, +) +from context_intelligence_server.routers.admin import ( + _BLOB_REF_CARRIER_PROPERTIES as admin_carrier_properties, +) +from context_intelligence_server.routers.admin import ( + _BLOB_REF_SCAN_QUERY, +) + +# --------------------------------------------------------------------------- +# 1. Regression lock -- current 4-item allowlist +# --------------------------------------------------------------------------- + + +def test_carrier_properties_locked() -> None: + """BLOB_REF_CARRIER_PROPERTIES is exactly the specified 4-item tuple.""" + assert BLOB_REF_CARRIER_PROPERTIES == ("data", "tool_input", "prompt", "response") + + +# --------------------------------------------------------------------------- +# 2. admin.py imports the SAME allowlist -- no local re-declaration +# --------------------------------------------------------------------------- + + +def test_admin_imports_same_allowlist_object() -> None: + """routers.admin re-exports blob_processor's tuple by identity, not a + hand-copied duplicate -- proves there is exactly ONE allowlist object.""" + assert admin_carrier_properties is BLOB_REF_CARRIER_PROPERTIES + + +# --------------------------------------------------------------------------- +# 3. Generated scan query references exactly the allowlist's properties +# --------------------------------------------------------------------------- + + +def test_scan_query_matches_allowlist_exactly() -> None: + """The Cypher query built for the reclaim scan mentions exactly the + properties in BLOB_REF_CARRIER_PROPERTIES -- no more, no less. + + This is the structural lock that makes mint/scan drift impossible: if a + property is ever added to (or removed from) the allowlist without the + query being regenerated from it, this test fails. + """ + referenced_props = set(re.findall(r"n\.(\w+)", _BLOB_REF_SCAN_QUERY)) + assert referenced_props == set(BLOB_REF_CARRIER_PROPERTIES) + + +# --------------------------------------------------------------------------- +# 4. assert_carrier_registered -- non-vacuous tripwire +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("carrier", list(BLOB_REF_CARRIER_PROPERTIES)) +def test_assert_carrier_registered_passes_for_registered_carrier( + carrier: str, +) -> None: + """A registered carrier (each of the current 4) does not trip the guard.""" + assert_carrier_registered(carrier) # must not raise + + +def test_assert_carrier_registered_raises_for_unregistered_carrier() -> None: + """An unregistered carrier property trips the guard immediately. + + Proves the tripwire is non-vacuous: it actually fires. Simulates the + exact future scenario described in WS-5 -- a new field-lifter/enricher + promoting a value onto a brand-new node property ("artifact_content") + that nobody added to BLOB_REF_CARRIER_PROPERTIES. + """ + with pytest.raises(UnregisteredBlobCarrierError, match="artifact_content"): + assert_carrier_registered("artifact_content") + + +# --------------------------------------------------------------------------- +# 5. process_event_data -- the real mint call path +# --------------------------------------------------------------------------- + + +async def test_process_event_data_fails_closed_when_data_carrier_unregistered( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """If "data" is (hypothetically) removed from the allowlist, the mint + call fails loud BEFORE writing any blob -- not after. + + Regression target: this is the exact failure mode WS-5 exists to catch + -- a carrier property silently dropping out of the allowlist while the + mint path keeps writing to it. blob_store.write must never be called: + the guard fires before any blob is persisted, matching "fail loud at the + source, not after a live blob is deleted." + """ + import context_intelligence_server.blob_processor as blob_processor_module + + monkeypatch.setattr( + blob_processor_module, + "BLOB_REF_CARRIER_PROPERTIES", + ("tool_input", "prompt", "response"), # "data" removed + ) + + data: dict[str, Any] = {"result": {"answer": 42}} + blob_store = AsyncMock() + blob_store.write = AsyncMock(return_value="ci-blob://sess/node__result") + + with pytest.raises(UnregisteredBlobCarrierError, match="'data'"): + await process_event_data(data, blob_store, "sess", "node") + + blob_store.write.assert_not_called() + # data must be untouched -- the guard fired before any mutation/write + assert data == {"result": {"answer": 42}} + + +async def test_process_event_data_succeeds_when_data_carrier_registered() -> None: + """Sanity/non-regression: with the real (unmodified) allowlist, the mint + path completes normally -- the guard does not false-trip on the + ordinary, correctly-registered path.""" + data: dict[str, Any] = {"result": {"answer": 42}} + blob_store = AsyncMock() + blob_store.write = AsyncMock(return_value="ci-blob://sess/node__result") + + await process_event_data(data, blob_store, "sess", "node") + + assert data["result"] == {"$blob_ref": "ci-blob://sess/node__result"} diff --git a/tests/test_deploy_safe_boot_process.py b/tests/test_deploy_safe_boot_process.py new file mode 100644 index 00000000..7ac8ce3d --- /dev/null +++ b/tests/test_deploy_safe_boot_process.py @@ -0,0 +1,236 @@ +"""REAL-PROCESS deploy-safe boot regression tests. + +Why this file exists (and why the unit tests in ``test_main.py`` were not +enough): the unit tests call ``lifespan()`` as a function inside the test +process, where ``/data`` and logging already work and the module is already +imported. That harness CANNOT catch a crash-loop caused by a step that runs +*before* the B1 try/except boundary -- e.g. ``setup_logging()`` doing +``mkdir('/data')`` on an unwritable path, or driver construction on a +misconfigured URL. A live boot test proved exactly that hole: the real +gunicorn worker died with ``PermissionError: '/data'`` from ``setup_logging``, +never reaching the boundary -> ``Worker failed to boot`` -> systemd +restart-loop, with NO Neo4j involvement at all. + +These tests launch the ACTUAL server entrypoint +(``context_intelligence_server.main:main`` -> ``run()`` -> gunicorn + +UvicornWorker -- the real deploy path) as a subprocess, pointed at an +UNREACHABLE Neo4j and with ALL data/log paths redirected under a tmp dir, +and assert the process stays alive and serves ``GET /status`` with a +non-green ``schema_health``. They need NO Neo4j -- that is the whole point: +a deploy against an unreachable graph must boot and serve, never crash-loop. + +Marked ``deploy_safe_boot`` so CI can select/deselect them explicitly. They +are self-contained (spare port, tmp paths) and tear down the subprocess. +""" + +from __future__ import annotations + +import contextlib +import json +import os +import signal +import socket +import subprocess +import sys +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + +import pytest + +pytestmark = pytest.mark.deploy_safe_boot + +_ENV_PREFIX = "AMPLIFIER_CONTEXT_INTELLIGENCE_SERVER_" +# Unreachable Neo4j: nothing listens on this localhost port, so a connection +# attempt fails fast (ECONNREFUSED) rather than hanging -- exactly the +# "graph unreachable at boot" scenario (ACA cold-start race / wrong URL). +_UNREACHABLE_NEO4J = "bolt://127.0.0.1:59999" + +# How long to give the real gunicorn worker to boot + serve. Generous because +# a real process fork + import + lifespan (with one fast-failing Neo4j probe) +# is slower than an in-process call, but bounded so a genuine crash-loop +# still fails the test quickly. +_BOOT_DEADLINE_S = 25.0 + + +def _free_port() -> int: + """Return a currently-free localhost TCP port.""" + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +def _base_env(tmp_path: Path, port: int, *, log_path: str) -> dict[str, str]: + """Build a subprocess env: all data/log paths under tmp, Neo4j unreachable. + + Every path the server writes to is redirected under *tmp_path* so the + real defaults (``/data/...``) are never touched, and ``log_path`` is + supplied by the caller so a variant can point it at a deliberately + unwritable location. + """ + env = dict(os.environ) + # WEB_CONCURRENCY must be unset/1 or run()'s _validate_single_worker trips. + env.pop("WEB_CONCURRENCY", None) + env.update( + { + f"{_ENV_PREFIX}SERVER_HOST": "127.0.0.1", + f"{_ENV_PREFIX}SERVER_PORT": str(port), + f"{_ENV_PREFIX}NEO4J_URL": _UNREACHABLE_NEO4J, + f"{_ENV_PREFIX}LOG_PATH": log_path, + f"{_ENV_PREFIX}BLOB_PATH": str(tmp_path / "blobs"), + f"{_ENV_PREFIX}QUEUES_PATH": str(tmp_path / "queues"), + f"{_ENV_PREFIX}API_KEYS_STORE_PATH": str(tmp_path / "api-keys.json"), + f"{_ENV_PREFIX}ENTRA_IDENTITIES_STORE_PATH": str( + tmp_path / "entra-identities.json" + ), + # Boot wide-open so no auth misconfig can mask the boot outcome; + # /status is auth-exempt anyway, but this removes a variable. + f"{_ENV_PREFIX}ALLOW_UNAUTHENTICATED": "true", + } + ) + return env + + +def _spawn_server(env: dict[str, str], repo_root: Path) -> subprocess.Popen[bytes]: + """Launch the REAL server entrypoint (gunicorn) as a subprocess. + + ``main()`` -> ``run()`` -> gunicorn + UvicornWorker is the exact deploy + path; the lifespan runs inside the worker, so this reproduces the real + boot sequence a systemd/ACA restart would exercise. + """ + return subprocess.Popen( + [sys.executable, "-c", "from context_intelligence_server.main import main; main()"], + cwd=str(repo_root), + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + # New session so we can signal the whole gunicorn process group on teardown. + start_new_session=True, + ) + + +def _terminate(proc: subprocess.Popen[bytes]) -> bytes: + """Terminate the process group and return captured output (best-effort).""" + if proc.poll() is None: + with contextlib.suppress(ProcessLookupError, OSError): + os.killpg(os.getpgid(proc.pid), signal.SIGTERM) + try: + proc.wait(timeout=8) + except subprocess.TimeoutExpired: + with contextlib.suppress(ProcessLookupError, OSError): + os.killpg(os.getpgid(proc.pid), signal.SIGKILL) + with contextlib.suppress(subprocess.TimeoutExpired): + proc.wait(timeout=5) + try: + out = proc.stdout.read() if proc.stdout else b"" + except Exception: # noqa: BLE001 - teardown diagnostics only + out = b"" + return out or b"" + + +def _poll_status(port: int, proc: subprocess.Popen[bytes]) -> dict[str, Any]: + """Poll GET /status until 200 or the boot deadline; assert liveness. + + Fails loudly (with captured subprocess output) if the process dies or + never serves -- that is the crash-loop this whole file guards against. + """ + url = f"http://127.0.0.1:{port}/status" + deadline = time.monotonic() + _BOOT_DEADLINE_S + last_err: str = "no attempt made" + while time.monotonic() < deadline: + if proc.poll() is not None: + out = _read_available(proc) + raise AssertionError( + "Server process EXITED during boot (crash-loop!) with code " + f"{proc.returncode}. Captured output:\n{out.decode(errors='replace')}" + ) + try: + with urllib.request.urlopen(url, timeout=2.0) as resp: + if resp.status == 200: + return json.loads(resp.read().decode()) + last_err = f"status {resp.status}" + except (urllib.error.URLError, ConnectionError, OSError) as exc: + last_err = str(exc) + time.sleep(0.5) + + out = _read_available(proc) + raise AssertionError( + f"Server did not serve GET /status within {_BOOT_DEADLINE_S}s " + f"(last error: {last_err}). Process alive={proc.poll() is None}. " + f"Captured output:\n{out.decode(errors='replace')}" + ) + + +def _read_available(proc: subprocess.Popen[bytes]) -> bytes: + """Best-effort read of whatever the process has emitted so far.""" + # The process may still be running; do a non-blocking-ish drain by + # terminating first if needed is the caller's job. Here we only read if + # the stream is already at EOF (process exited); otherwise return empty + # to avoid blocking. + if proc.poll() is None: + return b"" + try: + return proc.stdout.read() if proc.stdout else b"" + except Exception: # noqa: BLE001 + return b"" + + +@pytest.mark.timeout(60) +def test_boot_serves_status_against_unreachable_neo4j(tmp_path: Path) -> None: + """The REAL server process boots and serves /status against an + UNREACHABLE Neo4j -- schema_health is unknown/degraded, NEVER healthy, + and the process stays alive (no crash-loop).""" + repo_root = Path(__file__).resolve().parent.parent + port = _free_port() + log_path = str(tmp_path / "logs" / "server.jsonl") # writable tmp path + env = _base_env(tmp_path, port, log_path=log_path) + + proc = _spawn_server(env, repo_root) + try: + data = _poll_status(port, proc) + assert data["schema_health"] in {"unknown", "degraded"}, ( + f"schema_health must be unknown/degraded against an unreachable " + f"Neo4j, never healthy. Got: {data.get('schema_health')!r}" + ) + # Still alive after serving -- not a one-shot that then dies. + assert proc.poll() is None, "Server exited right after serving /status." + finally: + _terminate(proc) + + +@pytest.mark.timeout(60) +def test_boot_serves_status_with_unwritable_log_path(tmp_path: Path) -> None: + """The REAL server process boots and serves /status even when the + configured log_path is UNWRITABLE (its parent cannot be created). + + This is the EXACT real-process failure a live boot test caught: + setup_logging() used to mkdir the log dir BEFORE the B1 boundary, and a + PermissionError there sank the worker with no Neo4j involvement. We force + the mkdir to fail deterministically -- even when the test runs as root, + where filesystem permission bits are bypassed -- by making the log path's + parent a REGULAR FILE, so mkdir(parents=True) raises NotADirectoryError. + setup_logging must fall back to console-only and boot must still serve. + """ + repo_root = Path(__file__).resolve().parent.parent + port = _free_port() + + # Create a regular file, then point log_path UNDER it. mkdir(parents=True) + # on ".../blocker_file/logs" fails with NotADirectoryError (an OSError) + # regardless of uid -- a root-proof way to force the log dir unwritable. + blocker = tmp_path / "blocker_file" + blocker.write_text("not a directory", encoding="utf-8") + log_path = str(blocker / "logs" / "server.jsonl") + env = _base_env(tmp_path, port, log_path=log_path) + + proc = _spawn_server(env, repo_root) + try: + data = _poll_status(port, proc) + assert data["schema_health"] in {"unknown", "degraded"}, ( + f"schema_health must be unknown/degraded (unreachable Neo4j + " + f"unwritable log). Got: {data.get('schema_health')!r}" + ) + assert proc.poll() is None, "Server exited right after serving /status." + finally: + _terminate(proc) diff --git a/tests/test_docs_entrypoint.py b/tests/test_docs_entrypoint.py new file mode 100644 index 00000000..fb804f6c --- /dev/null +++ b/tests/test_docs_entrypoint.py @@ -0,0 +1,66 @@ +"""WS-3a sec 3a-5 tripwire: no shipped doc instructs `uvicorn ...main:app`. + +`main:app` is the bare FastAPI app -- no BearerTokenMiddleware, so /admin/* +(including /admin/maintenance) is UNAUTHENTICATED under that command. The +correct entrypoint for a real run is `main:asgi_app`. This test asserts the +*command shape* (`uvicorn ...main:app`), not the bare token `main:app` -- +`docs/auth-troubleshooting-and-upgrades.md` legitimately mentions `main:app` +in prose *about* this exact bug, and must not trip the tripwire. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +_REPO_ROOT = Path(__file__).resolve().parent.parent + +# Matches a uvicorn invocation targeting the un-middlewared bare app, e.g.: +# uvicorn context_intelligence_server.main:app --host ... +# uv run uvicorn context_intelligence_server.main:app --reload +# but NOT `context_intelligence_server.main:asgi_app` (asgi_app doesn't end +# in a boundary right after `:app`). +_BARE_APP_TARGET_RE = re.compile(r"uvicorn\s+context_intelligence_server\.main:app\b") + + +def _shipped_docs() -> list[Path]: + """README.md plus every markdown file under docs/ (product docs only).""" + paths = [_REPO_ROOT / "README.md"] + docs_dir = _REPO_ROOT / "docs" + if docs_dir.is_dir(): + paths.extend(sorted(docs_dir.rglob("*.md"))) + return [p for p in paths if p.is_file()] + + +def test_no_shipped_doc_instructs_bare_main_app() -> None: + """No README.md/docs/*.md tells an operator to run the un-middlewared app. + + docs/auth-troubleshooting-and-upgrades.md is allowed to keep discussing + the bug in prose (it does not contain an actual `uvicorn ...main:app` + command), so this test does not special-case any file -- it just checks + for the dangerous COMMAND SHAPE everywhere. + """ + offenders: dict[str, list[str]] = {} + for path in _shipped_docs(): + text = path.read_text(encoding="utf-8") + matches = _BARE_APP_TARGET_RE.findall(text) + if matches: + offenders[str(path.relative_to(_REPO_ROOT))] = matches + + assert not offenders, ( + "Found a documented `uvicorn ...main:app` command (the bare, " + "un-middlewared app -- /admin/* is UNAUTHENTICATED under it). Use " + "`main:asgi_app` instead. Offending files: " + f"{offenders!r}" + ) + + +def test_auth_troubleshooting_doc_still_mentions_the_bug_in_prose() -> None: + """Sanity check for the test above: the discussion doc still exists and + still names `main:app` in prose (proving the regex correctly did NOT + flag it, rather than the file having been silently deleted/rewritten).""" + doc = _REPO_ROOT / "docs" / "auth-troubleshooting-and-upgrades.md" + assert doc.is_file() + text = doc.read_text(encoding="utf-8") + assert "main:app" in text + assert not _BARE_APP_TARGET_RE.search(text) diff --git a/tests/test_main.py b/tests/test_main.py index 302fa1bf..f62f40d9 100644 --- a/tests/test_main.py +++ b/tests/test_main.py @@ -6,7 +6,7 @@ import logging from pathlib import Path from collections.abc import AsyncGenerator -from typing import Any +from typing import Any, Self from unittest.mock import AsyncMock, MagicMock, patch import httpx @@ -1037,7 +1037,7 @@ async def test_crash_recovery_topup_drains_deferred_tail_across_passes( # stops reporting them (exactly what a real drainer does on completion). for sid in sids[:2]: batch = await qm.read_batch(sid, max_items=10) - await qm.commit(sid, batch.end_offset) + await qm.commit(sid, batch.end_offset, None) # Pass 2: the previously-DEFERRED tail is now dispatched -- not stranded. spawned.clear() @@ -1135,25 +1135,26 @@ async def test_lifespan_no_sweep_when_interval_zero( # --------------------------------------------------------------------------- -# Cold start FAILS LOUD on schema/data corruption that requires `doctor -# --fix` (design decision, reversing the lifespan half of f4d8bab): an -# un-migrated graph -- duplicate legacy nodes (caught by the :Node -# constraint via fail_on_data_conflict=True) OR nodes lacking the :Node -# label altogether (caught by the O(1) count_untagged_nodes guard) -- must -# refuse to boot. Nothing has been written yet at cold start, so refusing to -# boot loses no data. The migration itself still lives ONLY in `doctor -# --fix` (run_repair); the flush path still self-heals (see -# tests/neo4j/test_node_identity_migration.py). A connectivity/probe -# failure (graph unreachable) is NOT treated as "confirmed un-migrated" and -# must not crash boot. +# Deploy-safe boot (council amendment, 2026-08-12; see +# docs/plans/2026-08-12-deploy-safe-boot-spec.md in the workspace root): +# a deploy (restart) MUST NEVER crash-loop on graph migration/reachability +# state. Cold start now calls ensure_neo4j_schema with the SAME +# fail_on_data_conflict=False default the mid-flight flush path always used, +# and the untagged-node probe no longer raises on a positive count -- both +# feed a tri-state schema_health signal ("healthy" / "degraded" / "unknown") +# surfaced on GET /status (never GET /version -- see B7). The B1 boundary +# additionally catches ANY exception from the startup sequence (not just +# these two named sites), so boot proceeds regardless of which of the ~11 +# startup raise sites fails. Only run_repair / `doctor --fix` still opts +# into fail_on_data_conflict=True -- see +# tests/neo4j/test_node_identity_migration.py for that (unchanged) contract. # --------------------------------------------------------------------------- -async def test_lifespan_calls_ensure_schema_with_fail_on_data_conflict() -> None: - """Cold start must call ensure_neo4j_schema with fail_on_data_conflict=True - -- boot refuses to proceed on a genuine :Node constraint data conflict - (duplicate legacy nodes), mirroring run_repair's contract. Safe at cold - start (nothing flushed yet); the flush path keeps the opposite default.""" +async def test_lifespan_calls_ensure_schema_with_fail_on_data_conflict_false() -> None: + """Cold start must call ensure_neo4j_schema with fail_on_data_conflict=False + -- boot never fails closed on a genuine :Node constraint data conflict. + That fail-closed contract now belongs ONLY to run_repair/`doctor --fix`.""" mock_driver = _patched_lifespan_deps() mock_ensure_schema = AsyncMock(return_value=True) with ( @@ -1176,17 +1177,23 @@ async def test_lifespan_calls_ensure_schema_with_fail_on_data_conflict() -> None mock_ensure_schema.assert_awaited_once() _args, kwargs = mock_ensure_schema.await_args - assert kwargs.get("fail_on_data_conflict") is True, ( - "lifespan must opt into fail_on_data_conflict=True -- cold start " - "fails loud on a genuine data conflict (that contract now applies " - "at boot too, not just to run_repair / `doctor --fix`)." + assert kwargs.get("fail_on_data_conflict") is False, ( + "lifespan must call ensure_neo4j_schema with fail_on_data_conflict=False " + "-- deploy-safe boot never fails closed on graph data state (only " + "run_repair/`doctor --fix` opts into True)." ) + assert main_module.app.state.schema_health == "healthy" -async def test_lifespan_raises_on_ensure_schema_data_conflict() -> None: - """When ensure_neo4j_schema itself raises (a genuine :Node constraint - data conflict under fail_on_data_conflict=True), lifespan must propagate - the RuntimeError -- boot refuses to start.""" +async def test_lifespan_does_not_raise_when_ensure_schema_itself_raises( + caplog: pytest.LogCaptureFixture, +) -> None: + """B1: a non-data-conflict exception escaping ensure_neo4j_schema itself + (e.g. a Neo4jError re-raised by _create_index, a TransientError during + the ordinary ACA cold-start reachability race, or a rejected credential) + is caught by the single lifespan try/except boundary -- boot proceeds + and schema_health reports "unknown" (never coerced to "healthy"), never + propagating a RuntimeError out of lifespan.""" mock_driver = _patched_lifespan_deps() with ( patch("context_intelligence_server.main.setup_logging"), @@ -1197,21 +1204,27 @@ async def test_lifespan_raises_on_ensure_schema_data_conflict() -> None: patch( "context_intelligence_server.main.ensure_neo4j_schema", new=AsyncMock( - side_effect=RuntimeError( - "Neo4j :Node constraint data conflict -- run doctor --fix" - ) + side_effect=RuntimeError("Neo4j unreachable (TransientError)") ), ), - pytest.raises(RuntimeError, match="doctor --fix"), + caplog.at_level(logging.ERROR), ): - async with lifespan(main_module.app): + async with lifespan(main_module.app): # MUST NOT raise pass + assert main_module.app.state.schema_health == "unknown" + assert "startup sequence failed" in ( + main_module.app.state.schema_degraded_reason or "" + ) + assert any( + "startup_degraded" in record.getMessage() for record in caplog.records + ), "Expected a loud ERROR-level startup_degraded log." -async def test_lifespan_raises_on_untagged_nodes() -> None: - """On an un-migrated graph (untagged :Node count > 0), startup MUST - raise a RuntimeError naming `doctor --fix` -- boot refuses to start - rather than silently risking write-path duplication.""" + +async def test_lifespan_does_not_raise_on_untagged_nodes() -> None: + """On an un-migrated graph (untagged :Node count > 0), startup no longer + raises -- boot proceeds and schema_health reports "degraded" with the + untagged count surfaced, rather than refusing to serve.""" mock_driver = _patched_lifespan_deps() with ( patch("context_intelligence_server.main.setup_logging"), @@ -1227,20 +1240,21 @@ async def test_lifespan_raises_on_untagged_nodes() -> None: "context_intelligence_server.main.count_untagged_nodes", new=AsyncMock(return_value=42), ), - pytest.raises(RuntimeError, match="doctor --fix") as exc_info, ): - async with lifespan(main_module.app): + async with lifespan(main_module.app): # MUST NOT raise pass - assert "42" in str(exc_info.value), ( - f"Expected the untagged count in the error message, got: {exc_info.value}" + assert main_module.app.state.schema_health == "degraded" + assert main_module.app.state.schema_untagged_nodes == 42 + assert "42" in (main_module.app.state.schema_degraded_reason or ""), ( + f"Expected the untagged count in degraded_reason, got: " + f"{main_module.app.state.schema_degraded_reason!r}" ) async def test_lifespan_does_not_raise_on_clean_graph() -> None: """On a fully-migrated graph (untagged count == 0, no constraint - conflict), startup does NOT raise -- the fail-loud guards are silent - when there is nothing to report.""" + conflict), startup does NOT raise and schema_health reports "healthy".""" mock_driver = _patched_lifespan_deps() with ( patch("context_intelligence_server.main.setup_logging"), @@ -1261,15 +1275,19 @@ async def test_lifespan_does_not_raise_on_clean_graph() -> None: pass mock_count.assert_awaited_once() + assert main_module.app.state.schema_health == "healthy" + assert main_module.app.state.schema_degraded_reason is None -async def test_lifespan_does_not_raise_when_health_check_itself_fails( +async def test_lifespan_reports_unknown_when_health_check_itself_fails( caplog: pytest.LogCaptureFixture, ) -> None: """A health-check probe failure (e.g. count_untagged_nodes raising due to - a transient connectivity blip) must NOT be treated as a confirmed - un-migrated graph -- it is logged at DEBUG and swallowed, and boot - proceeds. Connectivity failure != confirmed data corruption.""" + a transient connectivity blip, or Neo4j being unreachable at boot) must + NOT be coerced to "healthy" (B3) -- schema_health reports "unknown" and + boot proceeds regardless. This is the exact ACA cold-start race the + deploy-safe boot fix targets: unreachable-then-reachable must never + crash-loop.""" mock_driver = _patched_lifespan_deps() with ( patch("context_intelligence_server.main.setup_logging"), @@ -1285,11 +1303,431 @@ async def test_lifespan_does_not_raise_when_health_check_itself_fails( "context_intelligence_server.main.count_untagged_nodes", new=AsyncMock(side_effect=RuntimeError("transient connectivity blip")), ), - caplog.at_level(logging.DEBUG), + caplog.at_level(logging.WARNING), ): async with lifespan(main_module.app): # MUST NOT raise pass + assert main_module.app.state.schema_health == "unknown" + assert main_module.app.state.schema_untagged_nodes is None + assert any( + "migration-health probe failed" in record.getMessage() + for record in caplog.records + ), "Expected a WARNING logging the probe failure." + + +async def test_lifespan_recovery_quarantines_one_bad_session_and_continues( + monkeypatch: pytest.MonkeyPatch, + caplog: pytest.LogCaptureFixture, +) -> None: + """B6: a single session whose recovery raises (a corrupt per-session + .offset/dead-letter) is quarantined -- logged and skipped -- while other + recovered sessions still respawn and boot still completes.""" + sid_bad = "sess-corrupt" + sid_good = "sess-good" + qm = registry.queue_manager + # Both lines parse fine at the JSON level (a malformed/torn line is + # already handled gracefully by _recover_one_session's own internal + # try/except -- see test_lifespan_skips_recovery_for_empty_workspace). + # B6's NEW defensive guard covers failures that surface only once + # recovery actually tries to respawn the drainer for that session (e.g. + # a corrupt on-disk queue file the registry discovers at get_or_create + # time) -- simulated here via a flaky get_or_create. + bad_body = json.dumps( + { + "event": "tool_use", + "workspace": "/bad-ws", + "data": {"session_id": sid_bad}, + } + ).encode("utf-8") + good_body = json.dumps( + { + "event": "tool_use", + "workspace": "/good-ws", + "data": {"session_id": sid_good}, + } + ).encode("utf-8") + await qm.append(sid_bad, bad_body) + await qm.append(sid_good, good_body) + + spawned: list[tuple] = [] + + def _flaky_get_or_create(sid: str, workspace: str, **kw: object) -> None: + if sid == sid_bad: + raise RuntimeError("simulated corrupt per-session recovery failure") + spawned.append((sid, workspace)) + + monkeypatch.setattr(registry, "get_or_create", _flaky_get_or_create) + + mock_driver = _patched_lifespan_deps() + with ( + patch("context_intelligence_server.main.setup_logging"), + patch( + "context_intelligence_server.main.AsyncGraphDatabase.driver", + return_value=mock_driver, + ), + patch( + "context_intelligence_server.main.ensure_neo4j_schema", + new=AsyncMock(return_value=True), + ), + patch( + "context_intelligence_server.main.count_untagged_nodes", + new=AsyncMock(return_value=0), + ), + caplog.at_level(logging.ERROR), + ): + async with lifespan(main_module.app): # MUST NOT raise + pass + + assert (sid_good, "/good-ws") in spawned, ( + "The healthy session must still be recovered despite the other " + "session's recovery failing." + ) + assert any( + "recovery_session_quarantined" in record.getMessage() + and sid_bad in record.getMessage() + for record in caplog.records + ), "Expected the corrupt session to be logged as quarantined." + # Boot still reaches a healthy schema state -- the quarantine did not + # propagate into (or get masked by) the B1 boundary. + assert main_module.app.state.schema_health == "healthy" + + +async def test_status_exposes_schema_health_fields( + client: httpx.AsyncClient, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """GET /status surfaces schema_health, untagged_nodes, schema_checked_at, + and degraded_reason. + + WS-3a DE-LATCHES schema_health/schema_checked_at/degraded_reason AND + (latch-fix) untagged_nodes: none of these are read verbatim from the + app.state boot snapshot anymore -- schema_health, degraded_reason and + untagged_nodes are all derived live from the MaintenanceCoordinator's + TTL-cached probe (the same probe/reason the 503 body reuses), and + schema_checked_at is the live-probe timestamp rather than the boot-time + one. This means an out-of-band repair self-clears every one of these + within one probe TTL, with no restart. + + This pins the fix for the stale-signal bug: `degraded_reason` AND + `untagged_nodes` used to be read verbatim from the app.state boot + snapshot, so they kept asserting a stale condition even after a live + repair de-latched `mode`/`schema_health`. Here the boot-time snapshot is + deliberately set to DIFFERENT values than the live coordinator's -- the + assertions below only pass if the response is sourced from the live + values, not the stale boot snapshot. + """ + from context_intelligence_server.maintenance import MaintenanceStatus, OpRecord + + # Stale boot-time snapshot -- deliberately DIFFERENT from the live + # coordinator values below, so the test fails if /status ever regresses + # back to reading the boot snapshot for any of these fields. + main_module.app.state.schema_untagged_nodes = 999 + main_module.app.state.schema_degraded_reason = "STALE boot-time reason" + monkeypatch.setattr( + main_module.coordinator, + "status", + AsyncMock( + return_value=MaintenanceStatus( + mode="degraded", + constraint_present=True, + reason="3 node(s) lacking the :Node label", + started_at=None, + elapsed_seconds=None, + op=OpRecord( + state="unknown", + run_id=None, + started_at=None, + completed_at=None, + records_affected=None, + error=None, + ), + untagged_nodes=3, + ) + ), + ) + try: + response = await client.get("/status") + data = response.json() + assert data["schema_health"] == "degraded" + assert data["untagged_nodes"] == 3 # LIVE coordinator value, not boot 999 + assert data["schema_checked_at"] is not None # live probe timestamp now + # LIVE coordinator reason, NOT the stale boot-time snapshot. + assert data["degraded_reason"] == "3 node(s) lacking the :Node label" + finally: + # Reset so this test doesn't leak state into siblings sharing the + # module-level app singleton. + main_module.app.state.schema_untagged_nodes = None + main_module.app.state.schema_degraded_reason = None + + +# --------------------------------------------------------------------------- +# W-4: GET /status advisory graph_schema_version / schema_version_current +# --------------------------------------------------------------------------- + + +class _SchemaMetaResult: + """Async-iterable result double yielding a fixed list of row dicts. + + Mirrors ``_RowsResult`` in ``tests/test_neo4j_store.py`` -- exercises the + ``async for record in result`` path ``read_graph_schema_version`` uses + (not ``.single()``), so this double alone is sufficient. + """ + + def __init__(self, rows: list[dict[str, Any]]) -> None: + self._rows = rows + + def __aiter__(self) -> Any: + return self._agen() + + async def _agen(self) -> Any: + for row in self._rows: + yield row + + +class _SchemaMetaDriverStub: + """Driver double for ``read_graph_schema_version``: doubles as its own + session context manager and returns a canned ``:SchemaMeta`` row (or no + rows at all, simulating an absent/uninitialized singleton). + """ + + def __init__(self, schema_version: int | None) -> None: + self._schema_version = schema_version + + def session(self, database: str = "neo4j") -> Self: + return self + + async def __aenter__(self) -> Self: + return self + + async def __aexit__(self, *args: object) -> None: + return None + + async def run(self, statement: str) -> _SchemaMetaResult: + rows: list[dict[str, Any]] = ( + [] + if self._schema_version is None + else [{"schema_version": self._schema_version}] + ) + return _SchemaMetaResult(rows) + + +async def test_status_exposes_graph_schema_version_when_current( + client: httpx.AsyncClient, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """GET /status surfaces graph_schema_version sourced from the STORED + :SchemaMeta singleton, and schema_version_current: true when it matches + the server's compiled-in SCHEMA_VERSION (advisory only -- no gating).""" + from context_intelligence_server.status import SCHEMA_VERSION + + monkeypatch.setattr( + main_module.app.state, + "neo4j_driver", + _SchemaMetaDriverStub(SCHEMA_VERSION), + raising=False, + ) + + response = await client.get("/status") + assert response.status_code == 200 + data = response.json() + assert data["graph_schema_version"] == SCHEMA_VERSION + assert data["schema_version_current"] is True + + +async def test_status_exposes_graph_schema_version_mismatch_detectable( + client: httpx.AsyncClient, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """W-4: a stored schema_version DIFFERENT from the compiled-in + SCHEMA_VERSION is reflected verbatim on /status (detectable drift) and + schema_version_current flips to false -- still purely advisory, no + behavior change results from the mismatch.""" + from context_intelligence_server.status import SCHEMA_VERSION + + stored_version = SCHEMA_VERSION - 1 if SCHEMA_VERSION > 0 else SCHEMA_VERSION + 1 + assert stored_version != SCHEMA_VERSION # sanity: the whole point of this test + monkeypatch.setattr( + main_module.app.state, + "neo4j_driver", + _SchemaMetaDriverStub(stored_version), + raising=False, + ) + + response = await client.get("/status") + assert response.status_code == 200 + data = response.json() + assert data["graph_schema_version"] == stored_version + assert data["schema_version_current"] is False + + +async def test_status_graph_schema_version_none_when_singleton_absent( + client: httpx.AsyncClient, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Bootstrap case: no :SchemaMeta singleton yet (server never completed + startup against this graph) -> graph_schema_version and + schema_version_current are both None, never an error / 500.""" + monkeypatch.setattr( + main_module.app.state, + "neo4j_driver", + _SchemaMetaDriverStub(None), + raising=False, + ) + + response = await client.get("/status") + assert response.status_code == 200 + data = response.json() + assert data["graph_schema_version"] is None + assert data["schema_version_current"] is None + + +async def test_status_graph_schema_version_none_when_no_driver( + client: httpx.AsyncClient, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Defensive: /status must never 500 when neo4j_driver is unset.""" + if hasattr(main_module.app.state, "neo4j_driver"): + monkeypatch.delattr(main_module.app.state, "neo4j_driver", raising=False) + + response = await client.get("/status") + assert response.status_code == 200 + data = response.json() + assert data["graph_schema_version"] is None + assert data["schema_version_current"] is None + + +async def test_status_degraded_reason_self_clears_after_live_repair_no_restart( + client: httpx.AsyncClient, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """THE STALE-SIGNAL BUG, DIRECTLY: after an in-server repair (e.g. + `POST /admin/maintenance` recreating the :Node uniqueness constraint), + `degraded_reason` must update to reflect the new live state -- WITHOUT a + restart -- exactly like `mode`/`schema_health` already do (WS-3a). + + Before the fix, `degraded_reason` was read verbatim from the app.state + boot snapshot, which is only ever written once during `lifespan`. A + live repair flips the coordinator's probe (and thus `mode`), but the + boot snapshot never changes for the life of the process -- so + `degraded_reason` kept asserting a constraint-absent condition that was + now false. Here the SAME running app (no restart, no re-entering + `lifespan`) observes the coordinator's probe result change between two + successive `GET /status` calls and the reason string tracks it. + """ + from context_intelligence_server.maintenance import MaintenanceStatus, OpRecord + + def _status( + *, constraint_present: bool | None, reason: str | None, mode: str + ) -> MaintenanceStatus: + return MaintenanceStatus( + mode=mode, # type: ignore[arg-type] + constraint_present=constraint_present, + reason=reason, + started_at=None, + elapsed_seconds=None, + op=OpRecord( + state="unknown", + run_id=None, + started_at=None, + completed_at=None, + records_affected=None, + error=None, + ), + ) + + # Boot-time snapshot: set ONCE, never touched again in this test -- + # simulating the real lifespan-populated app.state that a stale + # implementation would keep reading forever. + main_module.app.state.schema_degraded_reason = ( + ":Node uniqueness constraint absent (data conflict)" + ) + try: + # --- Before repair: constraint absent, live-degraded. --- + monkeypatch.setattr( + main_module.coordinator, + "status", + AsyncMock( + return_value=_status( + constraint_present=False, + reason=( + ":Node uniqueness constraint absent -- migration required" + ), + mode="maintenance", + ) + ), + ) + before = (await client.get("/status")).json() + assert before["degraded_reason"] == ( + ":Node uniqueness constraint absent -- migration required" + ) + + # --- Repair happens out-of-band (no restart of this process): the + # constraint is recreated, the coordinator's live probe now reports + # present/healthy. Re-point the SAME coordinator's `status` method -- + # nothing about app.state.schema_degraded_reason changes. + monkeypatch.setattr( + main_module.coordinator, + "status", + AsyncMock( + return_value=_status( + constraint_present=True, + reason=None, + mode="healthy", + ) + ), + ) + after = (await client.get("/status")).json() + + # THE FIX: degraded_reason clears to null, tracking the live probe -- + # it does NOT still say "constraint absent" (the stale boot value). + assert after["degraded_reason"] is None, ( + f"degraded_reason must clear once the live probe reports the " + f"constraint present again; got stale value: " + f"{after['degraded_reason']!r}" + ) + # Non-vacuity: prove the two calls actually observed different + # coordinator states (otherwise this test would trivially pass). + assert before["degraded_reason"] != after["degraded_reason"] + # And prove it never degenerated into the OLD stale-boot-snapshot + # behavior, which would have returned this sentence unchanged on + # both calls. + assert after["degraded_reason"] != ( + main_module.app.state.schema_degraded_reason + ) + finally: + main_module.app.state.schema_degraded_reason = None + + +async def test_status_schema_health_defaults_to_unknown_when_unset( + client: httpx.AsyncClient, +) -> None: + """Before lifespan has run (or if app.state was never populated), GET + /status must report schema_health="unknown", never a false "healthy" + (B3: a probe that hasn't run yet is not the same as a clean graph). + + degraded_reason is now sourced LIVE from the coordinator (see the + stale-signal fix), so with no driver bound the coordinator's own probe + reports "unknown" WITH a reason explaining why (unlike the old + boot-snapshot-only field, which could be a bare None here purely because + the app.state attributes were deleted, never reflecting the true + tri-state semantics).""" + for attr in ( + "schema_health", + "schema_untagged_nodes", + "schema_checked_at", + "schema_degraded_reason", + ): + if hasattr(main_module.app.state, attr): + delattr(main_module.app.state, attr) + + response = await client.get("/status") + data = response.json() + assert data["schema_health"] == "unknown" + assert data["untagged_nodes"] is None + assert data["degraded_reason"] is not None + assert "could not determine" in data["degraded_reason"] + async def test_registry_exposed_on_app_state() -> None: """The module registry singleton is exposed on app.state for routers. @@ -1316,7 +1754,7 @@ async def test_lifespan_seeds_counters_from_disk(tmp_path: Path) -> None: await seed_qm.append(sid, line1) await seed_qm.append(sid, line2) committed = len(line1) + 1 # +1 for the appended trailing newline - await seed_qm.commit(sid, committed) + await seed_qm.commit(sid, committed, None) # Fresh registry reusing the same on-disk queue dir. reg = SessionRegistry() @@ -1783,6 +2221,73 @@ async def _fake_append(worker_key: str, raw: bytes) -> None: assert body_obj["created_by"] is None +async def test_post_events_lifts_working_dir_into_data( + client: httpx.AsyncClient, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """I1: a top-level working_dir envelope field is lifted into data.working_dir.""" + monkeypatch.setattr( + main_module.registry, "get_or_create", lambda *args, **kwargs: MagicMock() + ) + captured: list[bytes] = [] + + async def _fake_append(worker_key: str, raw: bytes) -> None: + captured.append(raw) + + monkeypatch.setattr(main_module.registry.queue_manager, "append", _fake_append) + + response = await client.post( + "/events", + json={ + "event": "session:start", + "workspace": "/ws", + "working_dir": "/home/user/my-project", + "data": { + "session_id": "s4", + "timestamp": "2026-06-16T20:17:11.604690+00:00", + }, + }, + ) + + assert response.status_code == 202 + assert len(captured) == 1 + body_obj = json.loads(captured[0]) + assert body_obj["data"]["working_dir"] == "/home/user/my-project" + + +async def test_post_events_absent_working_dir_leaves_data_unset( + client: httpx.AsyncClient, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """I1: omitting working_dir must not add the key to data at all (forward-only).""" + monkeypatch.setattr( + main_module.registry, "get_or_create", lambda *args, **kwargs: MagicMock() + ) + captured: list[bytes] = [] + + async def _fake_append(worker_key: str, raw: bytes) -> None: + captured.append(raw) + + monkeypatch.setattr(main_module.registry.queue_manager, "append", _fake_append) + + response = await client.post( + "/events", + json={ + "event": "session:start", + "workspace": "/ws", + "data": { + "session_id": "s5", + "timestamp": "2026-06-16T20:17:11.604690+00:00", + }, + }, + ) + + assert response.status_code == 202 + assert len(captured) == 1 + body_obj = json.loads(captured[0]) + assert "working_dir" not in body_obj["data"] + + async def test_crash_recovery_passes_created_by_to_get_or_create( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -2061,3 +2566,234 @@ async def test_admin_schema_not_disclosed_in_openapi(self) -> None: f"/admin/* must not appear in the unauthenticated OpenAPI schema; " f"found: {admin_paths}" ) + + +# --------------------------------------------------------------------------- +# WS-3a: maintenance-mode gate + /status additive fields (spec sec 7a) +# --------------------------------------------------------------------------- + + +def _maintenance_mode_status() -> Any: + """Build a MaintenanceStatus reporting mode=="maintenance", for tests + that force the gate closed without a real Neo4j driver bound.""" + from context_intelligence_server.maintenance import MaintenanceStatus, OpRecord + + return MaintenanceStatus( + mode="maintenance", + constraint_present=False, + reason=":Node uniqueness constraint absent -- migration required", + started_at="2026-08-13T00:00:00+00:00", + elapsed_seconds=1.0, + op=OpRecord( + state="unknown", + run_id=None, + started_at=None, + completed_at=None, + records_affected=None, + error=None, + ), + ) + + +class TestMaintenanceGateHttp: + """A8-A11 (WS-3a spec sec 7a).""" + + async def test_allow_listed_paths_bypass_the_gate( + self, client: httpx.AsyncClient, monkeypatch: pytest.MonkeyPatch + ) -> None: + """A9: /status, /version, /admin/maintenance are NEVER 503'd by the + maintenance gate, even while mode == "maintenance".""" + monkeypatch.setattr( + main_module.coordinator, + "status", + AsyncMock(return_value=_maintenance_mode_status()), + ) + status_resp = await client.get("/status") + version_resp = await client.get("/version") + # /admin/maintenance IS on the allow-list, so the maintenance GATE must + # never intercept it. The route now EXISTS (WS-3c) and may legitimately + # return 503 for an UNRELATED reason (no admin_api_key configured in this + # test app -> "admin API not enabled"), so assert specifically that this + # is NOT the gate's structured maintenance-503, rather than a blanket + # != 503. The gate's 503 carries a Retry-After header + a + # {"status": "maintenance"} body; the admin-auth 503 carries neither. + admin_maint_resp = await client.post("/admin/maintenance") + + assert status_resp.status_code != 503 + assert version_resp.status_code != 503 + gate_intercepted = ( + admin_maint_resp.status_code == 503 + and admin_maint_resp.headers.get("retry-after") is not None + ) + assert not gate_intercepted, ( + "maintenance gate must not intercept the allow-listed /admin/maintenance" + ) + + async def test_non_allow_listed_path_is_503d_while_gated( + self, client: httpx.AsyncClient, monkeypatch: pytest.MonkeyPatch + ) -> None: + """A8 (partial -- unauthenticated path): a non-allow-listed route + returns the structured 503 (status/reason/retry_after/schema_health/ + maintenance_started_at) with a Retry-After header, while gated.""" + monkeypatch.setattr( + main_module.coordinator, + "status", + AsyncMock(return_value=_maintenance_mode_status()), + ) + resp = await client.get("/blobs/does-not-matter") + assert resp.status_code == 503 + assert "Retry-After" in resp.headers + body = resp.json() + assert body["status"] == "maintenance" + assert body["reason"] == ( + ":Node uniqueness constraint absent -- migration required" + ) + assert body["retry_after"] == int(resp.headers["Retry-After"]) + assert body["schema_health"] == "degraded" + assert body["maintenance_started_at"] == "2026-08-13T00:00:00+00:00" + + async def test_events_cypher_and_dead_letter_replay_503_while_gated( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """A8: all three drainer-spawn-adjacent paths (POST /events, POST + /cypher, POST /queues/dead-letter/{k}/replay) are gated, through the + SAME auth-wrapped app real clients use.""" + monkeypatch.setattr( + main_module.coordinator, + "status", + AsyncMock(return_value=_maintenance_mode_status()), + ) + headers = {"Authorization": "Bearer test-secret"} + async with _auth_client() as c: + events_resp = await c.post( + "/events", + json={ + "event": "tool_use", + "workspace": "/ws", + "data": { + "session_id": "s-gated", + "timestamp": "2026-08-13T00:00:00+00:00", + }, + }, + headers=headers, + ) + cypher_resp = await c.post( + "/cypher", json={"query": "MATCH (n) RETURN n"}, headers=headers + ) + replay_resp = await c.post( + "/queues/dead-letter/some-key/replay", headers=headers + ) + + for resp in (events_resp, cypher_resp, replay_resp): + assert resp.status_code == 503 + assert "Retry-After" in resp.headers + assert resp.json()["status"] == "maintenance" + + async def test_status_exposes_maintenance_mode_fields_additively( + self, client: httpx.AsyncClient + ) -> None: + """A11: /status exposes mode/maintenance_started_at/ + maintenance_elapsed_seconds, and every pre-existing key is still + present (additive, no regression).""" + response = await client.get("/status") + assert response.status_code == 200 + data = response.json() + for key in ( + "mode", + "maintenance_started_at", + "maintenance_elapsed_seconds", + # pre-existing keys, unchanged: + "schema_health", + "untagged_nodes", + "schema_checked_at", + "degraded_reason", + "neo4j_connected", + "neo4j_query_connected", + "metrics", + "auth", + "queue_health", + ): + assert key in data, f"expected pre-existing/additive key {key!r} in /status" + assert data["mode"] in {"healthy", "maintenance", "degraded", "unknown"} + + def test_maintenance_endpoint_allow_listed_assertion_passes_today(self) -> None: + """A10 (positive case): the real MAINTENANCE_ALLOW_LIST satisfies the + startup assertion as shipped.""" + main_module._assert_maintenance_endpoint_allow_listed() + + def test_maintenance_endpoint_allow_listed_assertion_raises_if_removed( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """A10: if /admin/maintenance were removed from the allow-list, the + startup assertion raises -- proving the assertion is load-bearing, + not a no-op (non-vacuity).""" + monkeypatch.setattr( + main_module, + "MAINTENANCE_ALLOW_LIST", + frozenset({"/status", "/version"}), + ) + with pytest.raises(RuntimeError, match="MAINTENANCE_ALLOW_LIST"): + main_module._assert_maintenance_endpoint_allow_listed() + + +class TestQueueHealthSeparateFromSchemaHealth: + """A14 (W-2): a queue-recovery exception sets queue_health="degraded" and + leaves schema_health untouched (a queue fault is not a schema fault).""" + + async def test_queue_recovery_failure_degrades_queue_health_only( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + mock_driver = MagicMock() + mock_driver.close = AsyncMock() + + with ( + patch("context_intelligence_server.main.setup_logging"), + patch( + "context_intelligence_server.main.AsyncGraphDatabase.driver", + return_value=mock_driver, + ), + patch( + "context_intelligence_server.main.ensure_neo4j_schema", + new=AsyncMock(return_value=True), + ), + patch( + "context_intelligence_server.main.count_untagged_nodes", + new=AsyncMock(return_value=0), + ), + patch.object( + registry.queue_manager, + "recovery_reconcile_dead", + AsyncMock(side_effect=RuntimeError("disk corrupt")), + ), + ): + async with lifespan(main_module.app): + pass + + assert main_module.app.state.queue_health == "degraded" + # The schema fault signal is UNAFFECTED by the queue fault (the + # entire point of separating the two try/except boundaries). + assert main_module.app.state.schema_health == "healthy" + + async def test_queue_recovery_success_leaves_queue_health_healthy(self) -> None: + mock_driver = MagicMock() + mock_driver.close = AsyncMock() + + with ( + patch("context_intelligence_server.main.setup_logging"), + patch( + "context_intelligence_server.main.AsyncGraphDatabase.driver", + return_value=mock_driver, + ), + patch( + "context_intelligence_server.main.ensure_neo4j_schema", + new=AsyncMock(return_value=True), + ), + patch( + "context_intelligence_server.main.count_untagged_nodes", + new=AsyncMock(return_value=0), + ), + ): + async with lifespan(main_module.app): + pass + + assert main_module.app.state.queue_health == "healthy" diff --git a/tests/test_maintenance.py b/tests/test_maintenance.py new file mode 100644 index 00000000..60e19617 --- /dev/null +++ b/tests/test_maintenance.py @@ -0,0 +1,443 @@ +"""Tests for the MaintenanceCoordinator seam (WS-3a). + +Covers the WS-3a spec's unit test plan (sec 7a), items A1-A5, A12, A13: +probe tri-state + fail-open, TTL caching, single-flight, the latch-defect +regression (A4), the op-running/constraint-present interaction (A5), and +transition logging (A12). Allow-list/gate/status HTTP-surface tests (A8-A11) +live in test_main.py; drain-loop offset-ordering tests (A6-A7) live in +test_registry.py; the docs tripwire (A15) lives in test_docs_entrypoint.py; +W-2 (A14) lives in test_main.py. +""" + +from __future__ import annotations + +import asyncio +import logging +from typing import Any, Self + +import pytest +from context_intelligence_server.maintenance import ( + MAINTENANCE_ALLOW_LIST, + MaintenanceCoordinator, +) + +# NOTE: asyncio_mode = "auto" (pyproject.toml) runs async tests automatically +# -- no pytest.mark.asyncio needed. A blanket `pytestmark` would incorrectly +# tag the sync tests below (TestOpSeam, test_allow_list_contains_required_paths). + + +# --------------------------------------------------------------------------- +# Fake Neo4j driver -- controls the constraint-probe result and counts real +# "catalog reads" so tests can assert single-flight/caching behavior. +# --------------------------------------------------------------------------- + + +class _FakeConstraintResult: + def __init__(self, count: int) -> None: + self._count = count + self._yielded = False + + def __aiter__(self) -> _FakeConstraintResult: + return self + + async def __anext__(self) -> dict[str, int]: + if self._yielded: + raise StopAsyncIteration + self._yielded = True + return {"c": self._count} + + +class _FakeConstraintSession: + def __init__(self, driver: _FakeConstraintDriver) -> None: + self._driver = driver + + async def __aenter__(self) -> Self: + return self + + async def __aexit__(self, *exc: object) -> None: + return None + + async def run(self, cypher: str) -> _FakeConstraintResult: + self._driver.call_count += 1 + if self._driver.raise_exc is not None: + raise self._driver.raise_exc + return _FakeConstraintResult(1 if self._driver.present else 0) + + +class _FakeConstraintDriver: + """Test double: counts real probe hits, lets tests flip the result.""" + + def __init__( + self, present: bool = True, raise_exc: Exception | None = None + ) -> None: + self.present = present + self.raise_exc = raise_exc + self.call_count = 0 + + def session(self, **kwargs: Any) -> _FakeConstraintSession: + return _FakeConstraintSession(self) + + +# --------------------------------------------------------------------------- +# A1 -- tri-state probe + fail-open rule (D-E) +# --------------------------------------------------------------------------- + + +class TestProbeTriState: + async def test_constraint_absent_closes_gate(self) -> None: + coord = MaintenanceCoordinator() + coord.bind_driver(_FakeConstraintDriver(present=False)) + assert await coord.gate_closed() is True + + async def test_constraint_present_opens_gate(self) -> None: + coord = MaintenanceCoordinator() + coord.bind_driver(_FakeConstraintDriver(present=True)) + assert await coord.gate_closed() is False + + async def test_probe_raises_yields_unknown_and_gate_open(self) -> None: + coord = MaintenanceCoordinator() + coord.bind_driver( + _FakeConstraintDriver(raise_exc=RuntimeError("neo4j unreachable")) + ) + st = await coord.status() + assert st.constraint_present is None + assert st.mode == "unknown" + assert await coord.gate_closed() is False # unknown must NOT close the gate + + +# --------------------------------------------------------------------------- +# A2 -- TTL cache works +# --------------------------------------------------------------------------- + + +class TestProbeTtlCache: + async def test_two_calls_within_ttl_hit_driver_once(self) -> None: + driver = _FakeConstraintDriver(present=True) + coord = MaintenanceCoordinator() + coord.bind_driver(driver, probe_ttl_seconds=10.0) + + await coord.gate_closed() + await coord.gate_closed() + + assert driver.call_count == 1 + + async def test_call_after_ttl_expiry_hits_driver_again(self) -> None: + driver = _FakeConstraintDriver(present=True) + coord = MaintenanceCoordinator() + coord.bind_driver(driver, probe_ttl_seconds=0.05) + + await coord.gate_closed() + # First in-window call: constraint catalog read (cold) + untagged + # count (seeded warm at bind_driver, so 0 reads here) == 1 read. + hits_after_first = driver.call_count + assert hits_after_first == 1 + await asyncio.sleep(0.1) # both caches expire + await coord.gate_closed() + + # After expiry a FRESH probe runs -- the load-bearing property is that + # the cache expired and re-probed, not the exact read count (which now + # also includes the live O(1) untagged count, present-constraint path). + assert driver.call_count > hits_after_first + + +# --------------------------------------------------------------------------- +# A3 -- single-flight: N concurrent callers at cache expiry -> ONE probe +# --------------------------------------------------------------------------- + + +class TestProbeSingleFlight: + async def test_fifty_concurrent_calls_at_expiry_yield_one_catalog_read( + self, + ) -> None: + driver = _FakeConstraintDriver(present=True) + coord = MaintenanceCoordinator() + coord.bind_driver(driver, probe_ttl_seconds=0.02) + + # Prime + expire the cache once, deterministically. + await coord.gate_closed() + await asyncio.sleep(0.05) + hits_before = driver.call_count + assert hits_before == 1 + + results = await asyncio.gather(*[coord.gate_closed() for _ in range(50)]) + + # Single-flight: all 50 callers at the expiry boundary collapse to ONE + # probe window, NOT 50. That window now reads the constraint catalog + # AND the live O(1) untagged count (constraint-present path), so the + # increment is a small constant (<= 3), never proportional to the 50 + # callers -- which is the property this test guards. + new_reads = driver.call_count - hits_before + assert new_reads <= 3 + assert all(r is False for r in results) # constraint present -> gate open + + +# --------------------------------------------------------------------------- +# A4 -- THE LATCH DEFECT, DIRECTLY: probe flips False->True with NO restart +# --------------------------------------------------------------------------- + + +class TestNoRestartSelfClear: + async def test_gate_opens_within_ttl_with_no_restart(self) -> None: + """This is the exact bug this spec exists to fix: a boot-latched + schema_health never re-probed until the next restart. Here the SAME + coordinator instance (no restart) observes the constraint appear.""" + driver = _FakeConstraintDriver(present=False) + coord = MaintenanceCoordinator() + coord.bind_driver(driver, probe_ttl_seconds=0.05) + + assert await coord.gate_closed() is True # migration required + + # Out-of-band repair happens (e.g. `doctor --fix` in another + # process) -- constraint now exists. NO restart of this process. + driver.present = True + await asyncio.sleep(0.1) # let the TTL expire + + assert await coord.gate_closed() is False # self-cleared, no restart + + async def test_non_vacuous_without_ttl_expiry_gate_stays_stale(self) -> None: + """Non-vacuity proof (WS-2 precedent): if we DON'T wait out the TTL, + the cached (stale) answer is returned -- proving the test above is + actually exercising the cache-expiry path, not a tautology.""" + driver = _FakeConstraintDriver(present=False) + coord = MaintenanceCoordinator() + coord.bind_driver(driver, probe_ttl_seconds=10.0) # long TTL + + assert await coord.gate_closed() is True + + driver.present = True # flips immediately, but cache is still warm + # NO sleep -- still within the 10s TTL window. + assert await coord.gate_closed() is True # stale cached answer + + +# --------------------------------------------------------------------------- +# A4b -- THE UNTAGGED LATCH: degraded->healthy self-clears with NO restart +# --------------------------------------------------------------------------- + + +class _FakeUntaggedResult: + def __init__(self, count: int) -> None: + self._count = count + self._yielded = False + + def __aiter__(self) -> _FakeUntaggedResult: + return self + + async def __anext__(self) -> dict[str, int]: + if self._yielded: + raise StopAsyncIteration + self._yielded = True + return {"c": self._count} + + +class _FakeUntaggedSession: + def __init__(self, driver: _FakeUntaggedDriver) -> None: + self._driver = driver + + async def __aenter__(self) -> Self: + return self + + async def __aexit__(self, *exc: object) -> None: + return None + + async def run(self, cypher: str, *args: Any, **kwargs: Any) -> _FakeUntaggedResult: + self._driver.call_count += 1 + if "SHOW CONSTRAINTS" in cypher: + return _FakeUntaggedResult(1) # constraint present + if ":Node)" in cypher: + return _FakeUntaggedResult(self._driver.tagged) # tagged-node count + return _FakeUntaggedResult(self._driver.total) # total-node count + + +class _FakeUntaggedDriver: + """Test double: constraint always present; untagged = total - tagged, and + both are flippable at runtime so a test can simulate an out-of-band repair + tagging the last untagged node WITHOUT a restart.""" + + def __init__(self, total: int, tagged: int) -> None: + self.total = total + self.tagged = tagged + self.call_count = 0 + + def session(self, **kwargs: Any) -> _FakeUntaggedSession: + return _FakeUntaggedSession(self) + + +class TestUntaggedNoRestartSelfClear: + async def test_degraded_self_clears_within_ttl_no_restart(self) -> None: + """The untagged half of the latch: schema_health/mode report `degraded` + (1 node lacking :Node) at boot; an out-of-band repair tags it; the SAME + coordinator (no restart) reports `healthy` within one TTL.""" + driver = _FakeUntaggedDriver(total=1, tagged=0) # 1 untagged -> degraded + coord = MaintenanceCoordinator() + coord.bind_driver(driver, untagged=1, probe_ttl_seconds=0.05) + + st = await coord.status() + assert st.mode == "degraded" + assert st.untagged_nodes == 1 + assert await coord.gate_closed() is False # degraded NEVER closes gate + + # Out-of-band repair tags the node -- untagged now 0. NO restart. + driver.tagged = 1 + await asyncio.sleep(0.1) # let the untagged-probe TTL expire + + st2 = await coord.status() + assert st2.mode == "healthy" # self-cleared, no restart + assert st2.untagged_nodes == 0 + assert st2.reason is None + + async def test_non_vacuous_within_ttl_stays_degraded(self) -> None: + """Non-vacuity: without waiting out the TTL the seeded/cached count is + returned -- proving the test above exercises the expiry path, not a + tautology.""" + driver = _FakeUntaggedDriver(total=1, tagged=0) + coord = MaintenanceCoordinator() + coord.bind_driver(driver, untagged=1, probe_ttl_seconds=10.0) # long TTL + + st = await coord.status() + assert st.mode == "degraded" + + driver.tagged = 1 # repaired immediately, but cache still warm + # NO sleep -- still within the 10s TTL window. + st2 = await coord.status() + assert st2.mode == "degraded" # stale cached count + assert st2.untagged_nodes == 1 + + +# --------------------------------------------------------------------------- +# A5 -- op_running holds the gate closed even when constraint IS present +# --------------------------------------------------------------------------- + + +class TestOpRunningKeepsGateClosed: + async def test_op_running_with_constraint_present_still_closed(self) -> None: + coord = MaintenanceCoordinator() + coord.bind_driver(_FakeConstraintDriver(present=True)) + assert await coord.gate_closed() is False # sanity: open beforehand + + run_id = coord.try_begin_op() + assert run_id is not None + + assert await coord.gate_closed() is True # op_running term (D-C) + + coord.finish_op(run_id, records_affected=0, error=None) + assert await coord.gate_closed() is False # reopens once finished + + +# --------------------------------------------------------------------------- +# A12 -- transition logging fires EXACTLY once per transition +# --------------------------------------------------------------------------- + + +class TestTransitionLogging: + async def test_closed_open_closed_emits_exactly_one_each( + self, caplog: pytest.LogCaptureFixture + ) -> None: + driver = _FakeConstraintDriver( + present=False + ) # start closed (migration required) + coord = MaintenanceCoordinator() + coord.bind_driver(driver, probe_ttl_seconds=0.02) + + with caplog.at_level(logging.INFO, logger="context_intelligence_server"): + # Already "maintenance" from bind (closed=migration required). + await coord.status() + await coord.status() # repeat call: must not double-log + + # Flip open. + driver.present = True + await asyncio.sleep(0.05) + await coord.status() + await coord.status() # repeat: must not double-log + + # Flip closed again. + driver.present = False + await asyncio.sleep(0.05) + await coord.status() + + entered = [r for r in caplog.records if r.getMessage() == "maintenance_entered"] + completed = [ + r for r in caplog.records if r.getMessage() == "maintenance_completed" + ] + # Two "entered" (initial closed state, then re-closed) and one + # "completed" (the middle open window). + assert len(entered) == 2 + assert len(completed) == 1 + + +# --------------------------------------------------------------------------- +# A13 -- no driver bound => gate OPEN (the existing-suite no-regression rule) +# --------------------------------------------------------------------------- + + +class TestNoDriverBound: + async def test_unbound_coordinator_gate_is_open(self) -> None: + coord = MaintenanceCoordinator() # bind_driver() never called + assert await coord.gate_closed() is False + st = await coord.status() + assert st.mode == "unknown" + assert st.constraint_present is None + + +# --------------------------------------------------------------------------- +# try_begin_op / finish_op / current_op -- the CAS seam itself +# --------------------------------------------------------------------------- + + +class TestOpSeam: + def test_try_begin_op_is_synchronous_cas(self) -> None: + coord = MaintenanceCoordinator() + run_id_1 = coord.try_begin_op() + assert run_id_1 is not None + run_id_2 = coord.try_begin_op() + assert run_id_2 is None # already running -- CAS refuses a second op + + def test_finish_op_sets_completed_at_and_state(self) -> None: + coord = MaintenanceCoordinator() + run_id = coord.try_begin_op() + assert run_id is not None + assert coord.current_op().completed_at is None + + coord.finish_op(run_id, records_affected=3, error=None) + + op = coord.current_op() + assert op.state == "succeeded" + assert op.completed_at is not None + assert op.records_affected == 3 + + def test_finish_op_with_error_marks_failed(self) -> None: + coord = MaintenanceCoordinator() + run_id = coord.try_begin_op() + assert run_id is not None + + coord.finish_op(run_id, records_affected=None, error="boom") + + op = coord.current_op() + assert op.state == "failed" + assert op.error == "boom" + + def test_op_state_initializes_unknown_not_succeeded(self) -> None: + """Council D4: never-run must not read as a false 'succeeded'.""" + coord = MaintenanceCoordinator() + assert coord.current_op().state == "unknown" + + def test_finish_op_stale_run_id_is_ignored(self) -> None: + coord = MaintenanceCoordinator() + run_id = coord.try_begin_op() + assert run_id is not None + + coord.finish_op("not-the-real-run-id", records_affected=1, error=None) + + # The real op is untouched -- still running, not clobbered by a + # foreign/stale completion signal. + assert coord.current_op().state == "running" + + +# --------------------------------------------------------------------------- +# Allow-list sanity (used by the startup assertion + the middleware) +# --------------------------------------------------------------------------- + + +def test_allow_list_contains_required_paths() -> None: + assert "/admin/maintenance" in MAINTENANCE_ALLOW_LIST + assert "/status" in MAINTENANCE_ALLOW_LIST + assert "/version" in MAINTENANCE_ALLOW_LIST diff --git a/tests/test_migrations_run.py b/tests/test_migrations_run.py new file mode 100644 index 00000000..36863d9b --- /dev/null +++ b/tests/test_migrations_run.py @@ -0,0 +1,203 @@ +"""Tests for migrations/run.py -- the standalone, out-of-band graph +rectification CLI. Unit-level only (mocked driver/config); real +rectification against a live Neo4j is DTU-validated separately. +""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest +from migrations import run as run_module + + +def _fake_driver() -> MagicMock: + driver = MagicMock() + driver.verify_connectivity = AsyncMock(return_value=None) + driver.close = AsyncMock(return_value=None) + session_cm = MagicMock() + session_cm.__aenter__ = AsyncMock(return_value=session_cm) + session_cm.__aexit__ = AsyncMock(return_value=False) + driver.session = MagicMock(return_value=session_cm) + return driver + + +@pytest.fixture(autouse=True) +def _patch_settings_and_driver(): + """Every test patches get_settings/build_neo4j_driver so no real config + or Neo4j connection is required (mirrors tests/test_doctor.py).""" + with ( + patch.object(run_module, "get_settings") as mock_get_settings, + patch.object(run_module, "build_neo4j_driver") as mock_build_driver, + ): + admin_config = MagicMock() + admin_config.model_copy.return_value = "admin-cfg" + mock_get_settings.return_value.resolve_neo4j_admin.return_value = admin_config + mock_build_driver.return_value = _fake_driver() + yield mock_build_driver.return_value + + +def test_module_imports_cleanly() -> None: + """The module must import without touching Neo4j (no side effects at + import time) -- required for the DTU update script's `grep` probe.""" + import migrations.run # noqa: F401 -- import-cleanliness check + + +def test_status_flag_exists_and_parses() -> None: + args = run_module.build_parser().parse_args(["--status"]) + assert args.status is True + assert args.apply is False + + +def test_apply_flag_exists_and_parses() -> None: + args = run_module.build_parser().parse_args(["--apply"]) + assert args.apply is True + assert args.status is False + + +def test_status_and_apply_are_mutually_exclusive() -> None: + with pytest.raises(SystemExit): + run_module.build_parser().parse_args(["--status", "--apply"]) + + +def test_one_of_status_or_apply_is_required() -> None: + with pytest.raises(SystemExit): + run_module.build_parser().parse_args([]) + + +def test_neo4j_overrides_are_optional_flags() -> None: + args = run_module.build_parser().parse_args( + [ + "--status", + "--neo4j-url", + "bolt://example:7687", + "--neo4j-user", + "u", + "--neo4j-password", + "p", + ] + ) + assert args.neo4j_url == "bolt://example:7687" + assert args.neo4j_user == "u" + assert args.neo4j_password == "p" + + +def test_banner_declares_from_to_and_out_of_band( + capsys: pytest.CaptureFixture[str], +) -> None: + run_module._print_banner() + out = capsys.readouterr().out + assert run_module.FROM_SERVER_VERSION in out + assert run_module.TO_SERVER_VERSION in out + assert str(run_module.FROM_SCHEMA_VERSION) in out + assert str(run_module.TO_SCHEMA_VERSION) in out + assert "OUT-OF-BAND" in out + assert "IDEMPOTENT" in out + assert "never runs at server startup" in out + + +async def test_status_mode_does_not_call_run_repair( + _patch_settings_and_driver: MagicMock, +) -> None: + with ( + patch.object( + run_module, + "diagnose", + AsyncMock(return_value={"untagged_nodes": 0, "duplicate_nodes": 0}), + ), + patch.object(run_module, "run_repair", AsyncMock()) as repair_mock, + patch.object(run_module, "_constraint_present", AsyncMock(return_value=True)), + ): + args = run_module.build_parser().parse_args(["--status"]) + code = await run_module._amain(args) + + assert code == 0 + repair_mock.assert_not_awaited() + + +async def test_apply_mode_calls_run_repair( + _patch_settings_and_driver: MagicMock, +) -> None: + diagnose_mock = AsyncMock( + side_effect=[ + {"untagged_nodes": 5, "duplicate_nodes": 2}, # before + {"untagged_nodes": 0, "duplicate_nodes": 0}, # after + ] + ) + repair_mock = AsyncMock(return_value={"duplicates_removed": 2, "nodes_tagged": 5}) + with ( + patch.object(run_module, "diagnose", diagnose_mock), + patch.object(run_module, "run_repair", repair_mock), + patch.object(run_module, "_constraint_present", AsyncMock(return_value=True)), + ): + args = run_module.build_parser().parse_args(["--apply"]) + code = await run_module._amain(args) + + assert code == 0 + repair_mock.assert_awaited_once() + assert diagnose_mock.await_count == 2 + + +async def test_apply_is_idempotent_noop_on_already_clean_graph( + _patch_settings_and_driver: MagicMock, +) -> None: + """--apply still calls run_repair (it is itself idempotent/no-op-safe), + but a clean before-state must still report healthy after.""" + diagnose_mock = AsyncMock(return_value={"untagged_nodes": 0, "duplicate_nodes": 0}) + repair_mock = AsyncMock(return_value={"duplicates_removed": 0, "nodes_tagged": 0}) + with ( + patch.object(run_module, "diagnose", diagnose_mock), + patch.object(run_module, "run_repair", repair_mock), + patch.object(run_module, "_constraint_present", AsyncMock(return_value=True)), + ): + args = run_module.build_parser().parse_args(["--apply"]) + code = await run_module._amain(args) + + assert code == 0 + repair_mock.assert_awaited_once() + + +async def test_status_unreachable_neo4j_fails_gracefully( + _patch_settings_and_driver: MagicMock, +) -> None: + """Pointed at an unreachable URL, --status must not crash/traceback -- + it reports the connectivity failure and returns a non-zero exit code.""" + driver = _patch_settings_and_driver + driver.verify_connectivity = AsyncMock( + side_effect=RuntimeError("connection refused") + ) + + args = run_module.build_parser().parse_args(["--status"]) + code = await run_module._amain(args) + + assert code != 0 + driver.close.assert_awaited_once() + + +async def test_status_unreachable_neo4j_never_touches_diagnose_or_repair( + _patch_settings_and_driver: MagicMock, +) -> None: + driver = _patch_settings_and_driver + driver.verify_connectivity = AsyncMock( + side_effect=RuntimeError("connection refused") + ) + with ( + patch.object(run_module, "diagnose", AsyncMock()) as diagnose_mock, + patch.object(run_module, "run_repair", AsyncMock()) as repair_mock, + ): + args = run_module.build_parser().parse_args(["--status"]) + await run_module._amain(args) + + diagnose_mock.assert_not_awaited() + repair_mock.assert_not_awaited() + + +async def test_constraint_present_returns_none_on_probe_failure( + _patch_settings_and_driver: MagicMock, +) -> None: + driver = _patch_settings_and_driver + driver.session.side_effect = RuntimeError("catalog read failed") + + result = await run_module._constraint_present(driver) + + assert result is None diff --git a/tests/test_models.py b/tests/test_models.py index 1cad9da8..7fbccf97 100644 --- a/tests/test_models.py +++ b/tests/test_models.py @@ -74,6 +74,61 @@ def test_event_request_workspace_non_empty_accepted(): assert req.workspace == "my-project-slug" +def test_event_request_working_dir_none_accepted(): + """EventRequest accepts working_dir=None (the default; older clients omit it).""" + req = EventRequest( + event="session:start", + workspace="main", + working_dir=None, + data={"session_id": "abc123"}, + ) + assert req.working_dir is None + + +def test_event_request_working_dir_omitted_defaults_to_none(): + """EventRequest defaults working_dir to None when the field is absent entirely.""" + req = EventRequest( + event="session:start", + workspace="main", + data={"session_id": "abc123"}, + ) + assert req.working_dir is None + + +def test_event_request_working_dir_valid_path_accepted(): + """EventRequest accepts a valid non-blank working_dir path.""" + req = EventRequest( + event="session:start", + workspace="main", + working_dir="/home/user/project", + data={"session_id": "abc123"}, + ) + assert req.working_dir == "/home/user/project" + + +def test_event_request_working_dir_empty_string_raises(): + """W-3 defect 1: EventRequest raises ValidationError when working_dir is "" .""" + with pytest.raises(ValidationError): + EventRequest( + event="session:start", + workspace="main", + working_dir="", + data={"session_id": "abc123"}, + ) + + +def test_event_request_working_dir_whitespace_only_raises(): + """W-3 defect 1: a whitespace-only working_dir (e.g. " ") must be rejected, + not written through to the Session node verbatim.""" + with pytest.raises(ValidationError): + EventRequest( + event="session:start", + workspace="main", + working_dir=" ", + data={"session_id": "abc123"}, + ) + + def test_event_request_data_without_session_id(): """EventRequest accepts data dict that has no session_id key.""" req = EventRequest( diff --git a/tests/test_neo4j_store.py b/tests/test_neo4j_store.py index 2051e04f..b82694b1 100644 --- a/tests/test_neo4j_store.py +++ b/tests/test_neo4j_store.py @@ -2680,6 +2680,145 @@ def test_node_props_preserves_legitimate_props(self) -> None: assert props.get("active") is True, "_build_node_props must preserve 'active'" +# --------------------------------------------------------------------------- +# W-3 defect 2: working_dir DB-level non-overwrite guarantee +# --------------------------------------------------------------------------- +# Unit-level coverage for the structural pieces (excluded from row.props, +# carried separately, coalesce SET present in the Session write). The actual +# non-overwrite BEHAVIOR against a real graph (existing value survives a +# conflicting later write) is proven against live Neo4j in +# tests/neo4j/test_working_dir_non_overwrite.py -- a mocked tx cannot execute +# Cypher, so it can only assert the query/rows shape, not runtime semantics. + + +class TestBuildNodePropsWorkingDir: + """_build_node_props excludes 'working_dir' from the generic props merge. + + working_dir must never ride in ``row.props`` (which is blindly merged via + ``SET n += row.props``) -- it travels separately so the Session write can + apply ``coalesce(n.working_dir, row.working_dir)`` instead of an overwrite. + """ + + def test_node_props_excludes_working_dir(self) -> None: + """data dict containing working_dir -> built props has NO working_dir key.""" + from context_intelligence_server.neo4j_store import _build_node_props + + data: dict = { + "labels": ["Session"], + "working_dir": "/home/user/project", + "status": "running", + } + props = _build_node_props(data, "ws-1") + + assert "working_dir" not in props, ( + "_build_node_props must NOT include working_dir in props; " + "working_dir travels separately so it can be coalesced, not overwritten" + ) + + def test_node_props_preserves_other_fields_alongside_working_dir(self) -> None: + """Excluding working_dir must not disturb other legitimate props.""" + from context_intelligence_server.neo4j_store import _build_node_props + + data: dict = { + "labels": ["Session"], + "working_dir": "/home/user/project", + "status": "running", + "agent": "root", + } + props = _build_node_props(data, "ws-1") + + assert props.get("status") == "running" + assert props.get("agent") == "root" + assert props.get("workspace") == "ws-1" + + +class TestSessionMergeWorkingDirCoalesce: + """The inline Session MERGE in _write_batch applies a DB-level coalesce for + working_dir instead of the blind ``SET n += row.props`` overwrite. + """ + + def test_session_merge_source_contains_coalesce_clause(self) -> None: + """_write_batch source contains the coalesce SET for working_dir on the + Session-node write path.""" + import inspect + + from context_intelligence_server import neo4j_store + + source = inspect.getsource(neo4j_store._write_batch) + assert "SET n.working_dir = coalesce(n.working_dir, row.working_dir)" in ( + source + ), ( + "_write_batch must apply a DB-level coalesce for working_dir on the " + "Session-node MERGE so an already-set value is never clobbered" + ) + + def test_session_merge_coalesce_is_separate_from_props_merge(self) -> None: + """The coalesce SET must be a clause distinct from `SET n += row.props`, + confirming working_dir cannot ride the blind merge AND be coalesced + redundantly (single source of truth for the value).""" + import inspect + + from context_intelligence_server import neo4j_store + + source = inspect.getsource(neo4j_store._write_batch) + merge_idx = source.index("SET n += row.props, n:Session") + coalesce_idx = source.index( + "SET n.working_dir = coalesce(n.working_dir, row.working_dir)" + ) + assert coalesce_idx > merge_idx, ( + "coalesce SET must follow the main props/label SET so it observes " + "the fully-applied node state before deciding the working_dir value" + ) + + @pytest.mark.asyncio + async def test_session_row_carries_working_dir_key_when_present(self) -> None: + """A Session node with working_dir in its buffered data produces a row + whose top-level 'working_dir' key is populated (not nested in props).""" + mock_tx, mock_session = _make_flush_mocks() + store = Neo4jGraphStore(uri="bolt://fake", auth=("u", "p"), workspace="ws-wd") + store._driver.session = lambda **_: mock_session # type: ignore[method-assign] + store._schema_initialized = True + + await store.upsert_node( + "sess-1", {"labels": ["Session"], "working_dir": "/repo"} + ) + await store.flush() + + session_calls = [ + c for c in mock_tx.run.call_args_list if "n:Session" in str(c.args[0]) + ] + assert session_calls, "Expected a Session-node MERGE call" + rows = session_calls[0].kwargs.get("rows") + assert rows and rows[0].get("working_dir") == "/repo", ( + f"Expected row['working_dir'] == '/repo', got rows={rows!r}" + ) + assert "working_dir" not in rows[0]["props"], ( + "working_dir must not also appear inside row['props']" + ) + + @pytest.mark.asyncio + async def test_session_row_omits_working_dir_key_when_absent(self) -> None: + """A Session node with no working_dir produces a row with NO top-level + 'working_dir' key at all (so coalesce(n.working_dir, row.working_dir) + sees a missing-map-key null, never an accidental empty string).""" + mock_tx, mock_session = _make_flush_mocks() + store = Neo4jGraphStore(uri="bolt://fake", auth=("u", "p"), workspace="ws-wd") + store._driver.session = lambda **_: mock_session # type: ignore[method-assign] + store._schema_initialized = True + + await store.upsert_node("sess-2", {"labels": ["Session"], "status": "running"}) + await store.flush() + + session_calls = [ + c for c in mock_tx.run.call_args_list if "n:Session" in str(c.args[0]) + ] + assert session_calls, "Expected a Session-node MERGE call" + rows = session_calls[0].kwargs.get("rows") + assert rows and "working_dir" not in rows[0], ( + f"Expected no 'working_dir' key when absent from source data, got {rows[0]!r}" + ) + + # --------------------------------------------------------------------------- # T21: Phase 2E — edge/relationship provenance (write-once, worker-level scalar) # --------------------------------------------------------------------------- @@ -3152,7 +3291,7 @@ async def test_benign_already_exists_does_not_raise(self) -> None: # --------------------------------------------------------------------------- # TestFailOnDataConflictDefault # -# Regression for the PR #67 merge-blocker (reviewer Salil, commit 14a6d30): +# Regression for the PR #67 merge-blocker (commit 14a6d30): # ensure_neo4j_schema's :Node constraint step used to hardcode # fail_on_data_conflict=True with no way for callers to opt out. That is # correct for cold start (main.py:174, refuse to boot) but WRONG for the @@ -3246,7 +3385,7 @@ class TestDataConflictOnFlushPathDoesNotDeadLetter: """END-TO-END regression: a genuine :Node data conflict on the flush path must NOT dead-letter the in-flight batch. - This is the exact bug reported by Salil in PR #67 review: commit 14a6d30 + This is the exact bug reported in the PR #67 review: commit 14a6d30 hardcoded fail_on_data_conflict=True inside ensure_neo4j_schema's Step 6, reachable from Neo4jGraphStore._ensure_schema (called from _flush_body's try block on every flush until the schema latches). A RuntimeError raised diff --git a/tests/test_queue_manager.py b/tests/test_queue_manager.py index bd270601..c316b5c3 100644 --- a/tests/test_queue_manager.py +++ b/tests/test_queue_manager.py @@ -2,10 +2,10 @@ from __future__ import annotations +import json import time import pytest - from context_intelligence_server.queue_manager import Batch, QueueManager @@ -136,7 +136,7 @@ async def test_commit_advances_offset(qm): await qm.append("s1", b"a") await qm.append("s1", b"b") first = await qm.read_batch("s1", max_items=1) - await qm.commit("s1", first.end_offset) + await qm.commit("s1", first.end_offset, None) await qm.append("s1", b"c") second = await qm.read_batch("s1", max_items=10) assert second.lines == [b"b", b"c"] @@ -149,7 +149,7 @@ async def test_commit_persists_across_a_new_instance(tmp_path): await qm1.append("s1", b"a") await qm1.append("s1", b"b") batch = await qm1.read_batch("s1", max_items=1) - await qm1.commit("s1", batch.end_offset) + await qm1.commit("s1", batch.end_offset, None) qm2 = QueueManager(queues_dir=qdir) # simulate restart resumed = await qm2.read_batch("s1", max_items=10) assert resumed.lines == [b"b"] @@ -157,9 +157,12 @@ async def test_commit_persists_across_a_new_instance(tmp_path): async def test_commit_is_atomic_no_temp_leftover(qm, tmp_path): await qm.append("s1", b"a") - await qm.commit("s1", 2) + await qm.commit("s1", 2, None) qdir = tmp_path / "queues" - assert (qdir / "s1.offset").read_text("utf-8") == "2" + # I5b: .offset is now a single JSON record {"v":1,"offset":...,"cursor":...} + # rather than a bare integer -- see queue_manager.py module docstring. + record = json.loads((qdir / "s1.offset").read_text("utf-8")) + assert record == {"v": 1, "offset": 2, "cursor": None} assert list(qdir.glob("*.tmp")) == [] @@ -167,7 +170,7 @@ async def test_active_sessions_excludes_fully_committed(qm): await qm.append("s_active", b"x") # appended, never committed -> undrained await qm.append("s_done", b"y") done = await qm.read_batch("s_done", max_items=10) - await qm.commit("s_done", done.end_offset) # drained + await qm.commit("s_done", done.end_offset, None) # drained active = await qm.active_sessions() assert active == ["s_active"] @@ -180,7 +183,7 @@ async def test_recover_reports_session_with_uncommitted_complete_line(qm, tmp_pa log = tmp_path / "queues" / "s1.log" log.write_bytes(b"a\nb\nTORN") # two complete lines + torn tail assert await qm.recover() == ["s1"] - await qm.commit("s1", 4) # past 'a\nb\n' == 4 bytes + await qm.commit("s1", 4, None) # past 'a\nb\n' == 4 bytes assert await qm.recover() == [] # only torn tail remains -> not recoverable @@ -211,7 +214,7 @@ async def test_read_batch_rejects_unsafe_session_id(qm, bad_id): @pytest.mark.parametrize("bad_id", ["", "a/b", "a\\b", "a\x00b"]) async def test_commit_rejects_unsafe_session_id(qm, bad_id): with pytest.raises(ValueError): - await qm.commit(bad_id, 0) + await qm.commit(bad_id, 0, None) @pytest.mark.parametrize("bad_id", ["", "a/b", "a\\b", "a\x00b"]) @@ -231,7 +234,7 @@ async def test_delete_drained_removes_log_and_offset_keeps_dead(tmp_path) -> Non qm = QueueManager(queues_dir=tmp_path) await qm.append("s", b"line") - await qm.commit("s", 5) + await qm.commit("s", 5, None) await qm.dead_letter("s", b"bad\n", "boom") await qm.delete_drained("s") @@ -325,7 +328,7 @@ async def test_recovery_seed_counts_pending_and_committed(qm): await qm.append("s1", b"a") await qm.append("s1", b"b") await qm.append("s1", b"c") - await qm.commit("s1", 4) # commit the first two complete lines + await qm.commit("s1", 4, None) # commit the first two complete lines accepted, written = await qm.recovery_seed_counts() @@ -336,7 +339,7 @@ async def test_recovery_seed_counts_pending_and_committed(qm): async def test_recovery_seed_counts_committed_includes_dead(qm): # C=1 committed, P=0 pending, D=1 dead. before-dead == 0. await qm.append("s2", b"a") - await qm.commit("s2", 2) + await qm.commit("s2", 2, None) await qm.dead_letter("s2", b"a", error="boom") accepted, written = await qm.recovery_seed_counts() @@ -360,10 +363,10 @@ async def test_recovery_seed_counts_residual_is_zero_mixed_shape(qm): await qm.append("a", b"1") await qm.append("a", b"2") await qm.append("a", b"3") - await qm.commit("a", 4) + await qm.commit("a", 4, None) # Key B: 1 committed + 1 dead. await qm.append("b", b"x") - await qm.commit("b", 2) + await qm.commit("b", 2, None) await qm.dead_letter("b", b"x", error="boom") # Key C: dead-only (log reclaimed). await qm.dead_letter("c", b"poison", error="boom") @@ -453,7 +456,7 @@ async def test_recovery_seed_counts_replay_window_residual_zero(qm): # log = [line0 committed][line0 re-appended pending]. C=1, P=1, D=1. # The re-appended line is absorbed into accepted_seed (counted in P and D). await qm.append("s6", b"a") - await qm.commit("s6", 2) + await qm.commit("s6", 2, None) await qm.dead_letter("s6", b"a", error="boom") await qm.append("s6", b"a") # re-append the dead line for replay @@ -493,7 +496,7 @@ async def test_spool_stats_fully_committed_session_not_pending(qm): (spool_bytes_total still reflects them).""" await qm.append("s1", b"a") line = b"a\n" - await qm.commit("s1", len(line)) + await qm.commit("s1", len(line), None) stats = await qm.spool_stats() @@ -518,7 +521,7 @@ async def test_spool_stats_multiple_sessions_aggregate(qm): await qm.append("s1", b"a") # pending await qm.append("s2", b"b") line = b"b\n" - await qm.commit("s2", len(line)) # fully committed, not pending + await qm.commit("s2", len(line), None) # fully committed, not pending await qm.append("s3", b"c") # pending stats = await qm.spool_stats() @@ -677,7 +680,7 @@ async def test_recovery_seed_counts_unchanged_under_streaming(qm): await qm.append("s1", b"a") await qm.append("s1", b"bb") line1 = b"a\n" - await qm.commit("s1", len(line1)) # 1 written, 1 still pending + await qm.commit("s1", len(line1), None) # 1 written, 1 still pending accepted, written = await qm.recovery_seed_counts() @@ -779,7 +782,7 @@ async def test_spool_stats_healthy_offsets_report_zero_corrupt(qm): fire on the normal committed-offset path).""" await qm.append("s1", b"a") line = b"a\n" - await qm.commit("s1", len(line)) # writes a valid numeric .offset + await qm.commit("s1", len(line), None) # writes a valid numeric .offset qm._spool_cache = None stats = await qm.spool_stats() diff --git a/tests/test_registry.py b/tests/test_registry.py index c31efeb4..348dba09 100644 --- a/tests/test_registry.py +++ b/tests/test_registry.py @@ -915,6 +915,138 @@ async def test_offset_not_committed_when_flush_fails( except asyncio.CancelledError: pass + +class TestMaintenanceGate: + """WS-3a spec sec 7a, A6/A7: the drain-loop gate placement guarantees + that NOTHING downstream (read_batch/process/flush/commit) runs while + the coordinator reports the gate closed, and that a batch gated then + un-gated is processed/committed exactly once (no duplicate, no loss).""" + + async def test_gate_closed_blocks_read_batch_and_commit( + self, reg_qm: tuple[SessionRegistry, Any], monkeypatch: pytest.MonkeyPatch + ) -> None: + """A6 (MUST-FIX #5 offset ordering): while gated, read_batch is never + reached, so qm.commit never runs and the on-disk offset never + advances -- proven by reverting: without the gate check, read_batch + IS reached immediately (see the companion revert-check below).""" + reg, qm = reg_qm + sid = "gated-session" + worker = SessionWorker( + session_id=sid, workspace="/ws", services=HookStateService(workspace="/ws") + ) + worker.services.graph.flush = AsyncMock() # type: ignore[method-assign] + reg._register_for_test(worker) + await qm.append(sid, _line("tool_call", "/ws", {"session_id": sid})) + + read_batch_spy = AsyncMock(wraps=qm.read_batch) + commit_spy = AsyncMock(wraps=qm.commit) + monkeypatch.setattr( + registry_module.coordinator, + "gate_closed", + AsyncMock(return_value=True), + ) + monkeypatch.setattr(qm, "read_batch", read_batch_spy) + monkeypatch.setattr(qm, "commit", commit_spy) + + task = asyncio.create_task(reg.drain_worker(worker, flush_timeout=10.0)) + await asyncio.sleep(0.2) + task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await task + + read_batch_spy.assert_not_called() + commit_spy.assert_not_called() + assert worker.events_processed == 0 + # The line is still fully queued -- offset never advanced. + monkeypatch.undo() + remaining = await qm.read_batch(sid, 10) + assert len(remaining.lines) == 1 + + async def test_revert_check_gate_closed_false_reaches_read_batch( + self, reg_qm: tuple[SessionRegistry, Any], monkeypatch: pytest.MonkeyPatch + ) -> None: + """Non-vacuity companion to the test above: with gate_closed() forced + FALSE (i.e. reverting the gate's effect), read_batch/process/commit + DO run -- proving the previous test's zero-calls assertion is a real + gate effect, not an artifact of the test setup.""" + reg, qm = reg_qm + sid = "ungated-session" + worker = SessionWorker( + session_id=sid, workspace="/ws", services=HookStateService(workspace="/ws") + ) + worker.services.graph.flush = AsyncMock() # type: ignore[method-assign] + reg._register_for_test(worker) + await qm.append(sid, _line("tool_call", "/ws", {"session_id": sid})) + + monkeypatch.setattr( + registry_module.coordinator, + "gate_closed", + AsyncMock(return_value=False), + ) + + with patch( + "context_intelligence_server.registry.process_event", + new_callable=AsyncMock, + ) as mock_process: + task = asyncio.create_task(reg.drain_worker(worker, flush_timeout=10.0)) + for _ in range(50): + await asyncio.sleep(0.02) + if (await qm.read_batch(sid, 10)).lines == []: + break + task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await task + + mock_process.assert_awaited() + assert (await qm.read_batch(sid, 10)).lines == [] # offset DID advance + + async def test_gate_reopens_batch_processed_exactly_once( + self, reg_qm: tuple[SessionRegistry, Any], monkeypatch: pytest.MonkeyPatch + ) -> None: + """A7: a batch that arrives while gated is processed/committed + EXACTLY once after the gate opens -- no duplicate dispatch, no loss.""" + reg, qm = reg_qm + sid = "gate-reopen-session" + worker = SessionWorker( + session_id=sid, workspace="/ws", services=HookStateService(workspace="/ws") + ) + worker.services.graph.flush = AsyncMock() # type: ignore[method-assign] + reg._register_for_test(worker) + await qm.append(sid, _line("tool_call", "/ws", {"session_id": sid})) + + gate_state = {"closed": True} + + async def fake_gate_closed() -> bool: + return gate_state["closed"] + + monkeypatch.setattr(registry_module, "_GATED_POLL_INTERVAL", 0.05) + monkeypatch.setattr( + registry_module.coordinator, "gate_closed", fake_gate_closed + ) + + with patch( + "context_intelligence_server.registry.process_event", + new_callable=AsyncMock, + ) as mock_process: + task = asyncio.create_task(reg.drain_worker(worker, flush_timeout=10.0)) + await asyncio.sleep(0.15) + assert mock_process.await_count == 0 # still gated: untouched + + gate_state["closed"] = False # reopen + for _ in range(100): + await asyncio.sleep(0.02) + if (await qm.read_batch(sid, 10)).lines == []: + break + task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await task + + assert mock_process.await_count == 1 + assert worker.events_processed == 1 + assert (await qm.read_batch(sid, 10)).lines == [] + + +class TestDrainWorkerMethodExists: def test_drain_worker_is_method_on_registry(self) -> None: """drain_worker must be an instance coroutine method on SessionRegistry (folded from the queue-era TestDrainLoopCallsProcessEvent).""" @@ -2190,3 +2322,155 @@ def test_invariant_violation_is_observed_and_not_overwritten( # 3. Worker is returned — no exception raised assert worker is not None + + +# --------------------------------------------------------------------------- +# Phantom-cursor guard, common retry branch (budget NOT exhausted): a +# transient flush failure (proxy: DeadlockDetected) that eventually SUCCEEDS +# via drain_worker's common retry path must not replay a cursor-mutating +# handler's write more than once for the same never-committed batch. Mirrors +# TestDurableLinearPoisonIsolation's BLOCKER-1 guard, but for the +# retry-then-succeed path (`_handle_exhausted_batch` is never reached here) +# rather than the give-up (dead-letter) path. +# +# These drive the REAL process_event / IterationHandler (not mocked) against +# the default in-memory GraphState so both the emitted Iteration node_ids and +# the iteration_count cursor are directly observable — chosen over extending +# _AccumBufferGraph because GraphState already records real node_ids keyed +# exactly as the handler creates them, with no test-double changes needed. +# --------------------------------------------------------------------------- + + +class TestPhantomCursorGuardCommonRetryPath: + async def test_transient_flush_failure_then_success_produces_single_iteration( + self, reg_qm: tuple[SessionRegistry, Any] + ) -> None: + """Attempt 1's flush fails transiently; attempt 2 succeeds through the + COMMON retry branch (budget not exhausted). Before the fix: attempt 1 + increments iteration_count to 1 and creates node '::iteration::1'; the + failed flush is never rolled back, so attempt 2 (replaying the SAME + batch) increments iteration_count again to 2 and creates a SECOND node + '::iteration::2' for a batch that only ever committed once. After the + fix: the cursor is restored before replay, so attempt 2 recomputes + iteration_number 1 and only ONE node/commit results. + """ + reg, qm = reg_qm + sid = "s-phantom-retry" + worker = SessionWorker( + session_id=sid, + workspace="/ws", + services=HookStateService(workspace="/ws"), + ) + + real_flush = worker.services.graph.flush + call_count = 0 + + async def _flaky_flush() -> None: + nonlocal call_count + call_count += 1 + if call_count == 1: + raise RuntimeError("DeadlockDetected") + await real_flush() + + worker.services.graph.flush = _flaky_flush # type: ignore[method-assign] + reg._register_for_test(worker) + + await qm.append( + sid, + _line( + "provider:request", + "/ws", + {"session_id": sid, "timestamp": "2026-06-11T12:00:00+00:00"}, + ), + ) + task = asyncio.create_task(reg.drain_worker(worker, flush_timeout=10.0)) + for _ in range(400): + await asyncio.sleep(0.01) + if (await qm.read_batch(sid, 10)).lines == []: + break + task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await task + + # Prove a real fail-then-succeed retry actually happened (not a + # vacuous pass from the batch never being retried at all). + assert call_count >= 2, f"Expected >=2 flush calls, got {call_count}" + + iteration_nodes = sorted( + node_id + for node_id in worker.services.graph._nodes + if "::iteration::" in node_id + ) + assert iteration_nodes == [f"{sid}::iteration::1"], ( + f"Expected exactly one Iteration node, got: {iteration_nodes}" + ) + assert worker.services.data_layer_2.iteration_count == 1, ( + "iteration_count cursor must not be advanced beyond the single " + "committed iteration; got " + f"{worker.services.data_layer_2.iteration_count}" + ) + + async def test_chained_transient_failures_then_success_produces_single_iteration( + self, reg_qm: tuple[SessionRegistry, Any] + ) -> None: + """Two chained transient failures (attempts 1 and 2) before a + succeeding attempt 3 — proving the restore-to-pre-batch-snapshot is + applied on EVERY failed attempt, not just the first, so a 3rd attempt + still rolls back to the ORIGINAL pre-batch cursor state rather than + compounding on attempt 2's (already-restored) state. + """ + reg, qm = reg_qm + sid = "s-phantom-retry-chain" + worker = SessionWorker( + session_id=sid, + workspace="/ws", + services=HookStateService(workspace="/ws"), + ) + + real_flush = worker.services.graph.flush + call_count = 0 + + async def _flaky_flush() -> None: + nonlocal call_count + call_count += 1 + if call_count <= 2: # two transient failures, then succeed + raise RuntimeError("DeadlockDetected") + await real_flush() + + worker.services.graph.flush = _flaky_flush # type: ignore[method-assign] + reg._register_for_test(worker) + + await qm.append( + sid, + _line( + "provider:request", + "/ws", + {"session_id": sid, "timestamp": "2026-06-11T12:00:00+00:00"}, + ), + ) + task = asyncio.create_task(reg.drain_worker(worker, flush_timeout=10.0)) + for _ in range(400): + await asyncio.sleep(0.01) + if (await qm.read_batch(sid, 10)).lines == []: + break + task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await task + + # Prove all three attempts (2 failures + 1 success) actually ran, + # well within the default max_delivery_attempts=5 budget. + assert call_count >= 3, f"Expected >=3 flush calls, got {call_count}" + + iteration_nodes = sorted( + node_id + for node_id in worker.services.graph._nodes + if "::iteration::" in node_id + ) + assert iteration_nodes == [f"{sid}::iteration::1"], ( + f"Expected exactly one Iteration node, got: {iteration_nodes}" + ) + assert worker.services.data_layer_2.iteration_count == 1, ( + "iteration_count cursor must not be advanced beyond the single " + "committed iteration; got " + f"{worker.services.data_layer_2.iteration_count}" + ) diff --git a/tests/test_services.py b/tests/test_services.py index 7b37014b..8cc06f4f 100644 --- a/tests/test_services.py +++ b/tests/test_services.py @@ -2,16 +2,15 @@ from __future__ import annotations -import pytest from unittest.mock import AsyncMock, patch +import pytest from context_intelligence_server.services import ( GraphState, HookConfig, HookStateService, ) - # --------------------------------------------------------------------------- # HookConfig tests # --------------------------------------------------------------------------- @@ -659,6 +658,186 @@ async def test_started_at_set_correctly( assert node.get("started_at") == expected_started_at +# --------------------------------------------------------------------------- +# I1: ensure_session_node lifts working_dir onto Session +# --------------------------------------------------------------------------- + + +class TestEnsureSessionNodeWorkingDir: + """ensure_session_node sets Session.working_dir from data['working_dir'] when present. + + Forward-only: absent/empty working_dir leaves the property unset (null on read). + """ + + async def test_working_dir_set_when_present(self) -> None: + """data['working_dir'] must be copied onto the new Session node.""" + svc = HookStateService() + await svc.ensure_session_node( + "sess-wd-present", + { + "timestamp": "2026-01-01T00:00:00Z", + "working_dir": "/home/user/my-project", + }, + ) + node = await svc.graph.get_node("sess-wd-present") + assert node is not None + assert node.get("working_dir") == "/home/user/my-project", ( + f"working_dir must be copied from data. Got: {node!r}" + ) + + async def test_working_dir_absent_leaves_property_unset(self) -> None: + """Missing data['working_dir'] must leave the Session node without the property.""" + svc = HookStateService() + await svc.ensure_session_node( + "sess-wd-absent", + {"timestamp": "2026-01-01T00:00:00Z"}, + ) + node = await svc.graph.get_node("sess-wd-absent") + assert node is not None + assert node.get("working_dir") is None, ( + f"working_dir must be absent/null when not supplied. Got: {node!r}" + ) + + async def test_working_dir_empty_string_leaves_property_unset(self) -> None: + """Empty-string data['working_dir'] must NOT be written (falsy guard, matches workspace).""" + svc = HookStateService() + await svc.ensure_session_node( + "sess-wd-empty", + {"timestamp": "2026-01-01T00:00:00Z", "working_dir": ""}, + ) + node = await svc.graph.get_node("sess-wd-empty") + assert node is not None + assert node.get("working_dir") is None, ( + f"empty-string working_dir must not be written. Got: {node!r}" + ) + + +# --------------------------------------------------------------------------- +# working_dir populate-if-missing on the ALREADY-EXISTS path. +# +# The gap this closes: ensure_session_node's already-exists branch used to +# upsert a fixed stub ({labels, status, session_id}) and return early, +# meaning re-importing an existing local JSONL session (e.g. via the upload +# CLI) never populated working_dir on a Session node that predates the +# field. Each test below uses two HookStateService instances sharing one +# GraphState to simulate a fresh process (cold _seen_sessions cache) +# re-encountering a session that already has a Session node in the graph. +# --------------------------------------------------------------------------- + + +class TestEnsureSessionNodeWorkingDirPopulateIfMissing: + """working_dir is populate-if-missing on the already-exists path too.""" + + async def test_fills_null_working_dir_from_later_event(self) -> None: + """Existing node with working_dir null + later event with working_dir -> filled.""" + graph = GraphState() + creator = HookStateService(graph_store=graph) + await creator.ensure_session_node( + "sess-wd-fill-gap", {"timestamp": "2026-01-01T00:00:00Z"} + ) + node = await graph.get_node("sess-wd-fill-gap") + assert node is not None + assert ( + node.get("working_dir") is None + ) # sanity: gap exists before the fix path runs + + # Fresh service instance == cold _seen_sessions cache, same underlying graph. + reimporter = HookStateService(graph_store=graph) + await reimporter.ensure_session_node( + "sess-wd-fill-gap", + { + "timestamp": "2026-02-01T00:00:00Z", + "working_dir": "/home/user/my-project", + }, + ) + node = await graph.get_node("sess-wd-fill-gap") + assert node is not None + assert node.get("working_dir") == "/home/user/my-project", ( + f"working_dir must be filled in on the already-exists path. Got: {node!r}" + ) + + async def test_does_not_clobber_already_populated_working_dir(self) -> None: + """Existing node with working_dir='/x' + later event with working_dir='/y' -> stays '/x'.""" + graph = GraphState() + creator = HookStateService(graph_store=graph) + await creator.ensure_session_node( + "sess-wd-no-clobber", + {"timestamp": "2026-01-01T00:00:00Z", "working_dir": "/x"}, + ) + + conflicting = HookStateService(graph_store=graph) + await conflicting.ensure_session_node( + "sess-wd-no-clobber", + {"timestamp": "2026-02-01T00:00:00Z", "working_dir": "/y"}, + ) + node = await graph.get_node("sess-wd-no-clobber") + assert node is not None + assert node.get("working_dir") == "/x", ( + f"an already-populated working_dir must never be overwritten. Got: {node!r}" + ) + + async def test_empty_incoming_value_does_not_clear_existing_value(self) -> None: + """Existing node with working_dir='/x' + later event with working_dir='' -> stays '/x'.""" + graph = GraphState() + creator = HookStateService(graph_store=graph) + await creator.ensure_session_node( + "sess-wd-empty-no-clear", + {"timestamp": "2026-01-01T00:00:00Z", "working_dir": "/x"}, + ) + + empty_event = HookStateService(graph_store=graph) + await empty_event.ensure_session_node( + "sess-wd-empty-no-clear", + {"timestamp": "2026-02-01T00:00:00Z", "working_dir": ""}, + ) + node = await graph.get_node("sess-wd-empty-no-clear") + assert node is not None + assert node.get("working_dir") == "/x", ( + f"empty incoming working_dir must never null out an existing value. Got: {node!r}" + ) + + async def test_absent_incoming_value_leaves_absent_value_absent(self) -> None: + """Existing node with no working_dir + later event without working_dir -> stays absent.""" + graph = GraphState() + creator = HookStateService(graph_store=graph) + await creator.ensure_session_node( + "sess-wd-stay-absent", {"timestamp": "2026-01-01T00:00:00Z"} + ) + + later = HookStateService(graph_store=graph) + await later.ensure_session_node( + "sess-wd-stay-absent", {"timestamp": "2026-02-01T00:00:00Z"} + ) + node = await graph.get_node("sess-wd-stay-absent") + assert node is not None + assert node.get("working_dir") is None, ( + f"working_dir must remain absent when no event ever supplies one. Got: {node!r}" + ) + + async def test_fill_is_idempotent_no_op_on_repeat(self) -> None: + """Once filled, a repeat of the same working_dir is a no-op (still fills correctly).""" + graph = GraphState() + creator = HookStateService(graph_store=graph) + await creator.ensure_session_node( + "sess-wd-idempotent", {"timestamp": "2026-01-01T00:00:00Z"} + ) + + filler = HookStateService(graph_store=graph) + await filler.ensure_session_node( + "sess-wd-idempotent", + {"timestamp": "2026-02-01T00:00:00Z", "working_dir": "/x"}, + ) + + repeat = HookStateService(graph_store=graph) + await repeat.ensure_session_node( + "sess-wd-idempotent", + {"timestamp": "2026-03-01T00:00:00Z", "working_dir": "/x"}, + ) + node = await graph.get_node("sess-wd-idempotent") + assert node is not None + assert node.get("working_dir") == "/x" + + @pytest.mark.asyncio async def test_graphstate_discard_buffer_is_noop(): """GraphState.discard_buffer is a no-op: must not raise, must not drop data.""" diff --git a/uv.lock b/uv.lock index 06f8fbaf..705457a3 100644 --- a/uv.lock +++ b/uv.lock @@ -233,7 +233,7 @@ wheels = [ [[package]] name = "context-intelligence-server" -version = "6.7.0" +version = "6.8.0" source = { editable = "." } dependencies = [ { name = "aiofiles" },