feat(analytics): a per-document retrieval trail, keyed by trace id

[dimittal]

Stacked on #20 — based on feat/search-contract so this diff shows only the audit work. Merge #20 first and this retargets to main cleanly.

Analytics recorded one row per read: the question, never the answer. That tells you a search happened and returned nine things, which is no help when an agent then does something odd on the back of it.

One row per document returned

retrieval_results holds rank, score, keyword score, semantic score, and why it matched — so the answerable question becomes "this turn retrieved that document, third, on a semantic match at 0.71".

2 read(s) under turn-42
  search  payment  (2.63 ms, via cli)
     1. issue:payment-declined        both      score=0.937
     4. issue:slow-checkout           semantic  score=0.238
  get_entity  issue:threed-secure-timeout
     1. issue:threed-secure-timeout   lookup

Two details that decide whether the trail can be trusted:

• Rank is the order the caller received, not score order. That is what the agent saw.
• Only rows that survived confidence and validity filtering are recorded. A document dropped on the way out would otherwise look like it was handed over.
• Lookups are recorded too. A trail showing what was searched but not what was then opened is half a trail.

trace_id, bound at the edges

HTTP	X-Trace-Id header, bound by middleware for the whole request and echoed back so a caller can confirm it took
MCP	an explicit trace_id argument, because a tool has no ambient request to read

A ContextVar carries it — per-task and per-thread, so concurrent requests never see each other's id, and the context manager restores on exit so a reused worker cannot credit the next caller's retrievals to the previous one.

Nothing generates an id. An absent one is recorded as NULL; inventing one would create ids that correlate nothing. Malformed ids are dropped rather than sanitised — a mangled id silently fails to match what the caller believes it sent, which is worse than having none.

Both directions

• by_trace — what did this turn retrieve?
• retrievals_of — what keeps retrieving this document?

Surfaced through open-index trace <id>, a lookup box on the Analytics tab, and a "what retrieved this" panel on every entity that links back to the trace.

Also here

A state volume, and the directory pre-created in the image with the right ownership. The trail lives in the container's state directory, so without this every redeploy silently discarded exactly the history you would want while debugging one. Same failure shape as the model-cache volume already documented in the Dockerfile.

Retention. Per-result rows multiply volume by the page size. Oldest reads go first with their documents, so a trace is wholly present or wholly gone rather than surviving as a read whose documents were collected out from under it. OPEN_INDEX_ANALYTICS_MAX tunes it; 0 disables it for a deployment shipping the file somewhere durable.

The navigation guide now covers mode, filters and trace_id. It is injected into the MCP handshake, so that is where agents will actually read it — including "read match before trusting a result: a semantic-only hit at a low score is a guess, not a fact."

A bug the coverage gap caught

One of my patches used str.replace, which hit both store classes — duplicating the new methods into NullAnalyticsStore, where they referenced a connection that does not exist. That fallback stands in when the state directory is unwritable, so a broken method there would have failed on precisely the machine where analytics were already degraded. There is now a test asserting the null store mirrors the real store's public surface.

I only noticed because coverage claimed methods were unexercised while the tests asserting their results were passing — the two facts could not both be true.

583 tests pass. tracing.py 100%, analytics.py 99% — the single miss is an unreachable defensive guard, left rather than covered by an artificial test.

🤖 Generated with Claude Code