feat(entity): H11b — partial data with transparency + background warm

[klappy]

Why

Applies the operator's stated principle uniformly: "It's a principle of partial data with transparency to come back for more. Background fetch will warm the cache before you come back."

PR #19 (H11) moved the entity-corpus scan off the query path onto the index-build path. Post-merge production probe showed this didn't actually solve the user-blocking problem:

Observation	Evidence
entity:Obadiah first-visit → HTTP 502 at 11.2s	cf-ray 9f1136be2ab319b0-ORD, no Worker outcome (Cloudflare frontend cut the stream while Worker was still running bootstrap)
entity:Obadiah second-visit → 78 matches in 588ms	R2 bootstrap cache warmed by the first attempt's background-completing scan
All per-resource entities.json blobs → miss	trace: storage:index/AquiferO…/6ebe2335…/entities.j…=104ms(miss) across every resource

The deployed H11 code is present, but populateEntityIndexes never ran — because buildIndex only executes on a composite-SHA change, and the deploy didn't rotate any repo SHAs. Every entity query today still routes through the tier-3 bootstrap fallback, masked by the per-session cache warming my probes accidentally did.

H11 fixed the wrong axis. The durable fix isn't "build the index eagerly instead of lazily." It's: the user path must never pay for a corpus scan regardless of when the scan runs.

What this PR does

src/registry.ts

• Removes await populateEntityIndexes(...) from buildIndex. Index builds return to pre-H11 fast path.
• Changes fanOutEntitySearch return type from Promise<ArticleRef[]> to Promise<FanOutEntityResult>. New exported interface carries matches, complete, scanned_resources, total_resources, and missing_resources: ResourceEntry[].
• Adds exported warmEntityIndexesForResources(resources, repoShas, env, storage) — scans the specified resources' content files with the same 4×8 bounded concurrency caps as J-003's bootstrap, writes per-resource entityIndexKey blobs to R2. Idempotency guarded by a pre-write getJSON check so concurrent warms don't duplicate work.
• Retains populateEntityIndexes as an internal helper (unused in hot path, preserved for potential diagnostic use; deletion tracked as H16).

src/tools.ts

• Both handleEntity and searchByEntity are rewired to:
  1. In-memory short-circuit via index.entity.get(normalized) (tier 1, unchanged).
  2. Call fanOutEntitySearch → unpack FanOutEntityResult.
  3. If missing_resources.length > 0: emit formatPartialFanOutNote(fan) and kick off ctx?.waitUntil?.(warmEntityIndexesForResources(...)). The user's response returns immediately with whatever fan-out found — no corpus scan on the request path.
  4. Return matches (complete or partial).
• Removes bootstrapEntityMatches from both user-facing paths. Function still exported for direct diagnostic use; full deletion is H16 once production observation confirms the new path is healthy.
• New helper formatPartialFanOutNote(r) produces the user-visible disclosure:

  ⚠ Partial result: 4/33 resources indexed, 29 still warming in the background. Retry in a few seconds for the complete result.

odd/ledger/journal.md

• Appends J-005 covering the observation, the H11-wrong-axis learning, the H11b decision, the cache-fetches-and-parses canon constraint (per-resource entity indexes ARE parse products and stay cached — only the trigger moves), and 4 handoffs (H15 post-deploy monitoring, H16 bootstrap deletion, H17 disclosure-note tuning, H18 explicit-retry tool consideration).

Verification

npm ci && npm run build && npm run test with GITHUB_TOKEN set, mirroring CI build-test:

Test Files  4 passed (4)
     Tests  168 passed (168)
  Duration  5.56s

168/168 tests pass (was 165 on main; +3 new H11b-specific tests, 5 pre-existing tests updated because they tested the old bootstrap-fallback path H11b retires). wrangler deploy --dry-run: clean, no binding or compatibility-flag changes.

H11b-specific test coverage:

• ctx.waitUntil is invoked exactly once with a Promise when missing_resources is non-empty
• Absent ctx doesn't throw — warm just skips, self-healing via next query
• Complete fan-out produces no partial note AND no waitUntil call
• Regression: empty fan-out does not invoke mockFetchJson (confirms the bootstrap-free user path)

Vodka check

• warmEntityIndexesForResources and fanOutEntitySearch are generic corpus-scanning / parallel-R2-loading functions. No domain branches.
• FanOutEntityResult fields describe scan completeness, not content structure.
• Zero new if (resource_type === ...) anywhere.
• cache-fetches-and-parses canon upheld: the parse products (per-resource entity indexes) stay cached — only the cache-population trigger changes. SHA-keyed lifecycle unchanged; no staleness risk per anti-cache-lying.

Pressure-tested against canon before writing code

Ran oddkit_challenge on the design before touching any file. Challenge returned block-until-addressed flagging missing disconfirmer + reversibility. Addressed inline:

• Disconfirmer: if post-deploy Workers Logs show clientDisconnected or non-success outcomes on first-visit entity calls, the fan-out itself is too slow — retract.
• Reversibility: fully reversible (git revert). Bootstrap function preserved during migration.
• Prior art: refreshAndUpdateCurrentIndex in registry.ts already uses this pattern for SHA-staleness refresh. H11b is cloning an existing pattern, not inventing one.
• Alternatives rejected: (B) background-warm only when fan-out is fully empty — rejected because partial-fanout is the common case and still blocks user; (keeping H11) — rejected per the evidence above.

oddkit_validate returned NEEDS_ARTIFACTS for session capture + change summary. This PR body + the J-005 journal entry provide both; the commit message provides DoD §5 self-audit detail.

Post-deploy validation plan (H15)

1. Merge this PR. Workers Builds redeploys.
2. Trigger entity entity_id=person:Obadiah (known-cold per pre-deploy probe). First call should return immediately with partial note; background warm kicks off in ctx.waitUntil.
3. Wait ~10 seconds. Retry. Should return complete result with NO partial note — per-resource entity indexes are now populated in R2.
4. Workers Logs outcome distribution for the next 24-48h: target zero exceededMemory, zero exceededCpu, zero edge 502s, no clientDisconnected from user-side timeouts on entity calls.

Mode trail

Investigation → principle extraction → planning (canon search + retrieve + challenge) → execution (code + tests) → validate → artifact (journal + PR + commit message). Claim-as-debt preserved: no post-deploy latency prediction is stated as fact in the journal; J-005 will get an update once H15 observation confirms or refutes the hypothesis.

---

Note

Medium Risk
Changes the entity lookup flow to remove the synchronous corpus-scan fallback and instead schedule background index warms via ctx.waitUntil, which affects user-visible completeness semantics and relies on correct cache-population behavior under load/failures.

Overview
Entity lookups now never run a corpus scan inline. buildIndex stops populating entity indexes, and fanOutEntitySearch switches from returning ArticleRef[] to a structured FanOutEntityResult that includes missing_resources/completeness metadata.

When per-resource entity indexes are missing, handleEntity and searchByEntity return whatever matches are available immediately, append a new disclosure note (formatPartialFanOutNote), and kick off warmEntityIndexesForResources(...) via ctx.waitUntil to populate the missing per-resource entities.json blobs in R2 for the next request.

Tests are updated to reflect the bootstrap-free request path and add coverage for the waitUntil scheduling contract (called once with a Promise when partial; skipped when ctx absent or fan-out complete). Documentation journal entry J-005 is added describing the H11b rationale and rollout/monitoring handoffs.

^{Reviewed by Cursor Bugbot for commit e94e0de. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	aquifer-mcp	e94e0de	Commit Preview URL Branch Preview URL	Apr 24 2026, 02:05 AM

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 739c5c3. Configure here.}