Enable Workers Logs (incident J-002 follow-up)#17
Merged
Conversation
Adds [observability] block to top-level wrangler.toml and to [env.staging], turning on Workers Logs (per-invocation logs, exceptions, CPU/wall time, subrequest counts) with full-rate head sampling and invocation_logs. Direct response to incident J-002 (transient 503s on 2026-04-22T00:17:02Z), which was un-investigable because Workers Logs was not enabled. Without this block, edge-injected 5xx responses cannot be correlated to a runtime cause — the application code never returns 503 itself, so every 503 the orchestrator sees is by definition a Cloudflare-injected response with no application signal. Also appends J-002 to odd/ledger/journal.md per the project convention that every PR carries an OLDC journal entry. The new entry covers Observations from the incident replay, Learnings about the application 503 ceiling, the Decision to enable logging now (this PR), Constraints (logs cannot be enabled retroactively), and four Handoffs including this PR (H8) and an encoder-defect note (H7) for separate follow-up.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
aquifer-mcp | 2059ac0 | Commit Preview URL Branch Preview URL |
Apr 23 2026, 04:36 AM |
This was referenced Apr 23, 2026
klappy
added a commit
to klappy/klappy.dev
that referenced
this pull request
Apr 24, 2026
Graduates the paired pattern observed across aquifer-mcp J-002 -> H11b (PRs klappy/aquifer-mcp#17 through #20) as tier-2 canon. Three deciding-argument recurrences under the manual-enforcement reading: 1. PR #18 BootstrapEntityResult + 9 Bugbot findings (3 silent-truth-loss bugs). 2. PR #20 FanOutEntityResult + High-severity ctx ReferenceError. 3. esbuild-transpile-only boundary condition — orchestrator decision to treat adversarial review as sole defense when pipeline bypasses types. Release-validation-gate: Bugbot clean (both HEADs); independent Sonnet 4.6 validator dispatched; all findings dispositioned in closeout comment #136 (comment)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Direct response to incident J-002 (transient 503s on
aquifer.klappy.dev/mcpat 2026-04-22T00:17:02Z, single trace3daca3e0-1024-4733-93af-510d65a8b7ae). Investigation 28 hours later established that:status: 500from its only two error paths andstatus: 503from zero. Every 503 the orchestrator sees from this service is, by mechanism, a Cloudflare-edge-injected response (Error 1101/1102, R2 transient, isolate cold-start collision, or upstream Cloudflare service blip) — not application logic.wrangler.tomlhad no[observability]block, so Workers Logs was never ingesting per-invocation logs / exceptions / CPU and wall time / subrequest counts. The data class that would have distinguished those four hypotheses for the J-002 incident does not exist and cannot be reconstructed.This PR closes that visibility gap so the next 503 is a one-query investigation instead of a four-hypothesis exploration.
What
wrangler.toml:[observability]block,enabled = true,[observability.logs]withinvocation_logs = trueandhead_sampling_rate = 1.[env.staging]for parity withwrangler dev --env stagingand the staging branch preview atstaging-aquifer-mcp.klappy.workers.dev.odd/ledger/journal.md:oddkit_encodedid not honor DOLCHE splitting on the planning session that produced this entry; collapsed five-section payload into one Constraint stub. Recorded for separate investigation against theoddkitworker.getarguments as{compound_key: "..."}instead of three separate fields; investigate tool-description presentation layer.Validation after merge
Workers Builds will redeploy on push to
main. Within ~1 hour of the deploy, the Workers Logs view should populate:Quick sanity check:
wrangler tail --name aquifer-mcp --format prettyshould stream live invocation events; previously this returned no events because no telemetry was being captured.Risk
Low. The change is purely additive — no code changed, no bindings changed, no compatibility flags changed. Cloudflare Workers Builds will redeploy the Worker on merge (the deployed bundle is bytes-identical to the current
2026-04-13T21:35:14Zdeploy plus the observability config flag). Athead_sampling_rate = 1the maximum possible log ingest cost is 1 row per invocation — within the Workers Paid free allotment for this service's traffic level.Mode trail
Investigation conducted under exploration → execution per the project model operating contract. J-002 entry was hand-written rather than copied verbatim from
oddkit_encodeoutput because the encoder collapsed the OLDC+H payload to a single Constraint stub (see H7); the hand-written entry follows the J-001 narrative precedent already at the bottom of the journal.Note
Low Risk
Low risk: configuration-only change enabling Workers Logs for production and staging; no runtime code paths, bindings, or caching behavior are modified.
Overview
Enables Cloudflare Workers invocation logs for
aquifer-mcpby adding[observability](and mirrored[env.staging.observability]) towrangler.toml, so transient edge-injected 5xxs can be investigated with per-invocation telemetry.Appends a new journal entry
J-002toodd/ledger/journal.mddocumenting the 2026-04-22 transient 503 incident and the decision to enable Workers Logs as the follow-up action.Reviewed by Cursor Bugbot for commit 2059ac0. Bugbot is set up for automated code reviews on this repo. Configure here.