Context lens: durable run history, ship sync, and run-capture hardening#161
Context lens: durable run history, ship sync, and run-capture hardening#161wca4a wants to merge 34 commits into
Conversation
Add richer Context Lens lifecycle states for queued, tool-running, completed, no-reply, timed-out, and error outcomes. Record run timing, queue timing, timeout thresholds, delivery counts, queued reply counts, and tool call counts in the redacted Lens payload. Serialize same-session Tlon dispatches so overlapping inbound messages are visible as queued instead of racing the active run. Add dispatch abort/timeout handling plus configurable run and Tlon CLI tool timeouts through channels.tlon.lifecycle. Update config schema, resolved account types, generated plugin schema, and focused tests for the lifecycle payload and timeout config.
Let the manual Docker gateway target an existing Urbit ship by honoring TLON_URL, TLON_SHIP, TLON_CODE, TLON_OWNER_SHIP, and TLON_DM_ALLOWLIST.\n\nThis keeps the default fake-ship setup intact while allowing local end-to-end tests against the tlon-apps rube ships used by %groups.
Route Context Lens events through the existing shared-state slot so the HTTP routes and lazy runtime monitor see the same recent-event buffer and listeners when OpenClaw loads plugin modules in separate contexts. Add a regression test that publishes through one module instance and looks up the Lens through a separately loaded instance.
Productionize the context lens POC on the gateway side: - contextLens config block (root + per-account): enabled (default off), ttlMs, maxEntries, visibilityDefault, authToken, allowedOrigins - routes extracted to src/context-lens-routes.ts behind a self-managed Bearer gate (timing-safe compare, CORS allowlist, OPTIONS preflight, no unauthenticated mode); SSE keepalive + Last-Event-ID replay - lens recording, blob stamping, and registry storage all no-op unless the lens is enabled with an auth token - registry honors configured TTL/capacity/visibility; update() takes a proper deep-partial patch type; background registry moved to a shared slot so dual module contexts share one instance Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Lets clients resolve the run record when the gateway is unreachable (future ship-relay lookup path). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Persist terminal-status lens runs to a JSONL file under the gateway state dir (configurable via contextLens.store) so /run can resolve lensIds stamped into old posts after a gateway restart. The in-memory event buffer stays the hot path; the store is a last-write-wins backstop with retainDays/maxStored retention and load-time compaction. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Subscribe to the lens event stream and poke %lens-action-1 on the bot ship: %run-event on status transitions, %run-final on terminal status. The agent fans records out to owner ships (contextLens.owners, falling back to ownerShip), making run history durable and mobile-reachable without clients touching the gateway. Owners are configured lazily per monitor connection over a serial poke queue; payloads truncate tool summaries (4KB each, 50KB total) since full runs stay on gateway disk. Ship sync now counts as a reader path, so recording and the disk store run without an authToken when owners resolve. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
%base already bills a %lens agent (the HTTP dojo API) and gall agent names are global across desks, so the ship agent is %context-lens. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The %context-lens agent now stores payloads as cords (embedding $json in mark sample types breaks ford tube builds), so the gateway serializes the run record before poking. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- record tool activity for any session without a conversation lens (cron wakes in the main session previously slipped through via their inherited sender-role entry); tag cron runs from agent-level hooks where the gateway exposes the job id - stamp gateway-delivered sends (cron announcements, CLI sends) with the active background lens and record them as run outputs; widen the finalize idle window so the reply lands before the run closes - attribute tool calls and sender roles across per-peer session key forms and 🧵 suffixed keys - replace (not stack) shared event-bus subscriptions on plugin re-init so reloads don't duplicate ship pokes and store writes Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two independent 60-second killers cleared the computing presence while a run was still live: - The SDK typing callbacks default to a 60s TTL that fires stopRun and seals the callbacks for the rest of the dispatch. Disable it; stopRun is already wired to deliver/idle/cleanup. - The ship's %presence agent expires %computing entries after ~m1, and the tracker's dedupe suppressed the 20s keepalive whenever the state was unchanged, so long model calls never refreshed the ship. Re-publish unchanged active state once it is 30s old. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Raise the typing keepalive failure trip from 2 to 5 so transient poke failures cannot permanently stop refreshes mid-run. - Tombstone recently stopped runIds so a late keepalive tick cannot resurrect a stopped run and leave a stale thinking indicator; real tool activity clears the tombstone and resumes the run. - Drop per-conversation published-state records once thinking:false is published so the maps do not grow unboundedly over gateway uptime. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
One-shot activation could silently hang on a poke that raced an SSE reconnect, leaving gateway-status dead for the process lifetime so the ship marked the gateway %down and auto-replied "bot is offline" to owner DMs. Bound each activation poke with a 15s timeout and retry every 30s until activation sticks or the monitor is torn down. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Record a JSON-pretty-printed argumentDetail (capped at 2000 chars) alongside the existing one-line argumentSummary, and include it in ship-synced run payloads so clients can expand tool calls in the run inspector. Note: argumentDetail carries raw tool params; payloads are owner-visibility only today — needs redaction before any visibility widening. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Auto-fix oxlint/oxfmt issues scoped to files this branch changed: curly braces, unnecessary String() conversions, param shadowing, and consistent-return in the before_tool_call hook. Pre-existing repo-wide issues left untouched to keep the diff reviewable. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0c0b665e80
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| try { | ||
| const computingPresence = createComputingPresenceTracker({ runtime }); | ||
| const contextLensConfig = account.contextLens; | ||
| const contextLensEnabled = contextLensConfig.enabled && Boolean(contextLensConfig.authToken); |
There was a problem hiding this comment.
When Context Lens is configured for ship sync only (for example contextLens.enabled: true plus owners or ownerShip, but no authToken), registerFull() still enables %context-lens ship sync, but the monitor disables its registry because this check requires an auth token. In that supported configuration conversation runs never get stored or published, so the ship-sync subscriber has no events to poke to the ship; use the same effective enablement logic as registration instead of gating recording on the HTTP route token.
Useful? React with 👍 / 👎.
| return; | ||
| } | ||
|
|
||
| unsubscribe = subscribeToContextLensEvents(sendOrClose); |
There was a problem hiding this comment.
For clients connecting to /tlon/context-lens/events, the handler snapshots and replays recent events before adding the live listener. If a run event is published after the snapshot at connection time but before subscribeToContextLensEvents runs, that event is neither in the replay nor delivered live, and the client stays connected missing that sequence until it reconnects; register the listener before/while replaying, or replay again after subscribing to close the gap.
Useful? React with 👍 / 👎.
The monitor gated the run registry on enabled && authToken, so a ship-sync-only config (owners set, no HTTP authToken) silently recorded nothing: no events for the ship sync to mirror and no reference blobs stamped on outbound messages. Gate on the shared effective-enablement instead: enabled && (authToken || resolvable owners), mirroring the index.ts routes/ship-sync OR. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Track the max seq sent during replay and have the live listener drop events at or below it, so the events route cannot double-send if a publish ever lands between the snapshot and the subscription. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a7440b65a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const contextLensRoutesEnabled = registerContextLensRoutes(api); | ||
| const contextLensShipSyncEnabled = initContextLensShipSync(api); |
There was a problem hiding this comment.
When more than one Tlon account is configured, these Context Lens consumers are still registered once using the default account config, while each monitor can publish events for its own accountId and the ship-sync path uses the shared last-writer API params slot. In that scenario, runs from account A can be exposed through account B/default's bearer token or poked to the wrong ship/owners; either disable these global readers for multi-account configs like gateway-status does, or key events/routes/sync by account.
Useful? React with 👍 / 👎.
| error: null, | ||
| createdAt: now, | ||
| updatedAt: now, | ||
| expiresAt: now + (input.ttlMs ?? ttlMs), |
There was a problem hiding this comment.
When contextLens.ttlMs is configured below a legitimate run duration (the schema allows 60s while the default dispatch timeout is 120s), expiresAt is fixed at creation, and later get() calls prune the lens before the final logContextLens(..., "final") publish. Those long-but-valid runs then never reach the durable store or ship sync even though the reply can still include a lens blob; extend expiry on updates or keep active runs until a terminal status is published.
Useful? React with 👍 / 👎.
2 participants
Summary
%context-lensagent (payloads as serialized-JSON cords), which fans out to owner ships for native/client consumption — no gateway reachability required from devices.botShipstamping in the reference blob so clients can resolve runs.argumentDetail): tool calls now carry their raw parameters for the client inspector's expandable tool rows.Dependencies
Part of a three-PR stack; this is the producer end:
%context-lensgall agent): this gateway pokes that agent on the bot ship with finalized runs. Without the agent installed, the pokes nack harmlessly (logged, gateway continues) — run recording, the disk store, the HTTP routes, and message stamping all still work.Security note
argumentDetailships raw tool parameters in run payloads synced to the ship. Today run visibility is owner-only, so this is acceptable, but these payloads must gain redaction/filtering before run visibility is ever widened beyond the owner (secrets, tokens, or file contents can appear in tool args).Test plan
pnpm tsc --noEmitcleanpnpm test:integration, ephemeral fakezods): all suites green (29 passed, 5 skipped)masterbaseline (58 vs 65 pre-existing errors); remaining diffs are type-aware artifacts in untouched files%context-lens, rendered in the Tlon client (wide sidebar, native screens/sheet)🤖 Generated with Claude Code