[stacked on #2387] bb.db retention: archived events, provider/unhandled cap, settled-interaction hygiene - #2393
Closed
vburojevic wants to merge 10 commits into
Closed
Conversation
The timeline response cache is keyed by maxSeq, so every appended event makes it cold, and it deliberately refuses to store windows over 200 rows (the streaming case). With several clients watching one streaming thread, each self-paced refetch triggered its own synchronous 130-260ms rebuild, and the rebuilds serialized on the event loop under every other request. Add a per-params-key last-build slot to the timeline cache: inside a 250ms share window, requests at the same key share the one completed build whatever its row count, and requests at a newer maxSeq for an over-cap window get the prior window back (a rebuild floor). The floor deliberately skips LRU-cacheable windows - those already coalesce at the same key via the LRU, and their rebuilds stay eager - and never crosses params keys, so a status flip (interrupt, completion) is never floored. The route now records the latest-rows snapshot under the response's own maxSeq rather than the route-computed one, so a floored response still names exactly the revision its rows reflect and delta bookkeeping stays coherent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
maybePruneActiveThreadEventHistory ran synchronously inside POST /internal/session/events - the hottest handler - every ~30s per active thread, and it was not wrapped in event-loop-work, so the stall monitor could not name it when it exceeded a second. Two of its DELETEs also walk their candidate rows with correlated subqueries even when the thread has nothing prunable, which is the common case. Move the prune into the existing deferEventFollowUpBatch as an "active-thread-event-prune" follow-up, wrapped in runEventLoopWorkSync so stalls are attributable to `event-prune <threadId>`. Pruning is output-preserving and never lowers maxSeq, so deferral only changes when redundant rows disappear, never what any read returns - the updated ingest test pins both the post-response ordering and that a timeline read between append and deferred prune projects identical rows. Cheapen the no-op case with a LIMIT 1 necessary-condition probe per walking prune class (usage CTEs, resolved item deltas, background-task progress): one covering (thread_id, type, sequence) index seek decides whether the DELETE can possibly match before it walks anything. The plain range-delete class needs no probe - it already is one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…chinery The app broadcast paths (notifyClientsByKeySet, notifyThreadListOnlySockets, notifyThreadOpen, notifyThreadPaneAction, notifyPluginSignal) looped bare socket.send: no bufferedAmount check, no queue cap, and no try/catch, so a suspended phone's socket buffer grew without bound into server RSS and one throwing socket truncated delivery to every subscriber after it. Generalize the terminal queue/drain/drop machinery into per-lane sendWithBackpressure: send below the socket high water, queue and poll-drain above it, and drop-and-close the socket (1013) when a send throws or the queue budget overflows. The terminal lane keeps its 1 MiB / 32 MiB limits and drop semantics verbatim; the new app lane uses 1 MiB / 4 MiB - app messages are small invalidation hints, and a socket that cannot flush them is wedged. Dropping an app socket unregisters it first so the rest of the fan-out and all later ones skip it; clients already reconnect and catch up via the watermark. Broadcast counts now report sockets that accepted (sent or queued) the signal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bb.realtime.publish serialized every payload three times on the event loop (stringify to check serializability, parse back, zod-parse the envelope, stringify the frame), had no size bound, and no rate bound - a runaway in-process publisher multiplied by every connected client. bb.log did mkdirSync + statSync + appendFileSync per line on the loop. publish now serializes the payload exactly once; that string is the size-cap measurement (64KB, rejected with a clear error) and the exact bytes on the wire - the hub validates the envelope fields through the strict outgoing schema and embeds the payload verbatim instead of re-parsing it. A per-plugin token bucket (10/s, burst 20) drops signals over the limit - they are ephemeral by contract - warning once per 10s window with the plugin id. bb.log now buffers lines and flushes them as one async appendFile per 200ms window or 8KB, creates the log directory once per writer, and tracks rotation from a cached size counter seeded by a single stat. The tail path flushes before reading so it stays read-your-writes, and the plugin API handle flushes and drops the writer in its dispose hooks so a pending flush timer cannot race a plugin data-dir removal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GET /threads and the projects-with-threads bootstrap ran unbounded list queries: the route accepted an optional limit nobody passed, and the per-projects variant had no limit parameter at all, so every sidebar bootstrap and list read scaled with a long-lived install's full thread history. listThreadsWithPendingInteractionState now defaults its limit to DEFAULT_THREAD_LIST_LIMIT (200) and the ForProjects variant gains the same defaulted limit across all listed projects; both order by the active-sidebar order, so the cap keeps the most relevant rows. The GET /threads route fetches one probe row past the effective limit and reports truncation in an x-bb-thread-list-has-more response header - the response body stays a bare array for compatibility - so clients can page with limit/offset once they hit the cap. Known behavior change, deliberate: a `bb thread list` or unpaged SDK list call now returns at most 200 rows (the two in-repo programmatic consumers already pass explicit limits and page). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Settled (resolved/interrupted) pending_interactions rows and prompt_history_entries had no retention: approval payloads (diffs, command text) and a second copy of every prompt accumulated forever. - pruneSettledPendingInteractions deletes settled rows created more than 30 days ago, pinned to pending_interactions_status_created_idx, bounded per pass; pending/resolving rows are never touched. - capPromptHistoryEntries keeps the newest 200 entries per (thread, scope) pair - four times the largest read window - deleting in exact read order, bounded per pass by a scope batch. - Both registered as retention sweep jobs next to closed-session-prune; the prompt-history cap runs on a 15-minute cadence because its over-cap probe walks the table grouped. Retention policy (30 days / 200 entries) is a product decision surfaced in the constants' doc comments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The events table had no lifetime retention for archived threads: only a few prunable event classes were trimmed on archive, so every item/completed, turn/*, and system/* row lived as long as its thread and bb.db grew monotonically with total work ever done (item/completed alone: 292 MB / 167k rows on the measured 916 MB live copy). pruneArchivedThreadEvents hard-deletes archived, non-deleted threads' events beyond the caller's keep-recent window - the same ARCHIVED_THREAD_EVENT_KEEP_RECENT = 120 window the on-archive prune already applies to its prunable classes, now owned and passed in by the server. The walk is a durable keyset cursor over thread ids in maintenance_scan_cursors (dimensions unused by this policy store empty strings; the cursor table's columns are documented as per-policy), bounded per pass by a thread batch and a total row budget, resuming mid-thread when the budget cuts a delete and wrapping after a full cycle so newly archived threads are picked up. Unarchiving stops future pruning; rows already pruned stay gone - the same contract the on-archive prune implies. Registered as a retention sweep job on a one-minute cadence. A server test verifies the timeline of a pruned archived thread still projects from the kept window (the same partial-history shape the timeline event budget already produces). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hival provider/unhandled rows are raw provider events bb could not translate, persisted for diagnostics (41k rows / 44 MB on the measured live copy). Verified read paths before choosing delete semantics: stored rows are read back only by the diagnostics timeline path (development builds or the showUnhandledProviderEvents setting, default off) and by the legacy claude-code model-fallback extraction for rows persisted before the typed provider/modelFallback event existed. Nothing replays them to a provider; session resume does not consult them. An age cap therefore only trades away historical diagnostic rows (and legacy fallback banners older than the window). pruneProviderUnhandledEvents deletes rows older than 30 days oldest-first, bounded per pass, pinned to a new tiny partial index events_provider_unhandled_created_idx (created_at, id) WHERE type='provider/unhandled' - required by the new delete, holding only retained rows of this one type. Migration generated with Drizzle (0108_provider_unhandled_retention_index) and made idempotent with IF NOT EXISTS per the convention of 0106/0107. Registered as a retention sweep job on a one-minute cadence. The 30-day window is a product decision surfaced in the constant's doc comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stat scan Two cheap visibility/ordering fixes for legacy auto_vacuum=NONE databases (the measured live DB is already INCREMENTAL, so these matter for other installs): - initDb logs the resolved auto_vacuum mode plus the O(1) freelist counters at startup, so an operator can see which compaction regime a database is in without waiting for the hourly maintenance sweep. - The non-incremental maintenance branch previously always ran getDatabaseCompactionStats, whose dbstat scan walks every page of the file (seconds of synchronous event-loop work on a multi-hundred-MB database). getDatabaseCompactionStatsFreelistFirst decides from the freelist alone when it already crosses the compaction thresholds - the dbstat unused bytes could only add - and only falls through to the full scan when the freelist cannot decide. The call is wrapped in runEventLoopWorkSync so stall snapshots attribute the scan. A maintenance test proves the ordering with a prepared-SQL spy: no dbstat statement is prepared on the short-circuit path, and the fallthrough path still computes freelist + unused. The plan's 'bb db compact' operator command was deliberately dropped from this branch: the live database is already auto_vacuum=INCREMENTAL with a zero freelist, so the unreachable-full-VACUUM concern it addressed is moot there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The server database maps/caches ~1.25 GiB by construction (256 MiB page cache + 1 GiB mmap), which dominates server RSS on large databases and makes RSS unreadable as a signal. This makes both budgets operator-tunable without changing the defaults: - BB_SQLITE_CACHE_SIZE_KIB overrides the page-cache budget in KiB. - BB_SQLITE_MMAP_SIZE_BYTES overrides the mmap window in bytes (0 disables memory mapping). Values are resolved per connection from the environment; unset, empty, or malformed values fall back to the defaults so a bad knob can never prevent the database from opening. Documented in docs/configuration.md, the bb-cli skill, and the CLI guide template, and registered as startup-only managed env keys in the bb-app launcher so 'bb-app env' reports them correctly. Default tuning is deliberately deferred: measure RSS after the retention sweeps shrink the file, then revisit with numbers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 25, 2026
Collaborator
|
Closing this PR for rework after an adversarial review of the #2385–#2393 set. Method: two independent multi-agent review passes over a worktree at this PR head; every finding went to three independent refuters (code trace, a throwaway experiment against the real code, and an impact judge), and only findings that survived at least two of three are listed. The branch stays as-is; please reopen this PR or open a new one when the blocking items are addressed. This PR is stacked on #2387, which is also closed for rework. Blocking — all in
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
STACKED PR — contains #2387 (server event loop); only the top 5 commits are new. Do not merge before it.
What was wrong
bb.db grows monotonically (916 MB live, ~586 KB/h at near-idle) because the
eventstable has no lifetime retention: measured on a copy, it is ~91% of the file (item/completed 292 MB/167k rows, item/started 84 MB, provider/unhandled 44 MB/41k rows). Only a few streaming event classes were ever pruned; every other row lives as long as its thread. Two smaller tables (pending_interactions,prompt_history_entries) also never delete rows — cheap hygiene, not growth drivers (<1 MB each). The non-incremental maintenance branch also ran a whole-file dbstat scan before deciding anything, and the ~1.25 GiB SQLite cache+mmap budget was not tunable.What changed
Retention sweeps (all bounded per pass, registered in periodic-sweeps):
pruneArchivedThreadEvents): hard-delete events beyond the existingARCHIVED_THREAD_EVENT_KEEP_RECENT = 120window. Durable keyset cursor inmaintenance_scan_cursors; resumes mid-thread under the row budget; unarchiving stops future pruning.pruneProviderUnhandledEvents): 30-day age cap regardless of archival. Read paths verified first: stored rows feed only the diagnostics timeline (dev builds /showUnhandledProviderEvents, default off) and legacy claude-code model-fallback extraction; nothing replays them. New partial index via Drizzle migration0109_provider_unhandled_retention_index(regenerated snapshot, no hand edits).(thread, scope).initDblogs the resolvedauto_vacuummode + freelist at startup; the O(1) freelist counters short-circuit the whole-file dbstat scan when they alone cross the compaction thresholds, wrapped inrunEventLoopWorkSync.BB_SQLITE_CACHE_SIZE_KIB/BB_SQLITE_MMAP_SIZE_BYTESenv overrides (defaults unchanged), documented in docs + bb-cli skill surfaces.Retention policy stated, not buried: archived threads keep 120 recent events; provider/unhandled diagnostics live 30 days; settled approval payloads 30 days; prompt history 200 per thread+scope. Active-thread
item/completedretention is deliberately NOT included — the review notes recommend a second-stage output-payload truncation as the next step instead of row deletion (product decision).How you verified
New db tests (in-memory SQLite +
migrate(db)) incl. row-budget mid-thread resume, cursor wrap, oldest-first age cap, pending/resolving never touched, prepared-SQL spy proving the freelist short-circuit prepares no dbstat statement, env-override fallbacks. Server test: a pruned archived thread's timeline still projects from the kept window. Fail-before/pass-after proven by stubbing each prune to a no-op (3+1 archived failures, 2 provider/unhandled). Rebased onto current origin/main over #2129/#2346 with migration renumbered via drizzle-kit:@bb/db423,@bb/server2,033, all green; measurements were taken on a copy of the live DB only.