feat: add periodic WAL checkpoint to idle maintenance#181
Merged
Conversation
…ends Closes #180 SQLite WAL files grow unbounded in long-running deployments because durably never explicitly checkpoints. Add PRAGMA wal_checkpoint(TRUNCATE) to the idle maintenance cycle, throttled to at most once per 60 seconds. - Only runs for local SQLite backends (skips PostgreSQL and browser/OPFS) - Checks the busy flag so failed checkpoints retry on the next cycle - Runs outside the write mutex since SQLite handles checkpoint concurrency Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
Contributor
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 4 minutes and 53 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughrunIdleMaintenance に SQLite WAL チェックポイントの周期実行とプローブを追加しました。マイグレーション完了後に非同期で WAL サポートを検出し、汎用バックエンドで定期的に Changes
Sequence Diagram(s)sequenceDiagram
participant Scheduler as Scheduler
participant Durably as Durably
participant Kysely as Kysely (sql)
participant SQLite as SQLite
Scheduler->>Durably: runIdleMaintenance(now)
Durably->>Durably: calculate nowMs, leases/purge checks
alt probe not yet run
Durably->>Kysely: sql`PRAGMA wal_checkpoint(PASSIVE)` (probe)
Kysely->>SQLite: execute PRAGMA
SQLite-->>Kysely: success / error
Kysely-->>Durably: probe result -> set walCheckpointSupported (or ignore error)
end
alt walCheckpointSupported && interval passed
Durably->>Kysely: sql`PRAGMA wal_checkpoint(TRUNCATE)`
Kysely->>SQLite: execute PRAGMA
SQLite-->>Kysely: {busy: 0/1}
Kysely-->>Durably: checkpoint result
Durably->>Durably: update lastCheckpointAt (throttle / retry timing)
Durably->>Durably: emit worker:error if SQL error (context: "wal-checkpoint")
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/durably/src/durably.ts`:
- Around line 993-1000: The checkpoint logic currently only updates
lastCheckpointAt when row?.busy === 0, causing repeated immediate retries during
long read transactions; change the behavior in the block that runs PRAGMA
wal_checkpoint(TRUNCATE) so that lastCheckpointAt is advanced to nowMs
regardless of row.busy (and still perform the checkpoint when busy === 0),
thereby enforcing CHECKPOINT_INTERVAL_MS throttling; locate the code around
PRAGMA wal_checkpoint(TRUNCATE), lastCheckpointAt, CHECKPOINT_INTERVAL_MS and
row?.busy to make this change.
- Around line 966-967: The current check uses backend === 'generic' to set
shouldWalCheckpoint which enables SQLite-specific PRAGMA
wal_checkpoint(TRUNCATE) for non-Postgres backends and incorrectly runs on
non-SQLite dialects; change the condition in the shouldWalCheckpoint logic to
narrowly detect SQLite (e.g., inspect the exported dialect value or a SQLite
capability) instead of relying on DatabaseBackend/generic — use the Dialect
parameter or an explicit isSqliteDialect/isSqliteCapability check and keep the
isBrowserLikeEnvironment() guard so PRAGMA wal_checkpoint only runs when the
dialect is actually SQLite.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e764f7a7-72d4-4f53-bdfa-6e0188919c69
📒 Files selected for processing (1)
packages/durably/src/durably.ts
Remote backends like Turso reject PRAGMA wal_checkpoint with SQL_PARSE_ERROR. Instead of guessing from dialect type, probe with PRAGMA wal_checkpoint(PASSIVE) during migrate() — if it succeeds, enable periodic TRUNCATE checkpoints; if it throws, silently disable. This correctly handles all backends: - Local better-sqlite3: probe succeeds, checkpoints enabled - Local libSQL: probe succeeds (safe no-op), checkpoints enabled - Remote Turso: probe throws SQL_PARSE_ERROR, checkpoints disabled - Browser OPFS: skipped entirely (isBrowserLikeEnvironment check) - PostgreSQL: skipped entirely (backend !== 'generic') Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Checkpoint errors now emit worker:error with context 'wal-checkpoint' instead of being silently swallowed, preserving operational visibility - Null out probeWalCheckpoint after migrate() to prevent stale references - Busy checkpoints retry after 30s (half interval) instead of every tick Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/durably/src/durably.ts`:
- Around line 983-1006: The checkpoint timing uses the earlier nowMs (used for
releaseExpiredLeases/purge) which may be stale by the time the checkpoint runs;
before checking/performing the WAL checkpoint (the block using
CHECKPOINT_INTERVAL_MS, lastCheckpointAt and the sql`PRAGMA
wal_checkpoint(TRUNCATE)` call), recompute the current time into a new variable
(e.g., checkpointNow = Date.now()), use checkpointNow for the throttle check
(checkpointNow - lastCheckpointAt), assign lastCheckpointAt = checkpointNow when
starting the checkpoint attempt, and when applying the busy backoff set
lastCheckpointAt = checkpointNow - CHECKPOINT_INTERVAL_MS/2 so the backoff
reflects the actual attempt time; leave releaseExpiredLeases and purgeRuns using
the original nowMs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 3d21510c-23e7-44fc-8bdf-90665b00476d
📒 Files selected for processing (1)
packages/durably/src/durably.ts
Check probe result: log === -1 indicates WAL is not active (e.g. libSQL local uses DELETE journal mode). Only enable periodic checkpoints when the probe confirms WAL is in use, avoiding no-op PRAGMA queries. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #180
PRAGMA wal_checkpoint(TRUNCATE)torunIdleMaintenance, throttled to at most once per 60 secondsisBrowserLikeEnvironment())busyflag from checkpoint result — retries on next cycle if SQLite reports busybusy=1instead)Test plan
pnpm validatepasses (244 tests, 28 tasks)/simplifyreview — removed Store interface leak, cached environment check, shared timestamp🤖 Generated with Claude Code
Summary by CodeRabbit
リリースノート
Chores
Bug Fixes