Repository navigation
Commit e01d347
Fixes #20106
Clause-②: no
`reclaimSpace()` now returns the whole SQLite freelist on every face.
Every reading below comes from a second connection. The file size is
read beside it. Each run has an empty-freelist control. Measured head:
`0adf65deb`.
## What was wrong
SQLite's incremental-vacuum program frees one page per `sqlite3_step()`,
and it yields a result row with no columns for each page. A caller that
steps it once frees one page. Two clients stepped it once:
- **`SqlDriver` on better-sqlite3, and `TursoDriver` in local mode**,
which is the same code. knex's better-sqlite3 client runs a statement
that declares no result columns with `Statement.run()`, and `run()`
steps once. `Statement.reader` is `false` for this pragma.
- **`TursoDriver` in remote mode.** The libSQL client's `execute()`
steps the statement once and leaves it unfinished. The page it freed
never reached the file. The unfinished statement also held the
connection's implicit transaction open, so a later write on that
connection never reached the file either.
## What changed
- `packages/drivers/driver-sql/src/sql-driver.ts`,
`SqlDriver.reclaimSpace`: when the knex client is better-sqlite3, the
pragma runs through that binding's own `exec()` on the pooled
connection. `exec()` steps every statement until SQLite reports done.
Every other SQLite client stays on `knex.raw`. sql.js steps every PRAGMA
to the end in `driver-sqlite-wasm`'s dialect (measured). knex's
node-sqlite3 client uses `Database.all()` (read from knex's source; the
binding is not installed here).
- `packages/drivers/driver-turso/src/turso-driver.ts`, the remote
`reclaimSpace` route: it reads `PRAGMA freelist_count` through the raw
door first. That read also connects the transport lazily, as every
remote door does. When the count is `0` nothing more is sent. Otherwise
the vacuum runs through the client's `executeMultiple()`. Both
statements keep the raw door's envelope (`DATABASE_ERROR` / 500).
- `packages/drivers/driver-turso/README.md`: the sentence on the remote
`reclaimSpace()` now says what the route sends.
- Tests in all three driver packages, and
`.changeset/20106-reclaim-space-full-freelist.md` (`patch` for
`driver-sql` and `driver-turso`).
## Readings: freelist / page count from a second connection
The fixture fills about 300 pages and deletes them. The control is an
empty freelist.
| face | before | `main` (`789b2ae54`) after one call | this PR after
one call |
|:--|:--|:--|:--|
| `SqlDriver` (better-sqlite3, WAL) | 300 / 304 | 299 / 303 | 0 / 4 |
| `SqliteWasmDriver` (sql.js image) | 300 / 304 | 0 / 4, already
complete | 0 / 4 |
| `TursoDriver` local (better-sqlite3, WAL) | 300 / 304 | 299 / 303 | 0
/ 4 |
| `TursoDriver` remote (`@libsql/client`, `file:`) | 300 / 304 | 300 /
304 (the issuing connection read 299) | 0 / 4 |
| control, every face | 0 / 4 | 0 / 4 | 0 / 4 |
File size in bytes:
- **Wasm image and remote `file:` database** (rollback journal):
1,245,184 → 16,384 with this PR. On `main` the remote file stayed at
1,245,184, even after disconnect.
- **The two WAL faces**, after disconnect: `main` 1,241,088, this PR
16,384.
Raw clients, 300 free pages, read from a second connection:
- **better-sqlite3.** `Statement.run('PRAGMA incremental_vacuum')` →
299. `incremental_vacuum(600)` through `run()` → 299 too. `db.pragma()`
and `db.exec()` → 0.
- **`@libsql/client` `file:`.**
- `execute()` → 300. The issuing connection read 299, and the file was
unchanged after `close()`.
- `execute('PRAGMA incremental_vacuum(600)')` → 300.
- `executeMultiple()` → 0.
- `batch([...], 'write')` → throws `SQLITE_BUSY: cannot commit
transaction - SQL statements in progress`.
- **sql.js.** A prepare-and-step loop (the wasm dialect's branch) → 0.
One step → 299.
Remote face, write after `reclaimSpace()`: a `create()` made after the
call is read back by the issuing connection. On `main` a second
connection counted 0 rows, and after disconnect it still counted 0: the
row was lost. With this PR it counts 1, and 1 after disconnect.
Cost at 25,660 free pages through knex on better-sqlite3 (shared box, so
read the ratio):
- `exec()`: 245 ms.
- N single-page `raw()` calls in one transaction: 1,352 ms.
- N autocommitted `raw()` calls: 1,606 ms.
- One `raw()` call (`main`): 1.3 ms, for one page.
## The dispatch's mechanism hypotheses
- **H1**: confirmed on `789b2ae54`: `if (!this.isSqlite) return; await
this.knex.raw('PRAGMA incremental_vacuum');`.
- **H2**: the first half is confirmed: one execution steps once and
frees one page. The second half is falsified. An explicit page count
does not help, because `incremental_vacuum(600)` freed one page through
both better-sqlite3 `run()` and libSQL `execute()`. The count is a
ceiling, not what ends the loop. The statement has to run to completion.
- **H3**: partly falsified. `SqliteWasmDriver` and local `TursoDriver`
do inherit `SqlDriver.reclaimSpace`. But the defect lived in the client
binding, not in the method's text, and the wasm face was already
complete on `main` (its dialect steps every PRAGMA to the end). For this
method there are three independent implementations:
- better-sqlite3 through knex: two faces, both broken, fixed at the
`SqlDriver` seam;
- the sql.js dialect: one face, already correct, now pinned;
- the libSQL remote route: one face, broken, fixed in `turso-driver.ts`.
- **H4**: confirmed, and worse than a reading. Through libSQL
`execute()`, a second connection saw nothing land: not the freelist, not
the file size, and not a later write on the same connection.
## Does `os db clean` reach this method?
No. `packages/cli/src/commands/db/clean.ts` never calls
`reclaimSpace()`. It runs `driver.execute('PRAGMA auto_vacuum =
INCREMENTAL')`, then `driver.execute('VACUUM')`, then disconnects. A
full `VACUUM` finishes in one step. Measured through `SqlDriver.execute`
on a legacy file (`auto_vacuum` 0) with 300 free pages:
- freelist 300 → 0;
- `auto_vacuum` 0 → 2;
- file 1,236,992 → 12,288 bytes.
It is not edited here.
## The remote face: measured and not measured
Measured over a libSQL `file:` client only (`@libsql/client` 0.17.4,
`libsql` 0.5.29), plus a scripted client for the call order and the
refusal envelope.
Not measured, because there is no live server: what a hosted libSQL /
Turso server does. That covers whether it accepts `PRAGMA
freelist_count` and `PRAGMA incremental_vacuum`, and how it steps
`execute()` against `executeMultiple()` (a hrana sequence request). The
route adds one read round trip when there are free pages. When the
freelist is empty it sends no write.
## Tests
- `driver-sql/src/sql-driver-sqlite-reclaim-space.test.ts` (new), 4
cases:
- WAL: freelist 0 and page count equal to before minus free, read from a
second connection; file size equal to pages times page size after close;
- DELETE journal: the same, and the file shrinks while the driver is
open;
- the empty-freelist control;
- the pooled connection is handed back.
- `driver-sqlite-wasm/src/sqlite-wasm-reclaim-space.test.ts` (new), 2
cases: the persisted image, read by a fresh sql.js database, plus the
control.
- `driver-turso/src/turso-remote-inherited-members.test.ts`: the old pin
read the freelist from the issuing client and asserted
`toBeLessThan(before)`, which a one-page, connection-local drop passed.
It now reads a second client and asserts equality. Added:
- the control on both faces;
- the remote file shrinking while open;
- a write after the call landing;
- the call order (count first, nothing more on 0, vacuum through
`executeMultiple`);
- the `DATABASE_ERROR` / 500 envelope when either statement is refused.
Full suites at `0adf65deb`:
- `driver-sql`: 195 files passed, 11 skipped; 3,224 tests passed, 178
skipped.
- `driver-turso`: 73 files; 1,927 tests passed, 16 skipped.
- `driver-sqlite-wasm`: 36 files; 658 tests passed.
- `typecheck` is green for all three. Each package's `tsconfig` includes
its tests; the typecheck read the turso test file and caught an
`Array.prototype.at` before it landed.
Ablations (committed state first; every leg went through
`scripts/ablation-replace.mjs`, and the restore was proven blob-equal to
HEAD):
- **A**: better-sqlite3 routed back to `knex.raw`. The `driver-sql` test
goes red on WAL and DELETE (`{ freelist: 299, pages: 303 }` against `{
freelist: 0, pages: 4 }`); the control and the pool case stay green.
With `driver-sql` rebuilt and the marker proven in `dist/` by
`ablation-dist-preflight`, the turso local face goes red (`{ freelist:
39, pages: 47 }`) and the wasm test stays green (2/2). That is the H3
reading. The restore was rebuilt, and `--absent` proved `dist/` and the
tree clean.
- **B**: the `knex.raw` arm made a no-op, `dist/` rebuilt, marker
proven. The wasm test goes red (`{ freelist: 300, pages: 304 }`) and its
control stays green. The restore was rebuilt and `--absent` passed.
- **C**: the remote route sent back through `execute()`. 5 turso cases
go red: second-connection freelist 40 against 0, the file size case, the
write after the call (0 against 1), the call order, and the vacuum
refusal. The other 77 stay green.
## Gates
`node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` at `0adf65deb` derives 63 commands. All 63
were run, and every exit code was recorded before any pipe: all 63
exited 0.
On the first run at `155234b97`:
- `check:query-options-erasure` was red: the test surface grew 236 → 237
from an `as any` on a query bag in the new tests. The bags are now
typed, and it holds at 236.
- `check:dual-build-cjs-loads`, `check:lean-entry-closure` and
`check:type-check-debt` refused with `PREREQUISITE NOT MET` (exit 3).
They passed after `turbo run build` over `./packages/*` and
`./packages/*/*`.
`--ran`: 63 derived, 63 run, 0 NOT-MEASURED, 0 UNRUN.
`check:driver-conformance`: before, at `789b2ae54`, 50 covered cells and
0 in the DEBT ledger (0 exempt). After, at `0adf65deb`, the same: 50
covered, 0 DEBT.
`pnpm lint` is CI's run. The narrowed run here: `eslint
--no-inline-config --format json` over the 5 changed TypeScript files
reports 5 files, 0 errors and 0 warnings, with no file reported as
ignored. `eslint.config.mjs` sets no `parserOptions.project` and
registers no typed rule, so a diff cannot move the verdict of an
untouched file.
## Acceptance notes
- **The `-wal` sidecar (a finding, reported for the seat to file; not
fixed here).** On a file-backed database in WAL mode, the default, a
to-completion vacuum passes the freed pages through the `-wal` file. At
25,754 free pages through `SqlDriver`:
- the second connection reads freelist 0 and page count 4;
- the database file goes 105,631,744 → 16,384;
- `-wal` goes 4,577,352 → 91,855,432 and keeps that size until the last
connection closes.
Measured options on raw better-sqlite3, 25,600 rows freed, one run on a
shared box (database file + `-wal`, while open):
| option | database file | `-wal` | time |
|:--|:--|:--|:--|
| `exec()` alone (this PR) | 12,288 | 90,112,672 | 267 ms |
| + `wal_checkpoint(TRUNCATE)` | 12,288 | 0 | 309 ms |
| chunked `incremental_vacuum(1000)` + `wal_checkpoint(PASSIVE)` |
12,288 | 210,152 | 71 ms |
A TRUNCATE checkpoint can wait on another process's readers for up to
the busy timeout. PASSIVE never waits.
- **File surface.** `packages/drivers/driver-turso/README.md` is outside
the claim's listed surface. It documents the exact route this PR
changes, and its old sentence ("sends the statement local mode issues")
would have become false.
- **Legacy files.** A file whose `auto_vacuum` is still `NONE` reclaims
nothing through `reclaimSpace()` (measured: 300 → 300), as the method's
comment already says. The lifecycle report still lists that datasource
as reclaimed. `os db clean` is the documented remedy.
- #20107 (the remote read arms) and #20355 (`crossFieldComparisonClass`)
had not landed when `main` was last merged, at `50e273fd7`.
- **Seat-added, not filed (zero pull; no producer calls it):** on the
Turso REMOTE face over a libSQL `file:` client (a supplied client, or
`mode: 'remote'` with a `file:` url), the raw door
`driver.execute('PRAGMA incremental_vacuum')` keeps the old hazard.
libSQL `execute()` leaves the statement unfinished, so a row created
afterwards on that connection is lost: the dev measured the issuer
reading 1, a second connection 0, and 0 after disconnect. The same
behaviour sits upstream in `@libsql/client`'s `file:` implementation
(os-dev-report 5868445205). Separately, the WAL sidecar reading above is
filed as #20426.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01N8TPEsoJxPsdSdNKGnNGEN)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
1 parent dbddf02 commit e01d347
7 files changed
Lines changed: 410 additions & 35 deletions
File tree
- .changeset
- packages/drivers
- driver-sqlite-wasm/src
- driver-sql/src
- driver-turso
- src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
Lines changed: 126 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10905 | 10905 | | |
10906 | 10906 | | |
10907 | 10907 | | |
10908 | | - | |
10909 | | - | |
10910 | | - | |
10911 | | - | |
10912 | | - | |
| 10908 | + | |
| 10909 | + | |
| 10910 | + | |
| 10911 | + | |
| 10912 | + | |
| 10913 | + | |
| 10914 | + | |
| 10915 | + | |
| 10916 | + | |
| 10917 | + | |
| 10918 | + | |
| 10919 | + | |
| 10920 | + | |
| 10921 | + | |
| 10922 | + | |
| 10923 | + | |
| 10924 | + | |
| 10925 | + | |
| 10926 | + | |
| 10927 | + | |
10913 | 10928 | | |
10914 | 10929 | | |
10915 | 10930 | | |
| 10931 | + | |
| 10932 | + | |
| 10933 | + | |
| 10934 | + | |
| 10935 | + | |
| 10936 | + | |
| 10937 | + | |
| 10938 | + | |
| 10939 | + | |
| 10940 | + | |
10916 | 10941 | | |
10917 | 10942 | | |
10918 | 10943 | | |
| |||
Lines changed: 92 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
166 | 166 | | |
167 | 167 | | |
168 | 168 | | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
174 | 176 | | |
175 | 177 | | |
176 | 178 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3222 | 3222 | | |
3223 | 3223 | | |
3224 | 3224 | | |
3225 | | - | |
3226 | | - | |
3227 | | - | |
3228 | | - | |
3229 | | - | |
| 3225 | + | |
| 3226 | + | |
| 3227 | + | |
| 3228 | + | |
| 3229 | + | |
| 3230 | + | |
| 3231 | + | |
| 3232 | + | |
| 3233 | + | |
| 3234 | + | |
| 3235 | + | |
| 3236 | + | |
| 3237 | + | |
| 3238 | + | |
| 3239 | + | |
| 3240 | + | |
| 3241 | + | |
| 3242 | + | |
| 3243 | + | |
| 3244 | + | |
3230 | 3245 | | |
3231 | 3246 | | |
3232 | 3247 | | |
3233 | 3248 | | |
| 3249 | + | |
| 3250 | + | |
| 3251 | + | |
| 3252 | + | |
| 3253 | + | |
| 3254 | + | |
| 3255 | + | |
| 3256 | + | |
| 3257 | + | |
| 3258 | + | |
3234 | 3259 | | |
3235 | 3260 | | |
3236 | | - | |
| 3261 | + | |
3237 | 3262 | | |
3238 | 3263 | | |
3239 | 3264 | | |
| |||
0 commit comments