Skip to content

Commit e01d347

Browse files
fix(driver-sql, driver-turso): reclaimSpace() returns the whole SQLite freelist, not one page per call (#20106) (#20425)
Fixes #20106 Clause-②: no `reclaimSpace()` now returns the whole SQLite freelist on every face. Every reading below comes from a second connection. The file size is read beside it. Each run has an empty-freelist control. Measured head: `0adf65deb`. ## What was wrong SQLite's incremental-vacuum program frees one page per `sqlite3_step()`, and it yields a result row with no columns for each page. A caller that steps it once frees one page. Two clients stepped it once: - **`SqlDriver` on better-sqlite3, and `TursoDriver` in local mode**, which is the same code. knex's better-sqlite3 client runs a statement that declares no result columns with `Statement.run()`, and `run()` steps once. `Statement.reader` is `false` for this pragma. - **`TursoDriver` in remote mode.** The libSQL client's `execute()` steps the statement once and leaves it unfinished. The page it freed never reached the file. The unfinished statement also held the connection's implicit transaction open, so a later write on that connection never reached the file either. ## What changed - `packages/drivers/driver-sql/src/sql-driver.ts`, `SqlDriver.reclaimSpace`: when the knex client is better-sqlite3, the pragma runs through that binding's own `exec()` on the pooled connection. `exec()` steps every statement until SQLite reports done. Every other SQLite client stays on `knex.raw`. sql.js steps every PRAGMA to the end in `driver-sqlite-wasm`'s dialect (measured). knex's node-sqlite3 client uses `Database.all()` (read from knex's source; the binding is not installed here). - `packages/drivers/driver-turso/src/turso-driver.ts`, the remote `reclaimSpace` route: it reads `PRAGMA freelist_count` through the raw door first. That read also connects the transport lazily, as every remote door does. When the count is `0` nothing more is sent. Otherwise the vacuum runs through the client's `executeMultiple()`. Both statements keep the raw door's envelope (`DATABASE_ERROR` / 500). - `packages/drivers/driver-turso/README.md`: the sentence on the remote `reclaimSpace()` now says what the route sends. - Tests in all three driver packages, and `.changeset/20106-reclaim-space-full-freelist.md` (`patch` for `driver-sql` and `driver-turso`). ## Readings: freelist / page count from a second connection The fixture fills about 300 pages and deletes them. The control is an empty freelist. | face | before | `main` (`789b2ae54`) after one call | this PR after one call | |:--|:--|:--|:--| | `SqlDriver` (better-sqlite3, WAL) | 300 / 304 | 299 / 303 | 0 / 4 | | `SqliteWasmDriver` (sql.js image) | 300 / 304 | 0 / 4, already complete | 0 / 4 | | `TursoDriver` local (better-sqlite3, WAL) | 300 / 304 | 299 / 303 | 0 / 4 | | `TursoDriver` remote (`@libsql/client`, `file:`) | 300 / 304 | 300 / 304 (the issuing connection read 299) | 0 / 4 | | control, every face | 0 / 4 | 0 / 4 | 0 / 4 | File size in bytes: - **Wasm image and remote `file:` database** (rollback journal): 1,245,184 → 16,384 with this PR. On `main` the remote file stayed at 1,245,184, even after disconnect. - **The two WAL faces**, after disconnect: `main` 1,241,088, this PR 16,384. Raw clients, 300 free pages, read from a second connection: - **better-sqlite3.** `Statement.run('PRAGMA incremental_vacuum')` → 299. `incremental_vacuum(600)` through `run()` → 299 too. `db.pragma()` and `db.exec()` → 0. - **`@libsql/client` `file:`.** - `execute()` → 300. The issuing connection read 299, and the file was unchanged after `close()`. - `execute('PRAGMA incremental_vacuum(600)')` → 300. - `executeMultiple()` → 0. - `batch([...], 'write')` → throws `SQLITE_BUSY: cannot commit transaction - SQL statements in progress`. - **sql.js.** A prepare-and-step loop (the wasm dialect's branch) → 0. One step → 299. Remote face, write after `reclaimSpace()`: a `create()` made after the call is read back by the issuing connection. On `main` a second connection counted 0 rows, and after disconnect it still counted 0: the row was lost. With this PR it counts 1, and 1 after disconnect. Cost at 25,660 free pages through knex on better-sqlite3 (shared box, so read the ratio): - `exec()`: 245 ms. - N single-page `raw()` calls in one transaction: 1,352 ms. - N autocommitted `raw()` calls: 1,606 ms. - One `raw()` call (`main`): 1.3 ms, for one page. ## The dispatch's mechanism hypotheses - **H1**: confirmed on `789b2ae54`: `if (!this.isSqlite) return; await this.knex.raw('PRAGMA incremental_vacuum');`. - **H2**: the first half is confirmed: one execution steps once and frees one page. The second half is falsified. An explicit page count does not help, because `incremental_vacuum(600)` freed one page through both better-sqlite3 `run()` and libSQL `execute()`. The count is a ceiling, not what ends the loop. The statement has to run to completion. - **H3**: partly falsified. `SqliteWasmDriver` and local `TursoDriver` do inherit `SqlDriver.reclaimSpace`. But the defect lived in the client binding, not in the method's text, and the wasm face was already complete on `main` (its dialect steps every PRAGMA to the end). For this method there are three independent implementations: - better-sqlite3 through knex: two faces, both broken, fixed at the `SqlDriver` seam; - the sql.js dialect: one face, already correct, now pinned; - the libSQL remote route: one face, broken, fixed in `turso-driver.ts`. - **H4**: confirmed, and worse than a reading. Through libSQL `execute()`, a second connection saw nothing land: not the freelist, not the file size, and not a later write on the same connection. ## Does `os db clean` reach this method? No. `packages/cli/src/commands/db/clean.ts` never calls `reclaimSpace()`. It runs `driver.execute('PRAGMA auto_vacuum = INCREMENTAL')`, then `driver.execute('VACUUM')`, then disconnects. A full `VACUUM` finishes in one step. Measured through `SqlDriver.execute` on a legacy file (`auto_vacuum` 0) with 300 free pages: - freelist 300 → 0; - `auto_vacuum` 0 → 2; - file 1,236,992 → 12,288 bytes. It is not edited here. ## The remote face: measured and not measured Measured over a libSQL `file:` client only (`@libsql/client` 0.17.4, `libsql` 0.5.29), plus a scripted client for the call order and the refusal envelope. Not measured, because there is no live server: what a hosted libSQL / Turso server does. That covers whether it accepts `PRAGMA freelist_count` and `PRAGMA incremental_vacuum`, and how it steps `execute()` against `executeMultiple()` (a hrana sequence request). The route adds one read round trip when there are free pages. When the freelist is empty it sends no write. ## Tests - `driver-sql/src/sql-driver-sqlite-reclaim-space.test.ts` (new), 4 cases: - WAL: freelist 0 and page count equal to before minus free, read from a second connection; file size equal to pages times page size after close; - DELETE journal: the same, and the file shrinks while the driver is open; - the empty-freelist control; - the pooled connection is handed back. - `driver-sqlite-wasm/src/sqlite-wasm-reclaim-space.test.ts` (new), 2 cases: the persisted image, read by a fresh sql.js database, plus the control. - `driver-turso/src/turso-remote-inherited-members.test.ts`: the old pin read the freelist from the issuing client and asserted `toBeLessThan(before)`, which a one-page, connection-local drop passed. It now reads a second client and asserts equality. Added: - the control on both faces; - the remote file shrinking while open; - a write after the call landing; - the call order (count first, nothing more on 0, vacuum through `executeMultiple`); - the `DATABASE_ERROR` / 500 envelope when either statement is refused. Full suites at `0adf65deb`: - `driver-sql`: 195 files passed, 11 skipped; 3,224 tests passed, 178 skipped. - `driver-turso`: 73 files; 1,927 tests passed, 16 skipped. - `driver-sqlite-wasm`: 36 files; 658 tests passed. - `typecheck` is green for all three. Each package's `tsconfig` includes its tests; the typecheck read the turso test file and caught an `Array.prototype.at` before it landed. Ablations (committed state first; every leg went through `scripts/ablation-replace.mjs`, and the restore was proven blob-equal to HEAD): - **A**: better-sqlite3 routed back to `knex.raw`. The `driver-sql` test goes red on WAL and DELETE (`{ freelist: 299, pages: 303 }` against `{ freelist: 0, pages: 4 }`); the control and the pool case stay green. With `driver-sql` rebuilt and the marker proven in `dist/` by `ablation-dist-preflight`, the turso local face goes red (`{ freelist: 39, pages: 47 }`) and the wasm test stays green (2/2). That is the H3 reading. The restore was rebuilt, and `--absent` proved `dist/` and the tree clean. - **B**: the `knex.raw` arm made a no-op, `dist/` rebuilt, marker proven. The wasm test goes red (`{ freelist: 300, pages: 304 }`) and its control stays green. The restore was rebuilt and `--absent` passed. - **C**: the remote route sent back through `execute()`. 5 turso cases go red: second-connection freelist 40 against 0, the file size case, the write after the call (0 against 1), the call order, and the vacuum refusal. The other 77 stay green. ## Gates `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` at `0adf65deb` derives 63 commands. All 63 were run, and every exit code was recorded before any pipe: all 63 exited 0. On the first run at `155234b97`: - `check:query-options-erasure` was red: the test surface grew 236 → 237 from an `as any` on a query bag in the new tests. The bags are now typed, and it holds at 236. - `check:dual-build-cjs-loads`, `check:lean-entry-closure` and `check:type-check-debt` refused with `PREREQUISITE NOT MET` (exit 3). They passed after `turbo run build` over `./packages/*` and `./packages/*/*`. `--ran`: 63 derived, 63 run, 0 NOT-MEASURED, 0 UNRUN. `check:driver-conformance`: before, at `789b2ae54`, 50 covered cells and 0 in the DEBT ledger (0 exempt). After, at `0adf65deb`, the same: 50 covered, 0 DEBT. `pnpm lint` is CI's run. The narrowed run here: `eslint --no-inline-config --format json` over the 5 changed TypeScript files reports 5 files, 0 errors and 0 warnings, with no file reported as ignored. `eslint.config.mjs` sets no `parserOptions.project` and registers no typed rule, so a diff cannot move the verdict of an untouched file. ## Acceptance notes - **The `-wal` sidecar (a finding, reported for the seat to file; not fixed here).** On a file-backed database in WAL mode, the default, a to-completion vacuum passes the freed pages through the `-wal` file. At 25,754 free pages through `SqlDriver`: - the second connection reads freelist 0 and page count 4; - the database file goes 105,631,744 → 16,384; - `-wal` goes 4,577,352 → 91,855,432 and keeps that size until the last connection closes. Measured options on raw better-sqlite3, 25,600 rows freed, one run on a shared box (database file + `-wal`, while open): | option | database file | `-wal` | time | |:--|:--|:--|:--| | `exec()` alone (this PR) | 12,288 | 90,112,672 | 267 ms | | + `wal_checkpoint(TRUNCATE)` | 12,288 | 0 | 309 ms | | chunked `incremental_vacuum(1000)` + `wal_checkpoint(PASSIVE)` | 12,288 | 210,152 | 71 ms | A TRUNCATE checkpoint can wait on another process's readers for up to the busy timeout. PASSIVE never waits. - **File surface.** `packages/drivers/driver-turso/README.md` is outside the claim's listed surface. It documents the exact route this PR changes, and its old sentence ("sends the statement local mode issues") would have become false. - **Legacy files.** A file whose `auto_vacuum` is still `NONE` reclaims nothing through `reclaimSpace()` (measured: 300 → 300), as the method's comment already says. The lifecycle report still lists that datasource as reclaimed. `os db clean` is the documented remedy. - #20107 (the remote read arms) and #20355 (`crossFieldComparisonClass`) had not landed when `main` was last merged, at `50e273fd7`. - **Seat-added, not filed (zero pull; no producer calls it):** on the Turso REMOTE face over a libSQL `file:` client (a supplied client, or `mode: 'remote'` with a `file:` url), the raw door `driver.execute('PRAGMA incremental_vacuum')` keeps the old hazard. libSQL `execute()` leaves the statement unfinished, so a row created afterwards on that connection is lost: the dev measured the issuer reading 1, a second connection 0, and 0 after disconnect. The same behaviour sits upstream in `@libsql/client`'s `file:` implementation (os-dev-report 5868445205). Separately, the WAL sidecar reading above is filed as #20426. --- _Generated by [Claude Code](https://claude.ai/code/session_01N8TPEsoJxPsdSdNKGnNGEN)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent dbddf02 commit e01d347

7 files changed

Lines changed: 410 additions & 35 deletions

File tree

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
'@objectstack/driver-sql': patch
3+
'@objectstack/driver-turso': patch
4+
---
5+
6+
fix(driver-sql, driver-turso): `reclaimSpace()` returns the whole SQLite freelist, not one page per call (#20106)
7+
8+
Clause-②: no
9+
10+
`reclaimSpace()` is what the lifecycle service calls after every sweep that deleted rows (ADR-0057 §3.4). On SQLite it runs `PRAGMA incremental_vacuum`, and that statement frees one page per step. Two clients stepped it once:
11+
12+
- **`SqlDriver` on better-sqlite3, and `TursoDriver` in local mode.** knex's better-sqlite3 client runs a statement that declares no result columns with `Statement.run()`, which steps it once. A database with 300 free pages had 299 after the call, read from a second connection, and the file barely shrank. `incremental_vacuum(N)` freed one page too. The method now drives that binding through its own `exec()`, which steps the statement until SQLite reports done: 300 → 0, and the file shrinks by those pages.
13+
- **`TursoDriver` in remote mode.** The libSQL client's `execute()` stepped the statement once and left it unfinished. Over a libSQL `file:` client, the issuing connection read one page fewer, but a second connection read the freelist and the file size unchanged, and a row written after the call on the same connection never reached the file. The remote route now reads `PRAGMA freelist_count`, sends nothing more when it is `0`, and otherwise runs the vacuum through the client's `executeMultiple()`: 300 → 0 from a second connection, and the later write lands. A server that refuses either statement answers `DATABASE_ERROR` / 500, as before. What a hosted libSQL server does with either call is not measured.
14+
15+
`SqliteWasmDriver` was already complete: its dialect steps every PRAGMA to the end (300 → 0 before and after this change).
16+
17+
On a file-backed database in WAL mode (the default) the database file shrinks once a checkpoint runs, and during the call the freed pages pass through the `-wal` file, which keeps its size until the last connection closes.
18+
19+
Nothing to migrate: `reclaimSpace()` keeps its signature, and a database whose `auto_vacuum` mode is not `INCREMENTAL` still reclaims nothing, as before.
Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// `reclaimSpace()` (ADR-0057 §3.4) returns the WHOLE freelist on SQLite, read
4+
// from a SECOND connection rather than from the one that issued the statement.
5+
//
6+
// SQLite's incremental-vacuum program frees one page per step. knex's
7+
// better-sqlite3 client runs a statement that declares no result columns with
8+
// `Statement.run()`, which steps it once, so this method used to free ONE page
9+
// per call: 300 → 299 from a second connection, and the lifecycle sweep that
10+
// calls it after every bulk delete left the file at its high-water mark.
11+
12+
import { afterEach, describe, expect, it } from 'vitest';
13+
import { mkdtempSync, rmSync, statSync } from 'node:fs';
14+
import { tmpdir } from 'node:os';
15+
import { join } from 'node:path';
16+
import knex, { type Knex } from 'knex';
17+
import { SqlDriver, type SqlDriverConfig } from './sql-driver.js';
18+
19+
const PAGE_SIZE = 4096;
20+
const cleanup: Array<() => Promise<void> | void> = [];
21+
afterEach(async () => {
22+
while (cleanup.length) await cleanup.pop()!();
23+
});
24+
25+
function tempDb(): string {
26+
const dir = mkdtempSync(join(tmpdir(), 'os-reclaim-'));
27+
cleanup.push(() => rmSync(dir, { recursive: true, force: true }));
28+
return join(dir, 'app.db');
29+
}
30+
31+
/** A connected driver on `filename`, and a `close()` the cleanup then skips. */
32+
async function openDriver(
33+
filename: string,
34+
cfg: Partial<SqlDriverConfig> = {},
35+
): Promise<{ driver: SqlDriver; close: () => Promise<void> }> {
36+
const driver = new SqlDriver({ client: 'better-sqlite3', connection: { filename }, useNullAsDefault: true, ...cfg });
37+
let open = true;
38+
const close = async () => {
39+
if (!open) return;
40+
open = false;
41+
await driver.disconnect();
42+
};
43+
cleanup.push(close);
44+
await driver.connect();
45+
return { driver, close };
46+
}
47+
48+
/** Fill `rows` rows of ~4 KB each, then delete them all: roughly one freelist page per row. */
49+
async function freePages(driver: SqlDriver, rows: number): Promise<void> {
50+
await driver.initObjects([{ name: 'bulk', fields: { body: { type: 'text' } } }]);
51+
const body = 'x'.repeat(4000);
52+
for (let i = 0; i < rows; i += 100) {
53+
const batch = Array.from({ length: Math.min(100, rows - i) }, (_, j) => ({ id: `r${i + j}`, body }));
54+
await driver.bulkCreate('bulk', batch);
55+
}
56+
await driver.deleteMany('bulk', { where: { id: { $ne: '' } } });
57+
}
58+
59+
/** The freelist and page count as a SECOND, read-only connection reads them. */
60+
async function secondConnection(filename: string): Promise<{ freelist: number; pages: number }> {
61+
const reader: Knex = knex({
62+
client: 'better-sqlite3',
63+
connection: { filename, options: { readonly: true } },
64+
useNullAsDefault: true,
65+
});
66+
try {
67+
const [free] = await reader.raw('PRAGMA freelist_count');
68+
const [count] = await reader.raw('PRAGMA page_count');
69+
return { freelist: Number(free.freelist_count), pages: Number(count.page_count) };
70+
} finally {
71+
await reader.destroy();
72+
}
73+
}
74+
75+
describe('SqlDriver.reclaimSpace() on better-sqlite3 returns the whole freelist', () => {
76+
it('WAL (the file-backed default): every free page leaves the database, and the file shrinks once closed', async () => {
77+
const file = tempDb();
78+
const { driver, close } = await openDriver(file);
79+
await freePages(driver, 300);
80+
const before = await secondConnection(file);
81+
expect(before.freelist).toBeGreaterThanOrEqual(250);
82+
83+
await driver.reclaimSpace();
84+
85+
const after = await secondConnection(file);
86+
expect(after).toEqual({ freelist: 0, pages: before.pages - before.freelist });
87+
await close();
88+
expect(statSync(file).size).toBe(after.pages * PAGE_SIZE);
89+
});
90+
91+
it('DELETE journal: the file shrinks while the driver is still open', async () => {
92+
const file = tempDb();
93+
const { driver } = await openDriver(file, { sqliteJournalMode: 'delete' });
94+
await freePages(driver, 300);
95+
const before = await secondConnection(file);
96+
expect(before.freelist).toBeGreaterThanOrEqual(250);
97+
expect(statSync(file).size).toBe(before.pages * PAGE_SIZE);
98+
99+
await driver.reclaimSpace();
100+
101+
const after = await secondConnection(file);
102+
expect(after).toEqual({ freelist: 0, pages: before.pages - before.freelist });
103+
expect(statSync(file).size).toBe(after.pages * PAGE_SIZE);
104+
});
105+
106+
it('control: an empty freelist resolves, and nothing changes', async () => {
107+
const file = tempDb();
108+
const { driver } = await openDriver(file, { sqliteJournalMode: 'delete' });
109+
await freePages(driver, 0);
110+
const before = await secondConnection(file);
111+
expect(before.freelist).toBe(0);
112+
113+
await expect(driver.reclaimSpace()).resolves.toBeUndefined();
114+
115+
expect(await secondConnection(file)).toEqual(before);
116+
});
117+
118+
it('the pooled connection is handed back: the driver answers a query after the call', async () => {
119+
const file = tempDb();
120+
const { driver } = await openDriver(file);
121+
await freePages(driver, 50);
122+
await driver.reclaimSpace();
123+
await driver.bulkCreate('bulk', [{ id: 'after', body: 'still writable' }]);
124+
expect(await driver.count('bulk', {})).toBe(1);
125+
});
126+
});

‎packages/drivers/driver-sql/src/sql-driver.ts‎

Lines changed: 30 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10905,14 +10905,39 @@ export class SqlDriver implements IDataDriver {
1090510905

1090610906
/**
1090710907
* Reclaim free pages after bulk deletions (ADR-0057 §3.4). On SQLite this
10908-
* issues `PRAGMA incremental_vacuum`, returning freelist pages to the OS —
10909-
* it pairs with the `auto_vacuum=INCREMENTAL` default set in {@link connect}
10910-
* (files created before that default need one full `VACUUM` to adopt it).
10911-
* Postgres/MySQL manage space via their own vacuum/purge machinery, so this
10912-
* is a no-op there.
10908+
* runs `PRAGMA incremental_vacuum` TO COMPLETION, returning every freelist
10909+
* page — it pairs with the `auto_vacuum=INCREMENTAL` default set in
10910+
* {@link connect} (files created before that default need one full `VACUUM`
10911+
* to adopt it). Postgres/MySQL manage space via their own vacuum/purge
10912+
* machinery, so this is a no-op there.
10913+
*
10914+
* "To completion" is a property of how the statement is STEPPED, not of its
10915+
* text. SQLite's incremental-vacuum program frees one page per step and
10916+
* yields a column-less result row for it, so a caller that steps once frees
10917+
* one page. knex's better-sqlite3 client runs a statement that declares no
10918+
* result columns with `Statement.run()`, which steps it once: through
10919+
* `knex.raw` one call freed ONE page (freelist 300 → 299, read from a second
10920+
* connection). An explicit page count, `incremental_vacuum(N)`, freed one
10921+
* page too — the count is a ceiling, not what stops the loop. So that binding
10922+
* is driven through its own `exec()`, which steps every statement until
10923+
* SQLite reports done (300 → 0). sql.js needs nothing: `driver-sqlite-wasm`'s
10924+
* dialect already iterates every row a PRAGMA yields (300 → 0 through the
10925+
* `knex.raw` below). knex's node-sqlite3 client runs a raw statement with
10926+
* `Database.all()`, which reads every row too (read from knex's source; that
10927+
* binding is not installed in this repository).
1091310928
*/
1091410929
async reclaimSpace(_options?: DriverOptions): Promise<void> {
1091510930
if (!this.isSqlite) return;
10931+
const client = this.knex.client;
10932+
if (client.driverName === 'better-sqlite3') {
10933+
const connection = await client.acquireConnection();
10934+
try {
10935+
connection.exec('PRAGMA incremental_vacuum');
10936+
} finally {
10937+
await client.releaseConnection(connection);
10938+
}
10939+
return;
10940+
}
1091610941
await this.knex.raw('PRAGMA incremental_vacuum');
1091710942
}
1091810943

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// `reclaimSpace()` (ADR-0057 §3.4) is inherited from `SqlDriver`, and on this
4+
// transport it returns the WHOLE freelist: the wasm dialect steps every PRAGMA
5+
// until SQLite reports done, so `PRAGMA incremental_vacuum` runs to its end
6+
// through `knex.raw`. `SqlDriver` drives better-sqlite3 through that binding's
7+
// own `exec()` instead, because knex's better-sqlite3 client stepped the
8+
// statement once and freed one page per call; this transport stays on the
9+
// `knex.raw` arm, and this file pins that it is complete there.
10+
//
11+
// Read from a second reader: the image this driver persists, opened by a fresh
12+
// sql.js database, never the live one that issued the statement.
13+
14+
import { afterEach, describe, expect, it } from 'vitest';
15+
import { mkdtempSync, readFileSync, rmSync, statSync } from 'node:fs';
16+
import { tmpdir } from 'node:os';
17+
import { join } from 'node:path';
18+
import initSqlJs from 'sql.js';
19+
20+
import { SqliteWasmDriver } from '../src/index.js';
21+
22+
const PAGE_SIZE = 4096;
23+
const dirs: string[] = [];
24+
const drivers: SqliteWasmDriver[] = [];
25+
26+
afterEach(async () => {
27+
await Promise.all(drivers.splice(0).map((d) => d.disconnect().catch(() => {})));
28+
for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true });
29+
});
30+
31+
async function openDriver(): Promise<{ driver: SqliteWasmDriver; file: string }> {
32+
const dir = mkdtempSync(join(tmpdir(), 'wasm-reclaim-'));
33+
dirs.push(dir);
34+
const file = join(dir, 'app.db');
35+
const driver = new SqliteWasmDriver({ filename: file, persist: 'on-write' });
36+
drivers.push(driver);
37+
await driver.connect();
38+
return { driver, file };
39+
}
40+
41+
/** Fill `rows` rows of ~4 KB each, then delete them all: roughly one freelist page per row. */
42+
async function freePages(driver: SqliteWasmDriver, rows: number): Promise<void> {
43+
await driver.initObjects([{ name: 'bulk', fields: { body: { type: 'text' } } }]);
44+
const body = 'x'.repeat(4000);
45+
for (let i = 0; i < rows; i += 100) {
46+
const batch = Array.from({ length: Math.min(100, rows - i) }, (_, j) => ({ id: `r${i + j}`, body }));
47+
await driver.bulkCreate('bulk', batch);
48+
}
49+
await driver.deleteMany('bulk', { where: { id: { $ne: '' } } });
50+
await driver.flush();
51+
}
52+
53+
/** The freelist and page count of the persisted image, read by a fresh sql.js database. */
54+
async function persistedImage(file: string): Promise<{ freelist: number; pages: number }> {
55+
const SQL = await initSqlJs();
56+
const db = new SQL.Database(readFileSync(file));
57+
try {
58+
const freelist = Number(db.exec('PRAGMA freelist_count')[0].values[0][0]);
59+
const pages = Number(db.exec('PRAGMA page_count')[0].values[0][0]);
60+
return { freelist, pages };
61+
} finally {
62+
db.close();
63+
}
64+
}
65+
66+
describe('SqliteWasmDriver.reclaimSpace() returns the whole freelist', () => {
67+
it('every free page leaves the persisted image, and the file shrinks to the pages left', async () => {
68+
const { driver, file } = await openDriver();
69+
await freePages(driver, 300);
70+
const before = await persistedImage(file);
71+
expect(before.freelist).toBeGreaterThanOrEqual(250);
72+
73+
await driver.reclaimSpace();
74+
await driver.flush();
75+
76+
const after = await persistedImage(file);
77+
expect(after).toEqual({ freelist: 0, pages: before.pages - before.freelist });
78+
expect(statSync(file).size).toBe(after.pages * PAGE_SIZE);
79+
});
80+
81+
it('control: an empty freelist resolves, and the image is unchanged', async () => {
82+
const { driver, file } = await openDriver();
83+
await freePages(driver, 0);
84+
const before = await persistedImage(file);
85+
expect(before.freelist).toBe(0);
86+
87+
await expect(driver.reclaimSpace()).resolves.toBeUndefined();
88+
await driver.flush();
89+
90+
expect(await persistedImage(file)).toEqual(before);
91+
});
92+
});

‎packages/drivers/driver-turso/README.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -166,11 +166,13 @@ Answered in remote mode, differently from local mode:
166166
presentation (a declared boolean reads back as `true` / `false`), and an
167167
unknown column refused with `INVALID_FIELD` / 400. A tenant-scoped call is
168168
refused (table above), because no remote read applies the tenant scope.
169-
- **`reclaimSpace()`** sends the statement local mode issues,
170-
`PRAGMA incremental_vacuum`, to the remote database. It returns free pages
171-
only on a database whose `auto_vacuum` mode is `INCREMENTAL`: local mode sets
172-
that mode when it connects, and remote mode does not. A server that refuses
173-
the statement answers `DATABASE_ERROR` / 500.
169+
- **`reclaimSpace()`** reads `PRAGMA freelist_count` from the remote database
170+
and, when there are free pages, runs the statement local mode issues,
171+
`PRAGMA incremental_vacuum`, to completion there (through the client's
172+
`executeMultiple()`). It returns free pages only on a database whose
173+
`auto_vacuum` mode is `INCREMENTAL`: local mode sets that mode when it
174+
connects, and remote mode does not. A server that refuses either statement
175+
answers `DATABASE_ERROR` / 500.
174176
- **`supportsRotation`** is `false`. The lifecycle service reads it, and for an
175177
object that declares `lifecycle.storage.strategy: 'rotation'` it then takes the
176178
path it has for a driver that cannot shard: an age-based reap bounded by the

‎packages/drivers/driver-turso/src/turso-driver.ts‎

Lines changed: 31 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3222,18 +3222,43 @@ export class TursoDriver extends SqlDriver {
32223222

32233223
/**
32243224
* Reclaim free pages — answered on the REMOTE face with the statement the
3225-
* local face issues, `PRAGMA incremental_vacuum`, sent to the remote database
3226-
* through the raw door's envelope (`DATABASE_ERROR` / 500 if the server
3227-
* refuses it). As on a local file, it returns pages only on a database whose
3228-
* `auto_vacuum` is `INCREMENTAL`; the remote face does not set that mode at
3229-
* connect, the local face does.
3225+
* local face issues, `PRAGMA incremental_vacuum`, run to completion on the
3226+
* remote database, in the raw door's envelope (`DATABASE_ERROR` / 500 if the
3227+
* server refuses it). As on a local file, it returns pages only on a database
3228+
* whose `auto_vacuum` is `INCREMENTAL`; the remote face does not set that
3229+
* mode at connect, the local face does.
3230+
*
3231+
* The statement goes through the client's `executeMultiple()`, not
3232+
* `execute()`. SQLite frees one page per step of this statement, and the
3233+
* libSQL client's `execute()` steps a statement that declares no result
3234+
* columns once and leaves it unfinished. Measured over a libSQL `file:`
3235+
* client: the issuing connection read one page fewer (300 → 299), a second
3236+
* connection read the freelist and the file size unchanged, and a row
3237+
* written after the call on the same connection never reached the file.
3238+
* `executeMultiple()` runs each statement to its end: a second connection
3239+
* reads 300 → 0, and the later write lands. What a hosted libSQL server does
3240+
* with either call is not measured here.
3241+
*
3242+
* The free-page count is read first, through the raw door, which connects
3243+
* the transport lazily as every remote door does. Nothing more is sent when
3244+
* the freelist is empty.
32303245
*/
32313246
override async reclaimSpace(options?: DriverOptions): Promise<void> {
32323247
this.assertRemoteTransactionUnsupported(options, 'reclaimSpace');
32333248
if (this.isRemote) {
3249+
const transport = this.remoteTransport!;
3250+
const count = 'PRAGMA freelist_count';
3251+
let freePages: number;
3252+
try {
3253+
const rows = (await transport.execute(count)) as ArrayLike<ArrayLike<unknown>>;
3254+
freePages = Number(rows[0]?.[0]);
3255+
} catch (error) {
3256+
throw this.rawStatementFault(count, error);
3257+
}
3258+
if (freePages === 0) return;
32343259
const statement = 'PRAGMA incremental_vacuum';
32353260
try {
3236-
await this.remoteTransport!.execute(statement);
3261+
await transport.getClient()!.executeMultiple(statement);
32373262
} catch (error) {
32383263
throw this.rawStatementFault(statement, error);
32393264
}

0 commit comments

Comments
 (0)