fix(driver-sqlite-wasm): a text value round-trips byte-for-byte — U+0000 no longer truncates it, a leading U+FEFF is no longer dropped on read

.changeset/19912-json-backfill-depth-limit.md

@@ -1,3 +1,3 @@
 ---
 "@objectstack/driver-sql": minor
 "@objectstack/driver-turso": patch
@@ -7,6 +7,6 @@
 
 The backfill that converges legacy json cells on their JSON-encoded form (it came with the change that made the SQLite write path JSON-encode every json value; the issue number that change cites no longer resolves, and its live record is the `SqlDriver.backfillCanonicalJsonEncoding` doc block and `sql-driver-12380-json-roundtrip.test.ts`) ran one `UPDATE … set col = json_quote(col)` over every TEXT cell SQLite's `json_valid()` rejects. `json_valid()` answers 0 for JSON nested more than 1000 levels deep (SQLite's JSON depth limit in every build this repository bundles), while the driver reads such a cell with `JSON.parse` without trouble. So a deep array written correctly through the driver was quoted into a JSON string by the next `syncSchema` / `initObjects`, and read back as a string from then on — silently, with no error.
 
-SQL now only pre-selects the candidate cells, a page at a time. The driver's own codec decides each one: a cell `JSON.parse` reads is left exactly as stored; a cell it cannot read is a legacy plain string and is rewritten to `JSON.stringify` of that string, byte-for-byte what the old statement wrote for it wherever the engine reads the stored bytes back verbatim. A cell the engine does not read back verbatim is left as stored and keeps reading as it did, where the old statement rewrote it: on `SqliteWasmDriver`, a legacy text with a leading U+FEFF or an embedded NUL (sql.js drops both when it reads the text); on any engine, text holding invalid UTF-8. Each rewrite is a compare-and-set on the text it was decided from, so a value written concurrently is never overwritten, and a re-run over a converged table still writes nothing. This covers every local SQLite face that inherits the backfill: `SqlDriver` on every client it treats as SQLite (`better-sqlite3`, `sqlite3` and its alias `sqlite`), `SqliteWasmDriver`, and `TursoDriver` in local mode.
+SQL now only pre-selects the candidate cells, a page at a time. The driver's own codec decides each one: a cell `JSON.parse` reads is left exactly as stored; a cell it cannot read is a legacy plain string and is rewritten to `JSON.stringify` of that string, byte-for-byte what the old statement wrote for it wherever the engine reads the stored bytes back verbatim. A cell the engine does not read back verbatim is left as stored and keeps reading as it did, where the old statement rewrote it: text holding invalid UTF-8, measured on better-sqlite3 and sql.js. `SqlDriver` on better-sqlite3, `TursoDriver` in local mode (which runs on better-sqlite3 too) and `SqliteWasmDriver` (sql.js) were each measured to read such a cell back with U+FFFD in place of the invalid bytes, so the text the rewrite would be decided from is not the stored text, and the cell is left alone. A legacy text with a leading U+FEFF or an embedded NUL is read back verbatim on all three faces, and is rewritten like any other plain string. Each rewrite is a compare-and-set on the text it was decided from, so a value written concurrently is never overwritten, and a re-run over a converged table still writes nothing. This covers every local SQLite face that inherits the backfill: `SqlDriver` on every client it treats as SQLite (`better-sqlite3`, `sqlite3` and its alias `sqlite`), `SqliteWasmDriver`, and `TursoDriver` in local mode.
 
 The decision rule is exported from `@objectstack/driver-sql` as `recoverUnencodedJsonText(stored)`, and `@objectstack/driver-turso`'s remote codec-residue backfill now imports it instead of carrying its own copy, so the local and remote backfills apply one rule. That is a new public export on `@objectstack/driver-sql`'s root entry, and the reason this package takes `minor`: the function returns `null` for text `JSON.parse` accepts and `JSON.stringify(stored)` for text it rejects, and it is exported so that `SqlDriver.backfillCanonicalJsonEncoding` and the remote backfill's `recoverResidueCell` decide each cell by one shared rule rather than by two copies that could drift apart. The remote backfill's behaviour is unchanged.

.changeset/19978-sqlite-wasm-text-roundtrip.md

@@ -0,0 +1,22 @@
+---
+"@objectstack/driver-sqlite-wasm": patch
+---
+
+fix(driver-sqlite-wasm): a text value now round-trips byte-for-byte, as it does through `SqlDriver` on better-sqlite3 — an embedded U+0000 no longer cuts the stored value short, and a leading U+FEFF is no longer dropped when the value is read (#19978)
+
+Clause-②: no
+
+`SqliteWasmDriver` changed a text value without raising, at two points in sql.js (measured on sql.js 1.14.1, the version the lockfile installs, against better-sqlite3, which round-trips every value below):
+
+- **Write.** sql.js binds a string with a length of `-1`, so SQLite stores it only up to the first U+0000: `'a'` + U+0000 + `'b'` was stored as the single byte `61`, and `'ab'` + U+0000 as `6162`.
+- **Read.** sql.js decodes a text cell up to the first NUL byte, through a decoder that drops a leading byte-order mark: a stored `610062` read back as `'a'`, and a stored text beginning with U+FEFF read back without it. A U+FEFF was always stored, because the write keeps it.
+
+What changes:
+
+- Every text cell the driver reads is decoded from its stored bytes, so a U+0000 anywhere in it and a U+FEFF at its start come back as stored. This includes values already on disk: a stored text that begins with U+FEFF now reads back with it.
+- A string value holding U+0000 is bound as its UTF-8 bytes, and the parameter that receives it becomes `+CAST(<parameter> AS TEXT)`, so SQLite stores the same TEXT value better-sqlite3 stores, and compares it the same way. Positional `?`, numbered `?NNN` and named parameters are all numbered as SQLite numbers them. A statement that binds no such string runs exactly as before.
+- An equality filter on such a value now compares the whole value. Before, the comparand was cut at the same U+0000 as the stored value, so `{ v: 'a' }` also matched a row written as `'a'` + U+0000 + `'b'`. It no longer does.
+- The local `Field.json` storage backfill (`SqlDriver.backfillCanonicalJsonEncoding`) now converges a legacy json text cell holding a leading U+FEFF or an embedded U+0000 on this driver too: measured on the next `initObjects`, a stored `EFBBBF78` becomes `22EFBBBF7822` and a stored `610062` becomes `22615C75303030306222`, the same bytes better-sqlite3 writes, where before this change both were left as stored because sql.js did not read them back verbatim.
+- If the loaded sql.js has no `Statement.getBlob` (the one sql.js read that carries a byte length), reading a text cell throws instead of decoding through the lossy path. sql.js 1.14.1 has it in its Node, browser and debug builds.
+
+What does not change: a value already stored cut short stays cut short. The bytes after the U+0000 were never written, so nothing can restore them.

packages/drivers/driver-sql/src/sql-driver.ts

@@ -11308,8 +11308,8 @@ export class SqlDriver implements IDataDriver {
    * the page read and the write is never overwritten. A page's writes commit as
    * one transaction. Because the new text is a function of the old text alone,
    * the compare-and-set also withholds, in the safe direction, a cell whose
-   * stored bytes the engine does not read back verbatim (sql.js drops an
-   * embedded NUL and a leading U+FEFF; any engine replaces invalid UTF-8): its
+   * stored bytes the engine does not read back verbatim (better-sqlite3 and
+   * sql.js, the engines measured, read invalid UTF-8 back as U+FFFD): its
    * bytes are not what was read, so it is not written, and it keeps reading as
    * it did. On better-sqlite3 every other text round-trips byte-for-byte, and
    * `JSON.stringify` of it is byte-identical to the `json_quote()` this method

packages/drivers/driver-sqlite-wasm/src/knex-wasm-dialect.ts

@@ -26,6 +26,7 @@ import { createRequire } from 'node:module';
 
 import type { SqlJsStatic } from 'sql.js';
 
+import { exactTextBindings, readExactRow } from './sqljs-exact-text.js';
 import {
   WasmSqliteConnection,
   type PersistMode,
@@ -229,7 +230,11 @@ export function getClient_WasmSqlite(): any {
       if (!connection) throw new Error('No connection provided');
 
       const db = connection.raw;
-      const bindings = formatBindings(obj.bindings);
+      // A text binding holding U+0000 is rebound so sql.js's text bind cannot
+      // cut it at the NUL; `sql` is then the statement actually run. Every
+      // other statement comes back unchanged. The classification below keeps
+      // reading `obj.sql` — the rewrite never touches a statement's keyword.
+      const { sql, bindings } = exactTextBindings(obj.sql, formatBindings(obj.bindings));
 
       // ── 1. EXECUTE ────────────────────────────────────────────────────────
       // Three execution shapes. None of them decides persistence: that is
@@ -244,20 +249,24 @@ export function getClient_WasmSqlite(): any {
       // return rows used by Knex's schema introspection/columnInfo, and
       // `db.run` discards those rows.
       if (DDL_RE.test(obj.sql)) {
-        db.run(obj.sql, bindings as any);
+        db.run(sql, bindings as any);
         obj.response = [];
       } else if (
         isRowReturningExecution(obj.method, obj.returning) ||
         /^\s*PRAGMA\b/i.test(obj.sql)
       ) {
         // Row-returning branch. NOTE this is also where `INSERT … RETURNING *`
         // and `UPDATE … RETURNING *` land — statements that very much write.
-        const stmt = db.prepare(obj.sql);
+        const stmt = db.prepare(sql);
         try {
           if (bindings.length) stmt.bind(bindings as any);
           const rows: Record<string, unknown>[] = [];
+          let columns: string[] | undefined;
           while (stmt.step()) {
-            rows.push(stmt.getAsObject());
+            // Not `getAsObject()`: its text decode stops at U+0000 and drops a
+            // leading U+FEFF (see `sqljs-exact-text.ts`).
+            columns ??= stmt.getColumnNames();
+            rows.push(readExactRow(stmt, columns));
           }
           obj.response = rows;
         } finally {
@@ -266,7 +275,7 @@ export function getClient_WasmSqlite(): any {
       } else {
         // Row-less write path: execute via `run` and capture SQLite's
         // per-connection lastID / changes counters.
-        db.run(obj.sql, bindings as any);
+        db.run(sql, bindings as any);
         const changes = db.getRowsModified();
         let lastID: number | bigint = 0;
         if (obj.method === 'insert') {