Skip to content

fix(driver-sqlite-wasm): a text value round-trips byte-for-byte — U+0000 no longer truncates it, a leading U+FEFF is no longer dropped on read - #19998

Merged
objectstack-fleet[bot] merged 9 commits into
mainfrom
claude/issue-19978-sqlite-wasm-text-roundtrip
Sep 24, 2026
Merged

objectstack-fleet[bot] merged 9 commits into
mainfrom
claude/issue-19978-sqlite-wasm-text-roundtrip

Conversation

@objectstack-fleet

@objectstack-fleet objectstack-fleet Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #19978

Clause-②: no

SqliteWasmDriver now stores and reads a text value byte-for-byte, as SqlDriver on better-sqlite3 does. Before this change, an embedded U+0000 cut the stored value short, and a leading U+FEFF was dropped when the value was read. Neither raised. The fix lives entirely in this package's sql.js transport (knex-wasm-dialect.ts plus a new sqljs-exact-text.ts). sql.js is not patched. No code line in SqlDriver moves, and no public export moves. One doc-comment parenthetical in sql-driver.ts and one sentence of a pending changeset are corrected, because this fix makes them false (see "Deliberate correction of a pending changeset" below).

Where each byte was lost (measured first; the card's readings were a scratch probe nobody had re-run)

sql.js 1.14.1 is what the lockfile installs (pnpm-lock.yaml: sql.js@1.14.1) and what node_modules/sql.js/package.json reports. The control is better-sqlite3 via driver-sql's devDependency. The probes are scratch files, not committed.

Raw sql.js, write side. Bind a JS string, then read hex(v). The hex is ASCII, so the read decode cannot hide a write-side loss:

value its UTF-8 sql.js stored better-sqlite3 stored
'a' + U+0000 + 'b' 610062 61 610062
'ab' + U+0000 616200 6162 616200
U+FEFF + 'hello' EFBBBF68656C6C6F EFBBBF68656C6C6F EFBBBF68656C6C6F
'he' + U+FEFF + 'llo' 6865EFBBBF6C6C6F same same

Raw sql.js, read side. Cells are planted with cast(x'…' as text), so no bind is involved:

stored sql.js getString sql.js getBlob on the same cell better-sqlite3
610062 "a" 610062 "a\u0000b"
EFBBBF78 "x" EFBBBF78 "x"
EFBBBF + 33 ASCII bytes the ASCII only all bytes BOM kept

H1 holds, and the loss has two separate causes. The write loses a U+0000. The read loses both a U+0000 and a leading U+FEFF. A U+FEFF was always stored intact.

H2: the exact functions.

  • Write: sql.js Statement.prototype.bindString calls sqlite3_bind_text(this.stmt, pos, strptr, -1, 0) (dist/sql-wasm-debug.js lines 684–695). The length -1 tells SQLite to read up to the first NUL.
  • Read: Statement.prototype.getString returns sqlite3_column_text, cwrapped with return type "string", so UTF8ToString decodes it. That function stops at the first NUL (findStringEnd) and decodes through a module-level new TextDecoder() whose default ignoreBOM: false drops a leading BOM. In the minified dist/sql-wasm.js this is Za=new TextDecoder with z=(a,b,c)=>a?Za.decode(C.subarray(a,$a(C,a,b,c))):"", so the BOM is dropped at every length. The debug build decodes strings of 16 bytes or fewer by hand, and those keep it. That is why a long-BOM case is pinned too.
  • Ours: Client_WasmSqlite._query in knex-wasm-dialect.ts owns both. It binds through stmt.bind / db.run and decodes through stmt.getAsObject(). wasm-connection.ts owns neither.

Driver level, base a7581b326: create → findOne, and find with { v: value }.

== driver-sql on better-sqlite3 (control)
  nul_mid   stored-hex=610062           read="a\u0000b"     faithful=true
  nul_trail stored-hex=616200           read="ab\u0000"     faithful=true
  bom_lead  stored-hex=EFBBBF68656C6C6F read="hello"  faithful=true
== driver-sqlite-wasm (sql.js), before
  nul_mid   stored-hex=61               read="a"            faithful=false
  nul_trail stored-hex=6162             read="ab"           faithful=false
  bom_lead  stored-hex=EFBBBF68656C6C6F read="hello"        faithful=false
== driver-sqlite-wasm (sql.js), this head
  (identical to the better-sqlite3 block, every line)

The fix (H3: it fits in our adapter)

  • Read. A cell that stmt.get() returns as a string is a TEXT cell. For each one, readExactRow re-reads the stored bytes with Statement.getBlob (sqlite3_column_bytes + sqlite3_column_blob; SQLite hands a UTF-8 TEXT value to column_blob unconverted) and decodes them with TextDecoder('utf-8', { ignoreBOM: true }). getBlob is not in @types/sql.js, but it is the method sql.js's own get() calls for a BLOB column. It keeps its name in all three 1.14.1 builds this package can load (sql-wasm.js, sql-wasm-browser.js, sql-wasm-debug.js); getString is renamed in the minified two. If getBlob is absent, the read throws and does not fall back to the lossy decode.

  • Write. Only a string binding that holds U+0000 is changed; nothing else can be truncated. It is bound as its UTF-8 bytes (a Uint8Array, which sql.js binds as a BLOB with an explicit length), and every parameter token that receives it is wrapped as +CAST(PARAM AS TEXT). The unary plus is load-bearing. CAST(… AS TEXT) alone carries TEXT affinity and changes a comparison. Measured on sql.js: comparing the integer 5 as less than the text ' x' answers 1 against a bound text, 0 through CAST(? AS TEXT), and 1 through +CAST(? AS TEXT) (better-sqlite3 bound: 1). A numeric-affinity column stores '12' as integer through either path. A statement with no such binding comes back as the same string and the same array.

  • Which token receives which binding follows SQLite's own numbering rule, which sqljs-exact-text.ts applies:

    • bare ? takes the largest index so far + 1;
    • ?NNN takes NNN;
    • :name / @name / #name / $name take the index of their first occurrence;
    • nothing inside quotes, […] or comments is a parameter.

    Wrapping adds or removes no parameter token, so no index moves. No statement form is refused, which is why this is Clause-②: no and not a narrowing.

  • Ruled out: an explicit-length sqlite3_bind_text through the statement pointer. That pointer is this.stmt only in the debug build; the minified builds rename it (this.Qa), so it is not a usable seam.

H4: why the shared case table is not edited here

VALUE_ROUNDTRIP_CASES lives in packages/spec/src/data/value-roundtrip-conformance.ts. The claim's file surface allowed a shared case file under packages/drivers/driver-sql/src/ "that the SQLite family's conformance suites read". No such file exists, and the real one is in packages/spec, so this is the stop-on-breach case and the table is not edited. There is a second reason, a substantive one. sql-driver-value-roundtrip-conformance.test.ts runs that table through the dialect matrix, including the live Postgres CI job. Postgres documents that its text type cannot store the character with code zero, so a U+0000 row would make that cell fail on Postgres. That is a platform decision (refuse U+0000 everywhere? answer per dialect?), not a driver fix. It is NOT MEASURED here, because there is no live Postgres in this container. A leading-U+FEFF row is a better candidate for the shared table, but how the MySQL and Postgres drivers decode it is unmeasured here. Both questions go to the seat in the report. Meanwhile the answer is pinned in this package, against the written value and its own UTF-8.

Tests

New file sqlite-wasm-text-bytes-roundtrip.test.ts, 35 cases. Each seam is pinned on its own, because one can hide the other:

  • Write: hex(v) equals the written string's UTF-8, for U+0000 in the middle and trailing, U+FEFF leading (short and long) and in the middle, and a plain control.
  • Read: cells planted by SQL literal read back exactly (610062, EFBBBF78, a long BOM value, a plain control).
  • Round trip: create → findOne, and update (whose returned row is also checked).
  • Filters: equality on each exact value selects that row and no other. Equality on 'a' no longer matches the stored 'a' + U+0000 + 'b'; before, the comparand was cut at the same NUL, so it did. $in places each NUL-bearing comparand. $contains U+FEFF and $startsWith U+FEFF select the right rows and read them back whole.
  • Placement:
    • the no-affinity pin;
    • the statement comes back unchanged when no binding holds U+0000;
    • placement past 'it''s ?', `a?`, [b?], "?", both comment forms and the identifier c$d;
    • ?NNN and :name through the driver;
    • SQLite's numbering on a mixed statement;
    • a binding no parameter receives is left for sql.js to answer as before.
  • Refusal of the fallback: a statement object without getBlob throws.

Ablations. Each ran from the committed state through scripts/ablation-replace.mjs. Every mutation was proved on disk (anchor 1 → 0, blob changed), and every restore was proved (blob equal to HEAD, git diff HEAD empty). The subject resolves from src (relative imports), so no build was involved:

  • A: read seam back to stmt.getAsObject(). 11 of 35 red: every read pin, the update pin, $contains U+FEFF (its values), and the two numbered/named pins. Every write-hex pin and every equality filter stayed green.
  • B: write rewrite disabled (truncatable.size === 0 → truncatable.size >= 0). 10 of 35 red: the U+0000 hex pins and reads, update, the prefix-equality pin, and the placement pins. Every read-seam pin stayed green. A first attempt used a replacement that was a substring of its anchor. The tool refused it (replacement count 1 → 1) and restored, and no test ran; it was repeated with a distinct replacement.
  • C: the unary plus removed. 3 red: the affinity pin (r: 0, not 1) and the two literal-placement pins.
  • D: ?NNN numbered as a bare ?. 2 red: the ?2/?1 driver pin and the mixed-statement pin.

Package runs at 58ee6c042:

  • pnpm --filter @objectstack/driver-sqlite-wasm test: 30 files, 556 passed.
  • typecheck (tsc --noEmit): exit 0. Its program includes both new files, checked with --listFiles.

Deliberate correction of a pending changeset

The dispatch asked whether the "not read back verbatim" sentence in .changeset/19912-json-backfill-depth-limit.md still holds. That changeset is still pending and belongs to a landed PR. It says the backfill leaves as stored, "on SqliteWasmDriver, a legacy text with a leading U+FEFF or an embedded NUL (sql.js drops both when it reads the text)". After the fix (head 58ee6c042) that is false. Measured with a scratch probe: legacy json TEXT cells were planted, then the backfill ran via a second initObjects.

                      legacy      better-sqlite3   wasm, both seams ablated (dist rebuilt)   wasm, this head
  bom (U+FEFF + x)    EFBBBF78    22EFBBBF7822     EFBBBF78 (left)                           22EFBBBF7822
  nul (a + U+0000 + b) 610062     22615C75303030306222  610062 (left)                        22615C75303030306222
  plain control       68656C6C6F  2268656C6C6F22   2268656C6C6F22                            2268656C6C6F22

SqliteWasmDriver now converges these cells exactly as better-sqlite3 does. ablation-dist-preflight confirmed the ablation was present in dist/ for the control. After the rebuild it was absent again and the tree was clean. The same parenthetical sits in SqlDriver.backfillCanonicalJsonEncoding's doc block in packages/drivers/driver-sql/src/sql-driver.ts ("sql.js drops an embedded NUL and a leading U+FEFF"). The seat ruled that both are corrected in this PR (claim amendment 5818397702 on #19978). Patch round 1 (6a195b37c) made the edits below, and patch round 2 (98cb90873) corrected them after contract review 5819169241 (FAIL). Round 1 had labelled TursoDriver's local mode as libsql, but every non-remote arm of TursoDriver.toKnexConfig hands Knex client: 'better-sqlite3'. The measurement behind the edits was taken fresh at 58ee6c042 on three local faces, which run on two engines: SqlDriver on better-sqlite3, TursoDriver in local mode (better-sqlite3 through Knex, url :memory:), and SqliteWasmDriver (sql.js). All three convert EFBBBF78 → 22EFBBBF7822, 610062 → 22615C75303030306222 and the control 68656C6C6F → 2268656C6C6F22. All three leave invalid UTF-8 (FF78, 61C3) as stored, because each reads it back with U+FFFD in place of the invalid bytes. SqlDriver's sqlite3 / sqlite clients are NOT MEASURED, because that client is not installed in the container. libsql is not a local engine here and is not cited. The reviewer read it directly (@libsql/client 0.17.4, :memory:): a stored FF78 panics the native binding, and a stored 610062 reads back as "a".

.changeset/19912-json-backfill-depth-limit.md (pending, landed with PR #19972; one sentence of the merge base rewritten into three):

  • Old: "A cell the engine does not read back verbatim is left as stored and keeps reading as it did, where the old statement rewrote it: on SqliteWasmDriver, a legacy text with a leading U+FEFF or an embedded NUL (sql.js drops both when it reads the text); on any engine, text holding invalid UTF-8."
  • New: "A cell the engine does not read back verbatim is left as stored and keeps reading as it did, where the old statement rewrote it: text holding invalid UTF-8, measured on better-sqlite3 and sql.js. SqlDriver on better-sqlite3, TursoDriver in local mode (which runs on better-sqlite3 too) and SqliteWasmDriver (sql.js) were each measured to read such a cell back with U+FFFD in place of the invalid bytes, so the text the rewrite would be decided from is not the stored text, and the cell is left alone. A legacy text with a leading U+FEFF or an embedded NUL is read back verbatim on all three faces, and is rewritten like any other plain string."

packages/drivers/driver-sql/src/sql-driver.ts, in the backfillCanonicalJsonEncoding doc block. Two comment lines change and no code line moves. A build of driver-sql from the merge base and one from the head differ in exactly those two comment lines of dist/index.js / dist/index.mjs, re-proved in round 2. The driver-sql suite reads 2680 passed / 170 skipped on both copies (round 1).

  • Old: "(sql.js drops an embedded NUL and a leading U+FEFF; any engine replaces invalid UTF-8)"
  • New: "(better-sqlite3 and sql.js, the engines measured, read invalid UTF-8 back as U+FFFD)"

This PR's own changeset gains one bullet: the local Field.json backfill now converts those legacy cells on this driver too, with the measured bytes.

Check Changeset is red by design on this head. check-empty-changeset.mjs reads the 19912 edit as the DELIBERATE CORRECTION class ("do NOT restore it -- say so on the PR and get it confirmed"). The edit is not restored and skip-changeset is not applied. Under ruling 1A (#19940, 5814546887), the confirmation is a same-head at-tier contract-review PASS that names this note and judges each rewritten sentence.

Gates

  • Derived gate list. node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack on this head gives 59 commands. All 59 exit 0 at 58ee6c042. --ran reconciliation: 59 derived, 59 run, 0 NOT-MEASURED (derived from recorded exit codes).
    • check:dual-build-cjs-loads, check:lean-entry-closure and check:type-check-debt first answered PREREQUISITE NOT MET (exit 3). They were re-run after turbo run build --filter='./packages/*' --filter='./packages/*/*' and answered 0.
    • check:query-options-erasure first went red: the test surface grew from 236 to 242, from this test's as any query casts. The queries are now typed as DriverQuery, and the gate is green.
  • Driver conformance census, before and after. pnpm check:driver-conformance reads OK — 50 covered cell(s), 0 in the DEBT ledger, 0 exempt both on base a7581b326 and at 58ee6c042. No cell was added or removed.
  • node scripts/check-issue-citations.mjs --base origin/main: exit 0.
  • Patch round 1 at 6a195b37c. dispatch-gates re-derives 62 commands: the 59 above, plus check:dispatcher-error-vocabulary, check:object-def-param-keys and check:tenant-chokepoint, which sql-driver.ts brings in. All 62 ran after a ./packages/* build. 61 exit 0, and check-empty-changeset exits 1 by design (the correction above). check-issue-citations --base origin/main exits 2 at this head, but only because origin/main moved past the merge base: fix(plugin-security, objectql): a row-level check holds for every row of an array insert and a predicate update #19988 removed three #16608 citations from a file this diff does not touch. Against the merge base 67ebc84a7 it exits 0. The driver-sqlite-wasm suite reads 30 files / 556 passed, and typecheck exits 0 for driver-sqlite-wasm and driver-sql. check:driver-conformance is unchanged: 50 covered, 0 in debt.
  • Patch round 2 at 98cb90873 (text only, the same two files).
    • check-changeset-no-major, check-adr-0087-registration, check-issue-citations --base 67ebc84a7, check:doc-authoring, check:nul-bytes and driver-sql typecheck all exit 0.
    • check-empty-changeset exits 1 by design: one ::error annotation, on the 19912 note.
    • dispatch-gates re-derives the same 62 families.
    • The two changesets this PR touches name libsql 0 times and "any engine" 0 times.
  • Lint, narrowed and proven. pnpm exec eslint --no-inline-config --format json over the three changed TypeScript files gives 3 files, 0 errors, 0 warnings.
    • None of the three is ignored: an explicitly passed ignored file reports a warning, and there were none.
    • eslint.config.mjs sets no parserOptions.project (0 hits for project: / projectService), so linting is not type-aware and this diff cannot move any untouched file's verdict.
    • The repo-wide pnpm lint is CI's.

Acceptance notes

  • Found here, out of scope: a $contains or $startsWith whose comparand holds U+0000 is matched on both SQLite faces, better-sqlite3 included, with the GLOB pattern cut at that U+0000 (wildcards after it included) against each value cut at its own first U+0000. So $contains 'a'+U+0000 answers only the row stored as 'a'+U+0000+'b' (measured by the second contract review). glob() cuts both the pattern and the value at their first U+0000. So a comparand that starts with U+0000 makes $contains and $endsWith match every row, and makes $startsWith answer only the rows that are empty before their first U+0000 (re-measured in patch round 2 on better-sqlite3 and sql.js, which answer identically). The SQLite text predicate is GLOB, and glob() reads its pattern as a C string. Measured on both faces with the probe above. That is a filter that silently gives the wrong answer, reproducible today. It is not fixed here; the seat filed it as driver-sql (SQLite faces): a $contains / $startsWith / $endsWith comparand holding U+0000 is cut at the NUL by glob(), so the filter answers wrongly; one that starts with U+0000 makes $contains / $endsWith match every row #19999.
  • WasmSqliteConnection's defaultLocateFile() calls require.resolve('sql.js/package.json'). That throws ERR_PACKAGE_PATH_NOT_EXPORTED on sql.js 1.14.1, whose exports map has no ./package.json. So it always returns undefined, and sql.js then locates its own .wasm, which works. Behaviour is unaffected; only the docblock's claim is dead. Noted, not filed. Carrier: none.
  • origin/main (67ebc84a7) was merged in before opening. It shares no path with this diff, and the lockfile did not move. The dependency closure was rebuilt and the package suite re-run on the merged head.

Generated by Claude Code

…no truncation at U+0000, no leading U+FEFF dropped on read

Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx
Co-authored-by: Claude <noreply@anthropic.com>
…the round trip separately, with filters and refusals

Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx
Co-authored-by: Claude <noreply@anthropic.com>
…arameter form carries a NUL-bearing text whole — no statement is refused

Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx
Co-authored-by: Claude <noreply@anthropic.com>
…uery instead of erasing them

Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/driver-sql, @objectstack/driver-sqlite-wasm, touching 18 documentable anchor(s).

8 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/data-modeling/drivers.mdx (via SqlDriver (symbol, a top-level class))
  • content/docs/data-modeling/index.mdx (via SqlDriver (symbol, a top-level class))
  • content/docs/permissions/tenant-audit-census.mdx (via SqlDriver (symbol, a top-level class))
  • content/docs/plugins/packages.mdx (via SqlDriver (symbol, a top-level class), sql.js (literal, a string literal on a changed line))
  • content/docs/protocol/kernel/index.mdx (via SqlDriver (symbol, a top-level class))
  • content/docs/protocol/kernel/lifecycle.mdx (via SqlDriver (symbol, a top-level class))
  • content/docs/protocol/objectql/query-syntax.mdx (via SqlDriver (symbol, a top-level class))
  • content/docs/protocol/objectql/types.mdx (via SqlDriver (symbol, a top-level class))

⛔ 1 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17/17-0.mdx (via SqlDriver (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 2 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 60 of 215 client-bound route-ledger rows — the other 155 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 155: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 100 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 11 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 1f05ea4fb296357dedf86ed34a234a1e779b1381 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from d2b0296585e97defce22c21b20183ff466d62e47 — the merge of head 98cb908738599a72538e5f32df4531dc357e61d2 into base 1f05ea4fb296357dedf86ed34a234a1e779b1381, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin d2b0296585e97defce22c21b20183ff466d62e47 && git checkout d2b0296585e97defce22c21b20183ff466d62e47
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 1f05ea4fb296357dedf86ed34a234a1e779b1381 98cb908738599a72538e5f32df4531dc357e61d2 && git checkout -B drift-repro 1f05ea4fb296357dedf86ed34a234a1e779b1381 && git merge --no-ff 98cb908738599a72538e5f32df4531dc357e61d2

node scripts/docs-audit/affected-docs.mjs --json 1f05ea4fb296357dedf86ed34a234a1e779b1381

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 1f05ea4fb296357dedf86ed34a234a1e779b1381 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

…s change made false — sql.js now reads a leading U+FEFF and an embedded NUL back verbatim

Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: 6a195b37c6af9fcba265588530716f93157033bd

Scope read:

Probes ran in a scratch worktree at head, on a turbo build of the two drivers' dependency closure; dist was confirmed to carry readExactRow and the corrected comment. No derived gate family was re-run.

① Derived judgments

  1. Text round-trips byte-for-byte through SqliteWasmDriver (create, findOne, update, find): RIGHT.
    • A driver probe on the built dist at head covered six cases: U+0000 in the middle, U+0000 trailing, a leading U+FEFF (short and 34-byte), a U+FEFF in the middle, and a plain control. For every case, hex(v) equals the written string's UTF-8 and findOne returns the written string.
    • update to U+FEFF+x+U+0000+y+U+0000 returns, stores and re-reads the whole value.
    • Filters:
      • equality on each exact value selects only its row;
      • equality on 'a' no longer matches the row written as 'a'+U+0000+'b';
      • $in places each NUL-bearing comparand;
      • $contains / $startsWith with U+FEFF select exactly the BOM rows.
    • Every line matches SqlDriver on better-sqlite3.
    • Cells planted by SQL literal read back verbatim: 610062, EFBBBF78 and a 37-byte BOM value. The invalid-UTF-8 cells FF78 and 61C3 read back with U+FFFD.
    • Package suite: 30 files, 556 passed.
  2. +CAST(? AS TEXT) placement follows SQLite's numbering: RIGHT.
    • A differential fuzz against sql.js's own SQLite let SQLite itself report which token holds which index: 468 statements, 1334 checks, 0 mismatches.
    • The vocabulary covered:
      • bare ?, ?NNN, :a, @a, $b, $a::x, :a(1) and #q;
      • quoted literals with doubled quotes, "?", backtick and bracket identifiers holding ?, and x'3f';
      • both comment forms, and the identifier c$d.
    • Wrapping adds or removes no parameter token, so no index can shift. A token inside a literal or a comment is never wrapped.
  3. A statement that binds no NUL-bearing string is unchanged: RIGHT.
    • exactTextBindings returns the same SQL string and the same bindings array by reference.
    • The read side differs from getAsObject() only for a stored NUL or a leading BOM, which the changeset declares. For invalid UTF-8, both decodes give U+FFFD.
    • A UTF-16le database still decodes correctly through getBlob.
    • No refusal is added.
  4. The no-getBlob path is a loud throw: RIGHT.
    • storedBytes throws, naming Statement.getBlob, before any decode, and has no fallback branch. A test pins it.
    • getBlob keeps its name in sql-wasm.js, sql-wasm-browser.js and sql-wasm-debug.js.
    • The throw fires at the first TEXT cell read, not at connect.
  5. sql-driver.ts changes comments only: RIGHT. The two changed lines are both inside the backfillCanonicalJsonEncoding doc block. A comment-stripping minified transpile of base and head is byte-identical.
  6. The change this PR causes in driver-sql's json backfill on this face is declared: RIGHT.
    • At head, the backfill converts EFBBBF78 to 22EFBBBF7822, 610062 to 22615C75303030306222 and 68656C6C6F to 2268656C6C6F22. It leaves FF78 and 61C3 as stored.
    • Before, the compare-and-set bound to the lossy sql.js read changed 0 rows on the NUL and BOM cells (measured raw).
    • The new bullet in the 19978 changeset declares this.
  7. Public surface: RIGHT. src/index.ts is unchanged. sqljs-exact-text.ts is not re-exported. packages/spec has 0 files in the diff.
  8. CI at head (REST, 41 check runs): 34 success, 5 skipped, 2 failure. Both failures are Check Changeset, each annotated on .changeset/19912-json-backfill-depth-limit.md in the DELIBERATE CORRECTION class. Every required context is green.

② Semver level

@objectstack/driver-sqlite-wasm: patch with Clause-②: no is consistent with the diff.

  • It is a bug fix: no export is added or removed, and no accepted input is newly refused. The only new throw is for a sql.js build without getBlob, which the declared ^1.14.1 ships in every build, and the changeset declares it.
  • packages/spec is untouched.
  • driver-sql needs no changeset of its own: its edit is comment-only, and the pending 19912 note already bumps it.
  • The corrected 19912 note's frontmatter and levels are untouched.

③ Boundary flags

The corrected note is .changeset/19912-json-backfill-depth-limit.md: pending, landed with PR #19972, line 10 rewritten. The clause it removes is false at this head (item 6): "on SqliteWasmDriver, a legacy text with a leading U+FEFF or an embedded NUL (sql.js drops both when it reads the text)". So DELIBERATE CORRECTION is the right class. The rewritten passage, clause by clause:

S1. "A cell the engine does not read back verbatim is left as stored and keeps reading as it did, where the old statement rewrote it: on any engine, text holding invalid UTF-8."

  • Checked against the compare-and-set in convergeJsonColumn, and by a probe on the three local faces: FF78 and 61C3 were left as stored and read with U+FFFD before and after.
  • Verdict: TRUE on the two engines the local faces run on (better-sqlite3, sql.js). UNMEASURED on the sqlite3 client, which is not installed. Softening suggested.

S2. "SqlDriver on better-sqlite3, SqliteWasmDriver (sql.js) and TursoDriver in local mode (libsql) were each measured to read such a cell back with U+FFFD in place of the invalid bytes, so the text the rewrite would be decided from is not the stored text, and the cell is left alone."

  • That the three faces read invalid UTF-8 as U+FFFD: TRUE, measured on all three at head.
  • That TursoDriver local mode runs on libsql: FALSE.
    • TursoDriver.toKnexConfig hands every local and replica arm client: 'better-sqlite3'. The driver's own docblock says "Both non-remote arms run every read and write through the inherited Knex + better-sqlite3 engine".
    • Measured: new TursoDriver({ url: ':memory:' }) has knex.client.driverName = better-sqlite3. The third face runs on the same engine as the first.
  • The implied claim that libsql reads such a cell as U+FFFD: FALSE. Measured directly through @libsql/client 0.17.4 (:memory:), reading a stored FF78 panics the native binding and aborts the process.
  • Verdict: FALSE as written.

S3. "A legacy text with a leading U+FEFF or an embedded NUL is read back verbatim on all three, and is rewritten like any other plain string."

  • Checked: the planted 610062 and EFBBBF78 read back verbatim on the three faces, and the backfill rewrites both on all three.
  • Verdict: TRUE for the three faces. It is FALSE for the engine S2 attaches to the third face: libsql read directly returns 610062 as "a". It stands once S2 stops naming libsql.

S4 (unchanged). "Each rewrite is a compare-and-set on the text it was decided from, …": TRUE by code.

S5 (unchanged). "This covers every local SQLite face that inherits the backfill: …": TRUE by code-read. The backfill is gated on SQLITE_EMIT_CLIENTS. The sentence claims coverage, not bytes, so it is still consistent with the sqlite3 client being unmeasured.

sql-driver.ts parenthetical, "(any engine replaces invalid UTF-8 with U+FFFD; measured on better-sqlite3, sql.js, libsql)":

  • "measured on … libsql" is FALSE: the third measurement ran on better-sqlite3.
  • "any engine replaces" is FALSE for libsql, which panics (measured).

The backfill bullet in .changeset/19978-sqlite-wasm-text-roundtrip.md: TRUE. The bytes were measured at head on sql.js and better-sqlite3. The "before" was measured raw.

Every other sentence of the 19978 changeset: TRUE against head. That covers the title and Clause-②: no, the 1.14.1 and lockfile claim, the write bullet (-1 length, 61 / 6162), the read bullet, and the "values already on disk", +CAST, equality, getBlob and "does not change" bullets.

PR body (it does not ship; the findings are still recorded):

  • "TursoDriver in local mode (libsql, url :memory:)" is FALSE: that configuration runs better-sqlite3.
  • "A comparand that starts with U+0000 therefore matches every row" holds for $contains only. $startsWith with U+0000 matches nothing on all three faces (measured).
  • The CI figures are stale at head.
  • Everything else in "Deliberate correction of a pending changeset" is TRUE, including the verbatim Old / New quotes and the dist claim.

Implemented-by: claude/issue-19978-sqlite-wasm-text-roundtrip
Reviewed-by: session_01Bvd69VPa6puiNzzPUroDBx

VERDICT: FAIL

What must change:

  1. The 19912 note, S2: stop naming libsql. The measured engines are better-sqlite3 (SqlDriver, and TursoDriver in local mode) and sql.js (SqliteWasmDriver). S3's "on all three" then reads as faces and stands.
  2. The sql-driver.ts parenthetical: name only the measured engines, better-sqlite3 and sql.js, and not libsql.
  3. The PR body (the seat's edit): the libsql label, the $startsWith sentence, and the CI figures.
  4. Optional: soften S1's "on any engine", because the sqlite3 client is unmeasured.

After the patch, a new same-head record is needed under ruling 1A. Nothing in ① or ② needs to move.

Isolated reviewer: a contract-review-tier subagent, fed only the card, the PR, ruling 1A and AGENTS.md; the seat adopted its record.

… sentences — TursoDriver local runs on better-sqlite3, not libsql

Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: 98cb908738599a72538e5f32df4531dc357e61d2

Scope read:

Probes ran on a detached scratch worktree at head, on a turbo build of the three drivers' dependency closure. The wasm dist was confirmed to carry readExactRow and exactTextBindings. The driver-sql dist was confirmed to carry the corrected comment and not the old one. No derived gate family was re-run.

① Derived judgments

  1. The prior record's ①1–①7 carry over: RIGHT.
    • git diff 6a195b37c 98cb90873 touches exactly .changeset/19912-json-backfill-depth-limit.md (line 10) and sql-driver.ts (lines 11311–11312).
    • Across the whole PR, sql-driver.ts has zero changed lines outside a * doc-comment line. A comment-stripped minified transpile of it at merge base and at head is byte-identical (167,241 bytes each).
  2. Text round-trip on SqliteWasmDriver at head: RIGHT.
    • Cells planted by cast(x'…' as text) read back verbatim through the built dist: 610062, EFBBBF78 and 68656C6C6F. FF78 and 61C3 read back with U+FFFD.
    • Raw sql.js 1.14.1, without the driver, still shows the pre-fix loss.
  3. The json-backfill change on this face is declared: RIGHT.
    • On all three local faces, the second initObjects rewrote EFBBBF78 to 22EFBBBF7822, 610062 to 22615C75303030306222 and the control to 2268656C6C6F22.
    • It left FF78 and 61C3 as stored, and a third pass changed nothing.
    • The 19978 changeset's backfill bullet declares exactly this.
  4. sql-driver.ts changes comments only: RIGHT (item 1).
  5. TursoDriver local mode's engine, the item the prior FAIL turned on: RIGHT at this head.
    • Every arm of TursoDriver.toKnexConfig hands Knex client: 'better-sqlite3', and the docblock says so.
    • At runtime, new TursoDriver({ url: ':memory:' }) reports knex.client.driverName = better-sqlite3 and isSqlite = true.
    • Neither touched note names libsql (0 hits).
  6. Public surface: RIGHT. Neither driver's src/index.ts is in the diff, and packages/spec has 0 files in it. sqljs-exact-text.ts is imported only by knex-wasm-dialect.ts.
  7. CI at head: RIGHT. The final poll found 41 check runs, all on this head: 34 success, 5 skipped, 2 failure.
    • Every required context is success: Lint & Repo Gates, TypeScript Type Check, Test Core, Dogfood Regression Gate, Build Core, Temporal Conformance (live PG + MySQL) and Governed Surface Queue Guard.
    • Both failures are Check Changeset, from two workflow runs on this head. Each is annotated on .changeset/19912-json-backfill-depth-limit.md in the DELIBERATE CORRECTION class.
    • The skips are Auto Label, Build Docs, one of two Check PR Size runs, Console Pin Gate and Packed-tarball smoke (opt-in).

② Semver level

Still consistent, and nothing needs to move.

  • @objectstack/driver-sqlite-wasm: patch with Clause-②: no.
  • No export is added or removed.
  • No accepted input is newly refused: a statement with no NUL-bearing binding returns the same SQL string and the same bindings array by reference, and parameter numbering follows SQLite's rule.
  • The only new throw is for a sql.js build without Statement.getBlob. The declared ^1.14.1 ships it in all three builds (measured).
  • The 19912 note's frontmatter (driver-sql: minor, driver-turso: patch) is untouched. driver-sql's own edit is comment-only, so it needs no note of its own.

③ Boundary flags

The corrected note: .changeset/19912-json-backfill-depth-limit.md. It is pending and landed with PR #19972. On line 10, one merge-base sentence is rewritten into three (S1–S3), and S4 and S5 are unchanged. The clause it removes ("on SqliteWasmDriver, a legacy text with a leading U+FEFF or an embedded NUL (sql.js drops both when it reads the text)") is false at this head (①3). So DELIBERATE CORRECTION is the right class, and ruling 1A applies.

S1. "A cell the engine does not read back verbatim is left as stored and keeps reading as it did, where the old statement rewrote it: text holding invalid UTF-8, measured on better-sqlite3 and sql.js."

  • Checked by code: convergeJsonColumn compare-and-sets on the text it read.
  • Measured on the three faces: after the backfill and a third pass, FF78 and 61C3 keep their stored hex, and the raw read gives U+FFFD before and after. json_valid answers 0 for both, which is the old statement's pre-filter.
  • Verdict: TRUE.

S2. "SqlDriver on better-sqlite3, TursoDriver in local mode (which runs on better-sqlite3 too) and SqliteWasmDriver (sql.js) were each measured to read such a cell back with U+FFFD in place of the invalid bytes, so the text the rewrite would be decided from is not the stored text, and the cell is left alone."

  • Checked: the engine attribution in source and at runtime (①5), the three-face measurement, and that the compare-and-set matched nothing on both invalid cells on all three faces.
  • Verdict: TRUE.

S3. "A legacy text with a leading U+FEFF or an embedded NUL is read back verbatim on all three faces, and is rewritten like any other plain string."

  • Checked: on all three faces, the raw read equals the planted hex, and the backfill wrote 22EFBBBF7822 and 22615C75303030306222, the same rule that turned the control into 2268656C6C6F22.
  • Verdict: TRUE.

S4 (unchanged). "Each rewrite is a compare-and-set on the text it was decided from, …": the code's whereRaw('rowid = ? and typeof(??) = 'text' and ?? = ?', …) runs in one transaction per page, and idempotence was measured. Verdict: TRUE.

S5 (unchanged). "This covers every local SQLite face that inherits the backfill: …": by code, isSqlite covers SQLITE_EMIT_CLIENTS, SqliteWasmDriver overrides isSqlite to true, and Turso local hands Knex better-sqlite3. isSqlite measured true on all three faces. The sentence claims coverage, which the code decides. The sqlite3 client is not installed. Verdict: TRUE.

sql-driver.ts parenthetical at head, "(better-sqlite3 and sql.js, the engines measured, read invalid UTF-8 back as U+FFFD)": measured on both engines. Verdict: TRUE.

.changeset/19978-sqlite-wasm-text-roundtrip.md (unchanged in round 2): every sentence re-judged at this head is TRUE. That covers the lockfile's sql.js@1.14.1, the -1 bind that stores 61 and 6162 (measured raw), the NUL-stopping, BOM-dropping read (measured raw, and in the source of both builds), the exact read and the +CAST binding, SQLite's numbering, the unchanged statements, whole-value equality, the backfill bullet, getBlob in all three builds with no fallback, and "a value already cut short stays cut short".

PR body (it does not ship; the findings are still recorded):

  • The Old and New quotes match the merge-base and head files verbatim (checked programmatically). The engine labelling is correct. The restated libsql readings were re-measured directly.
  • Acceptance notes, bullet 1: the "starts with U+0000" sentence is TRUE (measured on both engines). The bullet's opening sentence, "matches as if the comparand ended at the U+0000", is FALSE as written. The GLOB pattern is cut at the U+0000 together with its trailing *: $contains 'a'+U+0000 answers only the row 'a'+U+0000+'b', not every row containing 'a'.
  • Gates: "The pending changesets name libsql 0 times" is FALSE as written. Seven other pending notes name libsql. It is true of the two notes this PR touches.
  • The prior FAIL's four required changes are all present at this head.

Implemented-by: claude/issue-19978-sqlite-wasm-text-roundtrip
Reviewed-by: session_01Bvd69VPa6puiNzzPUroDBx

VERDICT: PASS

No shipped sentence is false at this head. The one change to the accept set that this fix causes elsewhere, driver-sql's json backfill now converging BOM/NUL legacy cells on the sql.js face, is declared in the 19978 note. Under ruling 1A, this record names the corrected note and judges each rewritten sentence. It is the confirmation of the DELIBERATE CORRECTION red. The two PR-body findings do not ship, and the seat corrects them in the body.

Isolated reviewer: a contract-review-tier subagent, second read on this PR, fed only the card, the PR, the prior record, ruling 1A and AGENTS.md; adopted by the seat.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Pending release-note correction on this PR: Check Changeset is red by design and confirmed

domain:engine#1, session_01Bvd69VPa6puiNzzPUroDBx, written 2026-09-24T18:10Z.


Generated by Claude Code

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review September 24, 2026 18:13
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Sep 24, 2026
Merged via the queue into main with commit fc6ddb8 Sep 24, 2026
47 of 50 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-19978-sqlite-wasm-text-roundtrip branch September 24, 2026 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/l tests tooling

Projects

None yet

1 participant