Repository navigation
fix(driver-turso): remote syncSchemasBatch registers read coercion and runs the canonical backfill - #19863
Conversation
…nd runs the canonical backfill The remote arm of TursoDriver.syncSchemasBatch -- the door ObjectQLPlugin's boot sync takes on this driver -- returned straight after its DDL, while the syncSchema and initObjects remote arms went on to register the read-coercion registries (and record the table as driver-created) and run the canonical temporal backfill. A booted remote app read booleans back as 1, JSON as a string, got no id tie-breaker on paged reads and never converged its temporal columns. All three remote doors now finish through one private helper, completeRemoteSchemaSync: register each object keyed by the object string, then run the backfill once for the call. DDL failures still reject before any registration. Claude-Session: https://claude.ai/code/session_01TEhopqrWQYBycZzyJHpAZr Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift CheckThis PR changes 1 package(s): 4 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 1 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 6 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin f507c7ce1608cf710822f8f24ebb155c97a6c0a4 && git checkout f507c7ce1608cf710822f8f24ebb155c97a6c0a4
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2bbb462335ad617f51ba7fc1a0c872f932f47513 e4bc63e2dc364a56e18cd179e42f62ed592e7d6a && git checkout -B drift-repro 2bbb462335ad617f51ba7fc1a0c872f932f47513 && git merge --no-ff e4bc63e2dc364a56e18cd179e42f62ed592e7d6a
node scripts/docs-audit/affected-docs.mjs --json 2bbb462335ad617f51ba7fc1a0c872f932f47513
|
Contract reviewServed-tier: ① Derived judgmentsHead = PR head; merge-base
② Semver level
③ Boundary flags
Implemented-by: VERDICT: FAIL: ① does not hold. The changeset's temporal bullet and its "stored correctly" sentence are false at head, the remote-scheme list omits Generated by Claude Code |
… reads, writes and filters; pin the write side The changeset now says what a remote app actually saw before the fix: reads returned stored forms, writes of datetime/time/date/scalar-json values were not converted, datetime/time filters compared text as spelled, and paged reads had no tie-breaker. It says the one write added is the canonical datetime/time backfill, and names the two kinds of cells this door wrote that nothing rewrites (full-timestamp date cells, unencoded scalar json). It lists all five remote URL schemes and scopes the boolean guard sentence to CEL / in-memory $ne over a raw record. A write-side pin joins the per-door cases: a datetime written with an offset reaches disk in the canonical spelling. The deferred-DDL suite's header table notes that the batch door now runs the backfill. Claude-Session: https://claude.ai/code/session_01TEhopqrWQYBycZzyJHpAZr Co-authored-by: Claude <noreply@anthropic.com>
Contract reviewServed-tier: ① Derived judgmentsHead = PR head = branch tip; merge-base
② Semver level
③ Boundary flags
Implemented-by: VERDICT: FAIL: one sentence. "Hook contexts were not affected" is false for Generated by Claude Code |
…ontexts and names the afterFind exception engine.find / findOne hand an afterFind hook the rows as the driver returned them, with no boolean conversion, so on the pre-fix batch door an afterFind hook saw 1/0. Only the write-side contexts are converted. The changeset now says that, and notes that the created_at / updated_at audit columns were always presented canonical. Claude-Session: https://claude.ai/code/session_01TEhopqrWQYBycZzyJHpAZr Co-authored-by: Claude <noreply@anthropic.com>
Contract reviewServed-tier: ① Derived judgmentsHead = PR head = branch tip; merge-base
② Semver level
③ Boundary flags
Implemented-by: VERDICT: FAIL: one clause. "they were always presented canonical" is false for a caller-supplied non-canonical audit value. The minimal fix is a one-clause prose edit, in flight. (The clause was added on the seat's own optional suggestion in the previous round.) Generated by Claude Code |
…tform-written values presentAuditTimestampOutput folds only a Date and a zone-naive timestamp string, so on the pre-fix batch door only the values the platform writes itself (the column default, the engine's stamps, a Date) read back canonical. An audit value supplied in another spelling, such as an offset string a preserveAudit import can send, read back as stored. Claude-Session: https://claude.ai/code/session_01TEhopqrWQYBycZzyJHpAZr Co-authored-by: Claude <noreply@anthropic.com>
Contract reviewServed-tier: ① Derived judgmentsHead = PR head = branch tip; merge-base
② Semver level
③ Boundary flags
Implemented-by: VERDICT: FAIL: one clause. "an audit value supplied in another spelling … read back as stored" is false for a supplied zone-naive wall clock. The wording that failed was the previous record's prescription, which described the fold by source; the fix states it by shape, derived from Generated by Claude Code |
…l shape presentAuditTimestampOutput decides by the cell's shape, not by who wrote it: repairNaiveUtcAuditTimestamp reads a zone-naive YYYY-MM-DD[ T]HH:MM:SS with an optional fraction as UTC and returns it canonical, and returns every other cell unchanged (Z- or offset-terminated strings, epoch numbers or text, unparseable values). The changeset's audit exception now says exactly that, in one sentence. Claude-Session: https://claude.ai/code/session_01TEhopqrWQYBycZzyJHpAZr Co-authored-by: Claude <noreply@anthropic.com>
Contract reviewServed-tier: ① Derived judgmentsHead = PR head = branch tip; merge-base with The new audit-column sentence, judged against
② Semver level
③ Boundary flags
Implemented-by: VERDICT: PASS Generated by Claude Code |
…swering "no drift" (objectstack-ai#19891) Fixes objectstack-ai#19845 Clause-②: no (narrowing) ## What changed A remote-mode `TursoDriver` now refuses `detectManagedDrift()` with the transport's `NOT_IMPLEMENTED` / `501` envelope, with or without explicit objects, where it used to answer `[]`. The local and embedded-replica modes inherit the Knex detector unchanged. - `packages/drivers/driver-turso/src/turso-driver.ts`: `refuseRemoteDriftDetection()` beside the deferred-DDL refusal, and a `detectManagedDrift` override in the schema-management section. The override spells the base's parameter shape key for key, as `check:object-def-param-keys` arm C requires. - `packages/drivers/driver-turso/src/turso-remote-drift-detection-refusal.test.ts`: the pin, with local and replica controls. - `.changeset/19845-turso-remote-drift-detection-refusal.md`: `minor`, BREAKING banner, `adr-0087: not-required (no-migration-prescription)`. This is the shape PR objectstack-ai#19842 used for deferred DDL on this face. This is the dispatched landing site. `packages/cli` is untouched. ## Zone 1: the chain is reachable Read on base `1f89ba0d70`: - `packages/cli/src/commands/serve.ts`: with `OS_ARTIFACT_URL` set, `pinnedArtifact` boots through `createDefaultHostConfig`, which calls `createStandaloneStack`. The `com.objectstack.cli.artifact-boot-migration-gate` plugin then runs `runArtifactBootMigrationGate({ driver: findSqlDriverForKernel(kernel) })` on `kernel:ready`. - `packages/runtime/src/standalone-stack.ts`, turso arm: a `libsql://` URL declares the `default` datasource with the Turso factory. `DefaultDatasourcePlugin.init` registers it as `driver.` plus the engine's default driver name. That name is the driver's `name`, `com.objectstack.driver.turso`. - `packages/cli/src/utils/schema-migrate.ts`: `SQL_DRIVER_SERVICES` lists `driver.com.objectstack.driver.turso`. Its duck-type check needs `detectManagedDrift` and `applyMigrationEntries`, and the driver inherits both. Measured once and not committed. The instrument was the real `findSqlDriverForKernel` and `runArtifactBootMigrationGate` from `packages/cli/src/utils/`, over a kernel stub that exposes the driver under `driver.com.objectstack.driver.turso`. Table `t` is synced, then an extra physical column `legacy` is added. | driver | found | gate verdict before the fix | gate verdict after the fix | |:--|:--|:--|:--| | local (`file:` URL) | yes | `ok: false`, 1 destructive entry, so the boot is refused | unchanged | | remote (synced through the batch door) | yes | `ok: true`, 0 entries, **no warning** | `ok: true`, 0 entries, plus the warning `⚠ Could not check the physical schema against the artifact (Schema drift detection is not supported by the Turso REMOTE transport …). Boot continues; …` | Not measured end to end: the `os serve` binary against a live remote libSQL endpoint. The boot-sync door is the batch door, as PR objectstack-ai#19863 measured on an `ObjectKernel` boot. ## A1: reproduced Setup: the `libsql` SQLite double; table `t` declared with `name: text`, plus an extra physical column `legacy`. | face | call | answer | |:--|:--|:--| | local | `detectManagedDrift()` | `t.legacy`: `unmapped_column`, op `drop_column`, `destructive` | | remote | `detectManagedDrift()` | `[]` | | remote | `detectManagedDrift([{ name: 't', fields }])` | `[]` | The remote table's physical columns were `id, created_at, updated_at, name, legacy`, so the drift was on disk. ## A2: how the gate treats a driver that cannot judge - **(i) A "cannot judge" channel exists.** `runArtifactBootMigrationGate` wraps `driver.detectManagedDrift()` in a `try`. On any throw it warns `Could not check the physical schema against the artifact (MESSAGE). Boot continues; run 'os migrate plan' to verify.` and returns `ok: true` with nothing applied. The CLI's own suite pins this: `artifact-boot-migration.test.ts`, "warns and continues when drift detection itself fails". I found no capability flag and no sentinel. The only other channel is `applyMigrationEntries`'s `skipped` list, for a driver that declines an op. - **(ii) A thrown `NOT_IMPLEMENTED` makes the gate warn and continue.** The boot does not stop and nothing crashes. This was measured after the fix; see the Zone 1 table. - **(iii) Real remote introspection is not cheap. The differ can be reused, but the reads that feed it cannot.** - The differ half, measured: tables built by the remote DDL (`plain`, and `rich` with 16 field types, a field-level `unique` and a declared index) were judged by a local Knex connection to the same SQLite file. `detectManagedDrift` reported 0 entries for each, the same as a local-face control. So the shared differ gives no false drift on remote-built tables. - The read half: every read that feeds the differ goes through `this.knex`. That covers `schema.hasTable`, `columnInfo` plus `PRAGMA table_info` ordering, the SQLite arm of `introspectIndexes` (`sqlite_master`, `index_list`, `index_info`) and `probeNullSafeUniqueDuplicates`. A remote version needs a second copy of each of those arms. - The remote managed registry would have to carry fields and indexes (see A4). - Once detection returns entries, the gate calls `applyMigrationEntries` on the safe ones. That inherited Knex path runs on the same placeholder. - A destructive verdict refuses the boot and names `os migrate apply --allow-destructive`. On this face that command refuses at `setDeferredDdl` (PR objectstack-ai#19842), so the refusal text would need a CLI change. The CLI is outside this card's surface. ## A3: the route taken (i) exists, and a refusal through it does not stop any remote boot: the gate warns and continues. (iii) is not cheap and would pull in a CLI change. So this PR takes the loud refusal, declared as a narrowing. The refusal message names the working remedy, running `os migrate plan` against a local SQLite copy (a `file:` URL). The gate embeds that message in its own warning line. ## A4: `managedObjectFields` Yes, the no-argument `detectManagedDrift()` needs it. That is the gate's call, and it iterates `managedObjectFields` / `managedObjectIndexes`. On the remote face these stay empty, because `registerRemoteFieldMetadata` → `registerExternalObject` fills the read-coercion registries and `remoteManagedObjects` only. It is deliberately **not** fed here. The explicit-objects call answered `[]` too, because the Knex `hasTable` probe reads the placeholder, so feeding the registry alone changes no answer. `managedObjectFields` also has other readers: `getManagedFields`, the base `paginationTieBreaker` and `planMediaColumnMove`. A real remote detector would need a remote registry of fields plus indexes. The remote doors already receive `indexes` on the object definition. ## Tests - New pin `turso-remote-drift-detection-refusal.test.ts`, 6 cases: - local and embedded-replica controls, no-argument and explicit calls: each still reports `t.legacy` as `unmapped_column` / `drop_column` / `destructive`; - remote, both call shapes: the call rejects with `code: 'NOT_IMPLEMENTED'`, `status: 501` and the operator-facing first sentence, and sends zero statements to the database. - `pnpm --filter @objectstack/driver-turso test`: 58 files, 1317 tests passed. - `pnpm --filter @objectstack/driver-turso typecheck`: exit 0. `tsc --listFiles` compiles all 58 test files, including the new one. **Ablation**, on HEAD `7d9a7e38c7`, through `scripts/ablation-replace.mjs`: - Mutation: the anchor `if (this.isRemote) refuseRemoteDriftDetection();` goes from x1 to x0, the marker from x0 to x1, and the blob from `4f2aabfcd133` to `87fe978db7c9`. - Result: the 2 remote cases turned red with `expected a refusal, got an answer: []`, which is the original defect. The 4 controls stayed green. - Restore: the blob equals HEAD (`4f2aabfcd133`), and `git diff HEAD` is empty. - No dist step was involved: the test imports `./turso-driver.js` from source. ## Gates, on HEAD `7d9a7e38c7` - `node scripts/pm/dispatch-gates.mjs --commands` (no paths) derived 61 commands. Every one exited 0. - `--ran` verdict: `✓ dispatch-gates --ran: 61 derived famil(ies) accounted for — 61 run, 0 NOT-MEASURED`. - The three dist-reading gates (`check:dual-build-cjs-loads`, `check:lean-entry-closure`, `check:type-check-debt`) first exited 3 (PREREQUISITE NOT MET). They exited 0 after a workspace `turbo run build`. - `check:object-def-param-keys` went red once, on arm C (the override derived the parameter type). It is corrected in `6d1e4e9619` and green on HEAD. - `node scripts/check-issue-citations.mjs` (live): exit 0, `every citation this change adds resolves`. - `pnpm check:driver-conformance`: exit 0, `OK — 50 covered cell(s), 0 in the DEBT ledger, 0 exempt`. - Targeted eslint, measured: - Scope: the 2 changed TS files, both inside the `**/*.{ts,…}` block of `eslint.config.mjs`, run with `--no-inline-config --format json`. - Result: 2 files, 0 errors, 0 warnings. - Why the narrowing is sound: the config enables no type-aware linting (`parserOptions.project` and `projectService` are both unset for these files), so this diff cannot move any untouched file's verdict. The changeset `.md` is outside eslint's population. - Stale-tree note: `origin/main` gained 3 commits after the branch point (metadata, objectql, `scripts/pm/close-cards.mjs`). They are disjoint from this diff, and the derived list came out identical. ## Acceptance notes - The gate's warning ends with the CLI's generic `run 'os migrate plan' to verify`. Pointed at the remote URL, that command refuses, because it arms deferred DDL. Its refusal names the local-copy route, and the driver message embedded earlier in the same warning names that route first. Owner: `domain:cli` (`packages/cli/src/utils/artifact-boot-migration.ts`). Noted, not filed. - After this PR, remote `applyMigrationEntries` still runs the inherited Knex path on the placeholder. No caller in this repo reaches it: the gate gets no entries from a remote detector, and `os migrate apply` is refused at `setDeferredDdl`. There is no repro, so this is noted only. ## Out-of-scope findings, for the seat to file 1. **class (a).** Remote `planMediaColumnMove()` answers `{ plans: [], refusals: [] }`. - Measured on the SQLite double: table `m` with `doc: file` and `pic: image` was synced through the batch door, and its physical `TEXT` columns are present. The remote face returned 0 plans and 0 refusals. The local control returned 2 `unquote` plans. - It is the same class as this card: an inherited schema read on the remote face that answers from the placeholder or the empty `managedObjectFields`. - Reach, read from source and not run: `os migrate files-to-references` boots without deferral and calls `stack.driver.planMediaColumnMove()`. - Dedupe words: `turso remote planMediaColumnMove empty` · `files-to-references remote turso nothing to move` · `remote placeholder knex inherited schema read`. 2. **class (a).** Replica mode built from a remote `url` plus `syncUrl` runs every Knex CRUD against a process-local `:memory:` database. - `detectMode` answers `replica` for this pair, and the last branch of `toKnexConfig` gives it a `:memory:` connection. - Probe on the SQLite double: a table synced and a row created through the driver read back through the driver, while the libsql client's database held no tables at all. - Dedupe words: `turso replica remote url syncUrl memory` · `embedded replica knex memory writes lost` · `turso replica mode libsql url syncUrl`. --- _Generated by [Claude Code](https://claude.ai/code/session_01TEhopqrWQYBycZzyJHpAZr)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
Fixes #19844
Clause-②: no
What was wrong
ObjectQLPlugin.syncRegisteredSchemastakes its batch branch when a driver declaressupports.batchSchemaSyncand implementssyncSchemasBatch, andTursoDriverdoes both. On the remote transport,syncSchemasBatchreturned straight after its DDL. The two sibling remote doors (syncSchema,initObjects) go on to callregisterRemoteFieldMetadataand then the canonical temporal backfill. So the door a remote-Turso boot actually takes was the one door that skipped all three halves: read-coercion registration, the managed-object record, and the backfill.Boot measurement (taken before any edit)
One boot of an
ObjectKernelwithObjectQLPlugin, a remoteTursoDriverregistered as thedriver.tursoservice over thelibsqlSQLite double (libsql-sqlite-stub.testkit.ts), and an app objectwwithflag: boolean,meta: json,at: datetime. It ran as a scratch test and is not committed (see Acceptance notes). Before =origin/main8cbc3c0; after = this branch at d8940d7.syncSchemasBatchtwice (phase 1 and phase 3); nosyncSchema,initObjects,registerExternalObjectorregisterObjectMetadatacallsyncSchemasBatchtwice, each followed byregisterExternalObjectfor all six objects in the batchdriver.findOneflag / meta1(number) / the string{"k":1}true(boolean) /{"k":1}(object)findOneflag / meta1(number) / a stringtrue(boolean) / an objectpaginationTieBreaker('w')nullidbooleanFields.w/jsonFields.w/datetimeFields.wflag/meta/created_at, updated_at, atcanonicalDatetimeFields.wcreated_at, updated_at, atSo the card stands at p1: nothing else on the boot path populated the registries.
The fix
The landing site is the one the card named, the
isRemotearm ofTursoDriver.syncSchemasBatchinpackages/drivers/driver-turso/src/turso-driver.ts. The producer is this driver, so no consumer changes.completeRemoteSchemaSync(objects), registers each object and then runsbackfillRemoteCanonicalTemporalQuietly()once for the call. The registration isregisterRemoteFieldMetadata: theremoteManagedObjectsrecord, plus the coercion, tenant and autonumber registriesregisterExternalObjectfills. An empty list is a no-op.syncSchemapasses one object, keyed by itsobjectargument.initObjectspasses its objects.syncSchemasBatchnow passes each entry as{ ...schema, name: object }, the same strict keyingsyncSchemauses.Not touched:
packages/objectql/src/plugin.ts, which was read only.detectManagedDriftin remote mode belongs to #19845. It is not in this diff, and #19845 remains open.Tests
The new file
packages/drivers/driver-turso/src/turso-remote-batch-door-registration.test.tshas 12 cases:describe.eachover the three remote doors, withsyncSchemasBatch(the boot door) as the case under test andsyncSchemaandinitObjectsas controls. Each syncs the reproduction object{ fields: { flag: boolean, meta: json } }, creates a row and callsfindOne. Expected:flag === true,metadeep-equals{ k: 1 }, and the raw row on disk is still{ flag: 1, meta: '{"k":1}' }, which proves the test is not vacuous.paginationTieBreaker('w')goes fromnullto'id'.datetimewritten as2025-07-28T08:00:00+08:00reaches disk as2025-07-28T00:00:00.000Z.object, never by aschema.namethat differs from it.backfillRemoteCanonicalTemporalis called exactly once for a two-object batch. The naive2025-07-28 00:00:00is rewritten on disk to2025-07-28T00:00:00.000Z, and both columns are marked canonical.Ablation (run once, after the fix was committed)
Mutation: delete only the batch door's
completeRemoteSchemaSync(...)call, usingnode scripts/ablation-replace.mjs. The anchor went from 1 hit to 0, the blob changed from b33d3987b0bc to 5f1d1c909d7d, and an on-disk grep count read 0. Command for both runs:pnpm --filter @objectstack/driver-turso exec vitest run --maxWorkers=2 src/turso-remote-batch-door-registration.test.ts.de31187a96): exit 1,Tests 5 failed | 7 passed (12). The new write pin reds on the batch door only (expected [ { at: '2025-07-28T08:00:00+08:00' } ] to deeply equal [ { at: '2025-07-28T00:00:00.000Z' } ]). The other four failures are the batch-door cases from the first run:expected 1 to be true,expected null to be 'id',expected undefined to deeply equal [ 'flag' ], andexpected "backfillRemoteCanonicalTemporal" to be called 1 times, but got 0 times. ThesyncSchemaandinitObjectscontrols (including their write pins) and the DDL-failure case stayed green.git diff HEADis empty; the restored file is green in the full-suite run below.dist/is involved: the suite imports./turso-driver.jsfrom source.Verification (HEAD d8940d7)
Re-run on
de31187a96after the patch round:pnpm --filter @objectstack/driver-turso testexited 0 (57 files, 1311 tests),typecheckexited 0,node scripts/check-issue-citations.mjsexited 0,node scripts/check-changeset-no-major.mjs --base origin/mainexited 0, andpnpm check:driver-conformanceexited 0. The--commandsderivation was byte-identical (61 commands), and--ranread 59 run and 2 NOT-MEASURED, as below.pnpm --filter @objectstack/driver-turso test: exit 0, 57 files and 1308 tests passed.pnpm --filter @objectstack/driver-turso typecheck: exit 0. The package's tsconfig includessrc/**/*, so the tests are type-checked too.node scripts/pm/dispatch-gates.mjs --commands(no paths) derived 61 commands. All 61 ran, each exit code captured before any pipe. The--ranverdict exited 0:61 derived famil(ies) accounted for — 59 run, 2 NOT-MEASURED.pnpm check:dual-build-cjs-loadsandpnpm check:type-check-debtboth exited 3 (PREREQUISITE NOT MET), because each needs the whole-workspace build. CI runs both over the full build. Declared narrowing for the first:requireof the rebuiltpackages/drivers/driver-turso/dist/index.jsloads (14 exports,TursoDrivera function), exit 0. For the second: driver-turso has no DEBT or TEST_DEBT entry, and its owntsc --noEmitis clean.pnpm check:driver-conformance,pnpm check:object-def-param-keys,pnpm check:issue-citationsandpnpm check:nul-bytes, each exit 0.node scripts/check-issue-citations.mjs, the live diff-scoped verdict: exit 0, with 3 citations judged and all 3 resolving.eslint --no-inline-config --format jsonover the two changed.tsfiles reported 2 files, 0 errors, 0 warnings. The changeset.mdis outside eslint's configured population ("no matching configuration").eslint.config.mjsenables no type-aware linting (noparserOptions.project), so this diff cannot change the verdict on any untouched file. The fullpnpm lintis CI's.Changeset
.changeset/19844-turso-remote-boot-read-coercion.md,@objectstack/driver-turso: patch. It was rewritten in the patch round after the first contract review (FAIL on prose, comment 5794937369). Every claim was re-measured on the SQLite double, including what the pre-fix door wrote and which of those cells the backfill does and does not converge.Acceptance notes
@objectstack/objectqlis not a dependency of@objectstack/driver-turso, and adding one (with its lockfile change) for a single test is outside this card's file surface. The door-level cases make the same call the boot's batch branch makes.turso-remote-deferred-ddl.test.tsrecords thesyncSchemasBatchrow of prediction (b) as "REFUTED, no row write on this door". It was measured before that refusal existed, so it stays historically true; the patch round adds a one-clause footnote saying the door now runs the backfill too.syncSchemaandinitObjectsalready did. The changeset says so, and it names the two kinds of cells the pre-fix door wrote unconverted that no remote backfill converges (adatestored as a full timestamp; a scalarjsonstored unencoded).Generated by Claude Code