fix(driver-turso): remote syncSchemasBatch registers read coercion and runs the canonical backfill

.changeset/19844-turso-remote-boot-read-coercion.md

@@ -0,0 +1,21 @@
+---
+"@objectstack/driver-turso": patch
+---
+
+A remote Turso deployment now reads, writes and filters the objects it synced at boot by their declared field types. "Remote" means a `libsql://`, `https://`, `http://`, `wss://` or `ws://` URL with no `syncUrl`, or an explicit `mode: 'remote'` (#19844).
+
+The engine's boot schema sync (`ObjectQLPlugin`) reaches this driver through `syncSchemasBatch`, because the driver declares `supports.batchSchemaSync`. On the remote transport that method ran the DDL and stopped. It skipped the field-type registration that its sibling doors, `syncSchema` and `initObjects`, run afterwards. For every object a remote app synced at boot, that meant:
+
+- **Reads came back as stored.** A declared `boolean` read back as `1`/`0`, and a `json` field as its stored text. A `datetime`, `time` or `date` field read back exactly as stored, for example as an offset-bearing string or epoch text rather than the canonical `…Z` form. The `created_at` / `updated_at` audit columns were a partial exception: a zone-naive cell shaped `YYYY-MM-DD HH:MM:SS` or `YYYY-MM-DDTHH:MM:SS`, optionally with a fractional second, was read as UTC and read back canonical (the column default writes the first shape); any other cell read back as stored, including one ending in `Z` or in an offset such as `+08:00` (so a canonical cell stays canonical), an epoch number or its text, and anything that does not parse as a date. This held for records read through the driver and through the engine's `find` / `findOne`, including the rows an `afterFind` hook receives. A CEL expression or an in-memory `$ne: true` filter evaluated over such a record, such as `field != true`, was therefore true even for a stored `true`. Write-side hook contexts did see `true`/`false`, because the engine converts declared booleans there: the `afterInsert` / `afterUpdate` results and the `previous` record on update and delete hooks.
+- **Writes were not converted either.** A `datetime` or `time` value in any spelling other than the canonical one (with an offset, a zone-naive wall clock, an epoch number) was stored as sent. A `Date` given to a `datetime` was the exception: it was stored in canonical form. A `date` given as a `Date` or a full timestamp was stored as a full timestamp. An object or array in a `json` field was stored as it would have been anyway. A scalar `json` value (a string, number or boolean) was stored without its JSON encoding.
+- **Filters compared text as spelled.** A filter on a `datetime` or `time` field compared the stored text with the comparand exactly as the caller wrote it, converting neither side. Rows whose cell or comparand used another spelling of the same value were missed or matched wrongly. For example, a bare-day upper bound `$lte: '2025-07-28'` left out that day's rows stored as ISO text.
+- **Paging was not deterministic.** A paged read with no `orderBy` got no `id` tie-breaker, so walking the pages could serve one row twice and skip another. The driver logged `Paged read of '…' is NOT deterministic`.
+
+`syncSchemasBatch` now finishes the way the other two doors do. It registers each object's field types, keyed by the `object` name it was given, and then runs the canonical temporal backfill once for the whole batch. A DDL failure still rejects before anything is registered. Reads, writes and filters on those objects now convert exactly as they do through `syncSchema` and `initObjects`.
+
+What happens on disk at the first boot after upgrading: the one write this change adds is that backfill, which the `syncSchema` and `initObjects` doors already ran. It rewrites `datetime` and `time` cells stored in a non-canonical spelling, including any this door wrote unconverted, into the canonical spelling of the same value. It leaves alone a cell it cannot safely read as a time. Nothing else on disk is touched, so two kinds of cells this door wrote unconverted stay as they are:
+
+- A `date` stored as a full timestamp reads back as its calendar day, but an equality filter on that day does not match it.
+- A scalar `json` value stored without its encoding reads back as whatever its text parses to. A stored `true` reads back as `1`, and a numeric-looking string reads back as a number.
+
+Local and embedded-replica deployments are unaffected.

packages/drivers/driver-turso/src/turso-driver.ts

@@ -359,8 +359,9 @@ function refuseRemoteTransaction(door: string, detail: string): never {
  * only when the method is absent) never fired — while every remote schema door
  * (`syncSchemasBatch`, the engine's boot sync; `syncSchema` / `initObjects`)
  * routes through `RemoteTransport`, which performs the DDL immediately, and the
- * latter two also run the #5770 canonical temporal backfill, which rewrites
- * stored rows. Measured (`turso-remote-deferred-ddl.test.ts`): the deferral was
+ * latter two also ran the #5770 canonical temporal backfill, which rewrites
+ * stored rows (the batch door runs it too since #19844). Measured
+ * (`turso-remote-deferred-ddl.test.ts`, before this refusal existed): the deferral was
  * accepted, CREATE/ALTER ran on every door, the backfill rewrote rows on two of
  * them, and preview and flush both answered `[]` — a dry run that changed the
  * database and then reported no pending work.
@@ -1698,9 +1699,10 @@ export class TursoDriver extends SqlDriver {
    *
    * It also records the object as one whose table this driver created, which is
    * the whole input to {@link paginationTieBreaker} in remote mode. That goes
-   * FIRST and outside the `try`: both callers reach here only after the DDL has
-   * already succeeded, so the table exists with its `id` primary key whether or
-   * not the best-effort coercion registration below does.
+   * FIRST and outside the `try`: its only caller, {@link completeRemoteSchemaSync},
+   * runs only after the DDL has already succeeded, so the table exists with its
+   * `id` primary key whether or not the best-effort coercion registration below
+   * does.
    */
   private registerRemoteFieldMetadata(obj: { name: string; fields?: Record<string, any>; tenancy?: any }): void {
     this.remoteManagedObjects.add(obj.name);
@@ -1711,6 +1713,39 @@ export class TursoDriver extends SqlDriver {
     }
   }
 
+  /**
+   * The post-DDL half every REMOTE schema door owes, in its one order: register
+   * each synced object's field metadata, then run the canonical temporal
+   * backfill ONCE for the whole call.
+   *
+   * All three remote doors (`syncSchema`, `initObjects`, `syncSchemasBatch`)
+   * send their DDL through `RemoteTransport` and so never reach
+   * `SqlDriver.initObjects`, which is what fills the read-coercion registries
+   * and runs the Knex backfill on the local faces. Each door has to finish the
+   * job itself, and they drifted apart once: `syncSchemasBatch` — the door
+   * `ObjectQLPlugin`'s boot sync takes whenever `supports.batchSchemaSync`
+   * holds, so every remote-Turso boot — returned straight after its DDL. A
+   * booted remote app then read a boolean back as `1` and JSON as a string, got
+   * no `id` tie-breaker on a paged read, and never converged its temporal
+   * columns (#19844). One helper called by all three is what keeps them from
+   * drifting again.
+   *
+   * Callers reach here only after their DDL resolved, so a DDL failure throws
+   * before anything is registered and no object is recorded as a table this
+   * driver created unless it exists. Registration precedes the backfill because
+   * the backfill reads it to learn which columns are temporal. The backfill
+   * probes every column it finds in one round-trip, so calling it once per call
+   * rather than once per object is what keeps a boot's steady state at a single
+   * round-trip.
+   */
+  private async completeRemoteSchemaSync(
+    objects: Array<{ name: string; fields?: Record<string, any>; tenancy?: any }>,
+  ): Promise<void> {
+    if (objects.length === 0) return;
+    for (const obj of objects) this.registerRemoteFieldMetadata(obj);
+    await this.backfillRemoteCanonicalTemporalQuietly();
+  }
+
   /**
    * Converge this driver's REMOTE `Field.datetime` / `Field.time` columns on the
    * canonical storage form and mark the ones that are PROVED converged, so their
@@ -2014,14 +2049,10 @@ export class TursoDriver extends SqlDriver {
     this.assertRemoteTransactionUnsupported(options, 'syncSchema');
     if (this.isRemote) {
       await this.remoteTransport!.syncSchema(object, schema);
-      // See initObjects(): populate the read-coercion registries for remote mode.
-      // Key strictly by `object` (what find()/formatOutput look up) — never let a
+      // Registration + canonical backfill, see completeRemoteSchemaSync(). Key
+      // strictly by `object` (what find()/formatOutput look up) — never let a
       // stray `schema.name` shadow it.
-      this.registerRemoteFieldMetadata({ ...(schema as Record<string, any>), name: object });
-      // #5770: the remote twin of the `backfillCanonicalDatetimes` call
-      // `SqlDriver.initObjects` makes at exactly this point. Must run AFTER the
-      // registration above — that is what tells it which columns are temporal.
-      await this.backfillRemoteCanonicalTemporalQuietly();
+      await this.completeRemoteSchemaSync([{ ...(schema as Record<string, any>), name: object }]);
       return;
     }
     return super.syncSchema(object, schema, options);
@@ -2064,17 +2095,11 @@ export class TursoDriver extends SqlDriver {
         objects.map((obj) => ({ object: obj.name, schema: obj })),
       );
       // Remote DDL bypasses SqlDriver.initObjects, which is what normally
-      // populates the boolean/json/date/numeric read-coercion registries.
-      // Register the field-type metadata explicitly (no DDL) so remote reads
-      // run the same formatOutput() coercion as local/replica mode — otherwise
-      // a boolean reads back as raw 0/1, JSON as a string, dates as raw text.
+      // populates the boolean/json/date/numeric read-coercion registries and
+      // runs the canonical temporal backfill. Without the registration a
+      // boolean reads back as raw 0/1, JSON as a string, dates as raw text.
       // (Root cause of the 2026-07-06 case_escalation `1 != true` incident.)
-      for (const obj of objects) this.registerRemoteFieldMetadata(obj);
-      // #5770: the remote twin of the `backfillCanonicalDatetimes` /
-      // `backfillCanonicalTimes` calls `SqlDriver.initObjects` makes per table.
-      // One batched probe covers every column synced here, so the steady state
-      // (nothing to converge) costs a single round-trip for the whole boot.
-      await this.backfillRemoteCanonicalTemporalQuietly();
+      await this.completeRemoteSchemaSync(objects);
       return;
     }
     return super.initObjects(objects);
@@ -2084,14 +2109,24 @@ export class TursoDriver extends SqlDriver {
    * Batch-synchronize multiple schemas in a single round-trip.
    *
    * In remote mode, delegates to `RemoteTransport.syncSchemasBatch()` which
-   * uses `client.batch()` to submit all DDL as one network call.
+   * uses `client.batch()` to submit all DDL as one network call, then finishes
+   * exactly as the other two remote doors do (see
+   * {@link completeRemoteSchemaSync}). This is the door `ObjectQLPlugin`'s boot
+   * sync takes on this driver, so it is the one that decides what a booted
+   * remote app reads back.
    * In local/replica mode, falls back to sequential `syncSchema()` calls
    * (Knex + better-sqlite3 is already local, so batching has no benefit).
    */
   async syncSchemasBatch(schemas: Array<{ object: string; schema: unknown }>, options?: DriverOptions): Promise<void> {
     this.assertRemoteTransactionUnsupported(options, 'syncSchemasBatch');
     if (this.isRemote) {
-      return this.remoteTransport!.syncSchemasBatch(schemas);
+      await this.remoteTransport!.syncSchemasBatch(schemas);
+      // Key strictly by `object`, as syncSchema() does: it is the name the
+      // engine hands every later read and write for this table.
+      await this.completeRemoteSchemaSync(
+        schemas.map(({ object, schema }) => ({ ...(schema as Record<string, any>), name: object })),
+      );
+      return;
     }
     // Local/replica fallback: sequential sync (already fast with local SQLite)
     for (const { object, schema } of schemas) {