fix(cli): os migrate unmapped-columns refuses a value JSON cannot carry as stored

Binary bytes, a bigint and a non-finite number reach a JSON document as a
stand-in a conversion would write as the value. The read now refuses them in
both faces, exit 1, naming the column and the record id, and emits no
record; a Date passes as its ISO 8601 text. No column the platform creates
for a field type answers with one, measured on SQLite and PostgreSQL. The
--json refusal path passes exit 1 to emitJson, as the family does.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016GiHYRmLSNWTfbX9gVQkpz

Author: claude

.changeset/21573-migrate-unmapped-columns.md

@@ -9,8 +9,8 @@ Clause-②: yes (widening)
 - **What it reads.** The columns `os migrate plan` reports as `unmapped_column` for one object's table: a column that is still in the table and that no metadata declares, typically one a retired field left behind. The column set is the plan's own findings, from the same differ on the same read-only boot, so the command never reads a column the plan does not report. Each record is emitted as `{ id, values }`. `--json` prints one document, `{ database, object, table, columns, count, records, duration }`. The text face lists the columns and each record's values.
 - **Why it exists.** A read or a write through the engine now serves an object's declared fields only, and naming an undeclared column is refused. An app that moves a retired field's values into the field that replaced it reads them once with this command, writes them with its own script, and then drops the columns with `os migrate apply --allow-destructive`. That is the route the read and write narrowing in `@objectstack/objectql` names for this case.
 - **Operator-only and read-only.** It runs under the database credentials you pass (`--database-url`, else `OS_DATABASE_URL`, else the project database), and it reads every organization's rows. No REST route, API flag or per-request option serves these values, and the runtime doors are unchanged. It boots the way `os migrate plan` does: no schema DDL, no seed data, and no database file created.
-- **Values as stored.** An unmapped column has no declared type, so each value is emitted as the database client returns it, with no field-type decoding.
-- **Answers.** An object with no unmapped column, or with no table yet: empty work, exit 0. No SQL driver: `os migrate plan`'s own `no_sql_driver` answer, exit 0. An undeclared object name: `OBJECT_NOT_FOUND`, exit 1. An object the plan does not diff (federated, or bound to another datasource): refused, exit 1. A read that cannot be complete, such as one stopped by `--max-records`: refused, exit 1, and no partial set is emitted.
+- **Values as stored.** An unmapped column has no declared type, so each value is emitted as the database client returns it, with no field-type decoding; a PostgreSQL `timestamp` arrives as a date and is emitted as its ISO 8601 text. A value JSON cannot carry as stored (binary bytes, a `bigint`, or a non-finite number) is refused in both faces with exit 1, naming the column and the record id, and no record is emitted: read that column with the database's own client. No column the platform creates for a field type answers with one of these, on SQLite or on PostgreSQL.
+- **Answers.** An object with no unmapped column, or with no table yet: empty work, exit 0. No SQL driver: `os migrate plan`'s own `no_sql_driver` answer, exit 0. An undeclared object name: `OBJECT_NOT_FOUND`, exit 1. An object the plan does not diff (federated, or bound to another datasource): refused, exit 1. A read that cannot be complete, such as one stopped by `--max-records`: refused, exit 1, and no partial set is emitted. A value JSON cannot carry as stored: refused, exit 1, as above.
 - `MigrateUnmappedColumnsCommand` is exported from `@objectstack/cli` beside the other `os migrate` commands.
 
 Nothing that ran before changes. This is a new command.

content/docs/deployment/cli.mdx

@@ -1082,14 +1082,19 @@ them into the declared fields with your own script, then run
 - **Values as stored.** An unmapped column has no declared type, so each value is
   emitted as the database client returns it, with no field-type decoding: a
   retired `json` field on SQLite reads as its stored text, a retired `boolean`
-  as `0` or `1`.
+  as `0` or `1`, and a retired `datetime` on PostgreSQL arrives as a date and is
+  emitted as its ISO 8601 text. A value JSON cannot carry as stored (binary
+  bytes, a `bigint`, or a non-finite number) is refused with exit 1, naming the
+  column and the record id, and no record is emitted; read that column with
+  the database's own client.
 - **Empty work, exit 0**, for an object with no unmapped column, or with no table
   in this database yet. `--json` prints one document:
   `{ object, table, columns, count, records: [{ id, values }] }`.
 - **Refused, exit 1**: an object name the deployment does not declare
   (`OBJECT_NOT_FOUND`); an object `plan` does not diff (federated, or bound to
-  another datasource), whose empty answer would be unmeasured; and a read that
-  cannot be complete, such as one stopped by `--max-records`. A partial set is
+  another datasource), whose empty answer would be unmeasured; a read that
+  cannot be complete, such as one stopped by `--max-records`; and a value JSON
+  cannot carry as stored, described above. A partial set is
   never emitted, because a conversion over part of a table, followed by the
   drop, loses the rest.
 

packages/cli/src/commands/migrate/unmapped-columns.integration.test.ts

@@ -25,7 +25,11 @@
  *  4. an object name the registry does not hold: `OBJECT_NOT_FOUND`, exit 1,
  *     one document;
  *  5. a row cap the table exceeds: refused, exit 1, and no records emitted;
- *  6. the door writes nothing: the schema and every row are byte-identical
+ *  6. a value JSON cannot carry as stored (bytes in a BLOB column added by
+ *     hand to `um_blob`; the platform creates no binary column for any field
+ *     type): refused in both faces, exit 1, naming the column and the record
+ *     id, no record emitted;
+ *  7. the door writes nothing: the schema and every row are byte-identical
  *     after it ran.
  *
  * The runtime half, that the engine's data door never serves these columns, is
@@ -64,6 +68,7 @@ const RELEASE_ONE = {
       fields: { name: { type: 'text' }, mailing_street: { type: 'text' }, mailing_city: { type: 'text' } },
     },
     { name: 'um_account', fields: { name: { type: 'text' } } },
+    { name: 'um_blob', fields: { name: { type: 'text' } } },
   ],
 };
 
@@ -73,6 +78,7 @@ const RELEASE_TWO = {
   objects: [
     { name: 'um_contact', fields: { name: { type: 'text' } } },
     { name: 'um_account', fields: { name: { type: 'text' } } },
+    { name: 'um_blob', fields: { name: { type: 'text' } } },
   ],
 };
 
@@ -95,7 +101,8 @@ function childEnv(overrides: Record<string, string | undefined>): Record<string,
 
 /**
  * Release one, served: DDL performed, three contacts and an account written
- * while the mailing fields are declared. Then the hash shadow, by hand.
+ * while the mailing fields are declared. Then, by hand, the hash shadow and a
+ * BLOB column on `um_blob` holding three bytes.
  */
 const SEED_CHILD = `
 const rt = await import('@objectstack/runtime');
@@ -115,11 +122,14 @@ await ql.insert('um_contact', { id: 'c1', name: 'Ann', mailing_street: '1 Retire
 await ql.insert('um_contact', { id: 'c2', name: 'Bob', mailing_street: '2 Retired Way', mailing_city: 'Newtown' }, SYSTEM);
 await ql.insert('um_contact', { id: 'c3', name: 'Cy' }, SYSTEM);
 await ql.insert('um_account', { id: 'a1', name: 'Acme' }, SYSTEM);
+await ql.insert('um_blob', { id: 'b1', name: 'Bin' }, SYSTEM);
 await kernel.shutdown();
 const { SqlDriver } = await import('@objectstack/driver-sql');
 const raw = new SqlDriver({ client: 'better-sqlite3', connection: { filename: process.env.FIXTURE_DB }, useNullAsDefault: true });
 await raw.knex.raw('ALTER TABLE um_contact ADD COLUMN ${HASH_SHADOW} text');
 await raw.knex('um_contact').update({ ${HASH_SHADOW}: 'shadow' });
+await raw.knex.raw('ALTER TABLE um_blob ADD COLUMN legacy_bytes blob');
+await raw.knex('um_blob').update({ legacy_bytes: Buffer.from([1, 2, 255]) });
 await raw.disconnect();
 process.stderr.write('[fixture] seeded\\n');
 process.exit(0);
@@ -205,6 +215,8 @@ let noneJson: Run;
 let noneHuman: Run;
 let unknownJson: Run;
 let cappedJson: Run;
+let bytesJson: Run;
+let bytesHuman: Run;
 
 beforeAll(async () => {
   dir = mkdtempSync(join(tmpdir(), 'os-21573-'));
@@ -227,6 +239,8 @@ beforeAll(async () => {
   noneHuman = await runCli(door('um_account'));
   unknownJson = await runCli(door('um_nope', '--json'));
   cappedJson = await runCli(door('um_contact', '--max-records', '2', '--json'));
+  bytesJson = await runCli(door('um_blob', '--json'));
+  bytesHuman = await runCli(door('um_blob'));
   after = readState();
   planJson = await runCli(['migrate', 'plan', '--json']);
 }, HOOK_TIMEOUT_MS);
@@ -302,4 +316,16 @@ describe('os migrate unmapped-columns: the other answers', () => {
     expect(doc.error).toMatch(/stopped at 2 row\(s\)[\s\S]*--max-records/);
     expect(doc).not.toHaveProperty('records');
   });
+
+  it('a value JSON cannot carry as stored: refused in both faces, exit 1, naming the column and the record id', () => {
+    const says = /Record b1 of um_blob holds binary bytes in the column legacy_bytes[\s\S]*database's own client/;
+    expect(bytesJson.code).toBe(1);
+    const doc = JSON.parse(bytesJson.stdout);
+    expect(doc.error).toMatch(says);
+    expect(doc).not.toHaveProperty('records');
+    expect(bytesHuman.code).toBe(1);
+    expect(bytesHuman.stdout).toMatch(says);
+    // Neither face carries the stand-in a JSON serialisation of the bytes would be.
+    for (const out of [bytesJson.stdout, bytesHuman.stdout]) expect(out).not.toContain('"type":"Buffer"');
+  });
 });

packages/cli/src/commands/migrate/unmapped-columns.test.ts

@@ -22,13 +22,24 @@
  * reported column (the SQL driver answers a projection naming a missing column
  * with the whole row, measured on SQLite and PostgreSQL) and an answer that is
  * not rows each REFUSE, and none of them returns records.
+ *
+ * ## …and rather than emit a value JSON cannot carry as stored
+ *
+ * Binary bytes, a `bigint` and a non-finite number each reach a JSON document
+ * as a stand-in (an object shaped like the bytes, a thrown serialisation, a
+ * `null`), which a conversion would write as the value. One case per class,
+ * each naming the column and the record id, and a control that a `Date`, a
+ * parsed json object and a string pass unchanged. What the drivers hand back
+ * for the columns the platform itself creates is measured on the pull request:
+ * none of them is in the refused set.
  */
 
 import { describe, it, expect } from 'vitest';
 import type { ManagedDriftEntry } from '@objectstack/driver-sql';
 import {
   unmappedColumnsOf,
   readUnmappedColumnValues,
+  unrepresentableKind,
   type UnmappedColumnReader,
 } from './unmapped-columns.js';
 
@@ -124,14 +135,52 @@ describe('readUnmappedColumnValues — every row, keyed by record id, values as
     expect(Object.keys(records[0]!.values)).not.toContain('name');
   });
 
-  it('decodes nothing: a JSON-looking string, a 0/1 and a Buffer reach the record as the driver handed them', async () => {
-    const blob = Buffer.from([1, 2, 255]);
-    const reader = rowsReader([{ id: 'c1', legacy_flags: '{"a":[1,2]}', legacy_on: 1, legacy_blob: blob }]);
-    const [record] = await readUnmappedColumnValues(reader, 'contact', ['legacy_flags', 'legacy_on', 'legacy_blob']);
+  it('decodes nothing: a JSON-looking string, a 0/1, a Date, a parsed json object and a string pass unchanged', async () => {
+    const at = new Date('2026-01-02T03:04:05.000Z');
+    const parsed = { a: [1, 2], b: 'x' };
+    const reader = rowsReader([
+      { id: 'c1', legacy_flags: '{"a":[1,2]}', legacy_on: 1, legacy_at: at, legacy_json: parsed, legacy_note: 'kept' },
+    ]);
+    const [record] = await readUnmappedColumnValues(reader, 'contact', [
+      'legacy_flags', 'legacy_on', 'legacy_at', 'legacy_json', 'legacy_note',
+    ]);
 
     expect(record!.values.legacy_flags).toBe('{"a":[1,2]}');
     expect(record!.values.legacy_on).toBe(1);
-    expect(record!.values.legacy_blob).toBe(blob);
+    // The same instances: nothing is converted on the way through.
+    expect(record!.values.legacy_at).toBe(at);
+    expect(record!.values.legacy_json).toBe(parsed);
+    expect(record!.values.legacy_note).toBe('kept');
+    // …and each one JSON carries unambiguously: a Date as its ISO 8601 text.
+    expect(JSON.parse(JSON.stringify(record!.values))).toEqual({
+      legacy_flags: '{"a":[1,2]}', legacy_on: 1, legacy_at: '2026-01-02T03:04:05.000Z', legacy_json: parsed, legacy_note: 'kept',
+    });
+  });
+
+  it.each([
+    ['a Buffer (a SQLite blob, a PostgreSQL bytea)', Buffer.from([1, 2, 255]), /holds binary bytes/],
+    ['another ArrayBufferView', new Float64Array([1.5]), /holds binary bytes/],
+    ['a bigint', 9007199254740993n, /holds a bigint/],
+    ['NaN', Number.NaN, /holds a non-finite number \(NaN\)/],
+    ['Infinity', Number.POSITIVE_INFINITY, /holds a non-finite number \(Infinity\)/],
+    ['-Infinity', Number.NEGATIVE_INFINITY, /holds a non-finite number \(-Infinity\)/],
+  ])('REFUSES %s, naming the column and the record id, and emits no record', async (_label, value, says) => {
+    // The bad value sits on the SECOND row: the first is never handed back on its own.
+    const reader = rowsReader([
+      { id: 'c1', legacy: 'fine' },
+      { id: 'c2', legacy: value },
+    ]);
+    const read = readUnmappedColumnValues(reader, 'contact', ['legacy']);
+    await expect(read).rejects.toThrow(says);
+    await expect(readUnmappedColumnValues(reader, 'contact', ['legacy'])).rejects.toThrow(
+      /Record c2 of contact holds [^\n]* in the column legacy, which JSON cannot carry as the database stored it/,
+    );
+  });
+
+  it('the predicate answers null for every value JSON carries as stored', () => {
+    for (const value of [null, 'text', '', 0, -1.5, 42, true, false, new Date(0), { a: 1 }, [1, 'x']]) {
+      expect(unrepresentableKind(value), String(value)).toBeNull();
+    }
   });
 
   it('walks past one page by seeking on id, and reads every row once', async () => {

packages/cli/src/commands/migrate/unmapped-columns.ts

@@ -62,6 +62,28 @@ export interface UnmappedColumnReader {
   find(object: string, query: Record<string, unknown>): Promise<unknown>;
 }
 
+/**
+ * The class of a value JSON cannot carry as the database stored it, or `null`
+ * when it can.
+ *
+ * Each of the three arrives in a JSON document as something else, with nothing
+ * to say so: binary bytes as an object shaped `{ type, data }` (a `Buffer`) or
+ * as an index-keyed object (another typed array), a `bigint` as a thrown
+ * serialisation error, and `NaN` / `±Infinity` as `null`. A conversion script
+ * would write the stand-in into the replacing field and report success, so the
+ * door refuses these instead of emitting them. ⛔ No codec: choosing a
+ * representation would make a representation part of this door's contract.
+ *
+ * A `Date` is not in the set: PostgreSQL hands a `timestamp` column back as
+ * one, and it serialises to its unambiguous ISO 8601 text.
+ */
+export function unrepresentableKind(value: unknown): string | null {
+  if (ArrayBuffer.isView(value)) return 'binary bytes';
+  if (typeof value === 'bigint') return 'a bigint';
+  if (typeof value === 'number' && !Number.isFinite(value)) return `a non-finite number (${String(value)})`;
+  return null;
+}
+
 /**
  * Read every row's unmapped values, keyed by record id.
  *
@@ -83,6 +105,8 @@ export interface UnmappedColumnReader {
  *  - a row comes back without a column the differ reported: the SQL driver
  *    answers a projection naming a missing column with the whole row instead,
  *    and a record emitted without its value would read as converted;
+ *  - a value is one JSON cannot carry as stored ({@link unrepresentableKind}):
+ *    it would be emitted as a stand-in a conversion would write as the value;
  *  - the driver answers something other than an array of rows.
  */
 export async function readUnmappedColumnValues(
@@ -122,6 +146,14 @@ export async function readUnmappedColumnValues(
               'the one asked for. Refusing rather than emitting the record without that value.',
           );
         }
+        const kind = unrepresentableKind(row[column]);
+        if (kind !== null) {
+          throw new Error(
+            `Record ${String(row.id)} of ${object} holds ${kind} in the column ${column}, which JSON cannot ` +
+              'carry as the database stored it. Read that column with the database\'s own client. Refusing ' +
+              'rather than emitting a stand-in a conversion would write as the value; no record was emitted.',
+          );
+        }
         values[column] = row[column];
       }
       records.push({ id: row.id, values });
@@ -182,8 +214,8 @@ export async function readUnmappedColumnValues(
  *  - an object the plan does not diff (federated, or bound to another
  *    datasource): refused, exit 1, because an empty answer there would be
  *    unmeasured rather than clean;
- *  - a read that cannot be complete: refused, exit 1
- *    ({@link readUnmappedColumnValues}).
+ *  - a read that cannot be complete, or a value JSON cannot carry as stored:
+ *    refused, exit 1, no record emitted ({@link readUnmappedColumnValues}).
  */
 export default class MigrateUnmappedColumns extends Command {
   // No tracker id in this string: a command description reaches operators,
@@ -396,7 +428,7 @@ export default class MigrateUnmappedColumns extends Command {
       // a second document after the first.
       if (isExitSignal(error)) throw error;
       if (flags.json) {
-        await emitJson({ error: error?.message ?? String(error), ...errorCodeFields(error) }, 0, { compact: true });
+        await emitJson({ error: error?.message ?? String(error), ...errorCodeFields(error) }, 1, { compact: true });
         this.exit(1);
       }
       printError(error?.message ?? String(error));