Merge remote-tracking branch 'origin/main' into claude/issue-21007-aggregation-filter-json-equality

Author: claude

.changeset/20821-plan-deferred-ddl-reads.md

@@ -0,0 +1,29 @@
+---
+'@objectstack/driver-sql': patch
+---
+
+fix(driver-sql): `os migrate plan` on a database that does not exist yet no longer prints `DATABASE_ERROR` for the tables whose DDL it deferred (#20821)
+
+`os migrate plan` (and the boot of `os migrate apply`) runs with the SQL driver's DDL deferred: the driver records every table as pending `create_table` and creates none of them. The same boot then reads `sys_metadata`, `sys_metadata_activation` and `sys_migration`. On a new database those tables do not exist yet, so each read was refused, and each refusal printed a line like this on the driver's warn channel (stderr by default):
+
+```text
+[sql-driver] DATABASE_ERROR — the backend refused a read on 'sys_metadata' (SQLITE_ERROR) ... no such table: sys_metadata
+```
+
+Nothing was wrong: every reader already answers from the refusal, and the plan lists the same tables as pending creates. A dry run on a new database printed six of these lines.
+
+The driver now sends such a refusal to the logger's `debug` channel instead of `warn`, when all three hold:
+
+- this driver has DDL deferred;
+- the refused statement targets a table whose DDL this driver deferred;
+- the shared `isMissingTableError` predicate from `@objectstack/types` recognises the refusal as that table being missing.
+
+The default logger has no `debug`, so the line is not printed. The refusal is still thrown to the caller with the same envelope (`DATABASE_ERROR`, status 500), and the plan's output is unchanged.
+
+What still warns:
+
+- every other refusal on a deferred driver, such as a malformed statement on a table that exists;
+- a missing table that the driver did not defer, such as a table nothing in the boot declares;
+- every refusal once the deferred DDL has been applied, or on a driver that never deferred any.
+
+There is nothing to migrate.

.changeset/20901-step18-rationale-form-view-subform-columns.md

@@ -0,0 +1,10 @@
+---
+'@objectstack/spec': patch
+---
+
+fix(spec): protocol 18's migration rationale now covers the form view's inline grid columns and the identity-only currency `scale` refusal (#20901)
+
+**`@objectstack/spec`**
+
+- **`MIGRATIONS_BY_MAJOR[18].rationale` gains one fragment, `form-view-subform-columns-closed`.** It is the paragraph `os migrate meta --step` shows for the protocol 17 → 18 hop. The new sentences say that a form view's `subforms[].columns` now takes the strict `InlineGridColumnSchema` a relationship field's `inlineColumns` takes, that the conversion `form-view-subform-columns-canonicalized` respells a `{ field }` column as `{ name }` in stored rows and assembled artifacts while an author writing `field` is refused, and that `defineStack` refuses `scale` on a column that declares no `type` when its `name` is a `currency` field of a child object declared in the same stack (`inline-grid-column-identity-only-currency-scale-refused`).
+- Text only: no schema, conversion or migration entry changes, and `conversionIds` and `semantic` for step 18 are unchanged.

.changeset/20919-core-import-runner.md

@@ -0,0 +1,17 @@
+---
+'@objectstack/core': minor
+---
+
+feat(core): the bulk-import runner, its row coercion, the mapping apply and the field-meta map now live in `@objectstack/core`, beside `bulkWrite` (#20919)
+
+`runImport` (with `sanitizeRowError` and its option/result types), the cell
+coercion (`coerceRow`, `coerceFieldValue`, `parseDateCell`, `parseNumberCell`,
+`parseBooleanCell`, `matchOption`, `splitMulti`, `isBlank`), the `mapping`
+artifact pipeline (`applyMappingToRows`, `refuseUnknownMappingTargets`,
+`MappingArtifactLike`, `MappingFailure`, `ApplyMappingOptions`) and the field
+metadata map (`buildFieldMetaMap`, `ExportFieldMeta`) are exported from
+`@objectstack/core`. They moved here unchanged from `@objectstack/rest` so the
+connector sync executor in `@objectstack/service-automation` writes through the
+same runner as the HTTP import door without depending on the HTTP layer. Nothing
+to change for consumers: `@objectstack/rest` re-exports every name it exported
+before.

.changeset/20919-rest-re-exports.md

@@ -0,0 +1,11 @@
+---
+'@objectstack/rest': patch
+---
+
+refactor(rest): the import runner, coercion, mapping apply, field-meta map and error classification moved to `@objectstack/core` / `@objectstack/types`; `rest` re-exports them (#20919)
+
+`runImport`, `coerceRow`, `buildFieldMetaMap` and their types (from the package
+index), and `mapDataError` with its sibling classification exports, are now
+re-exported from their new homes — byte-identical code, the same names, the same
+behaviour at both import routes and at `plugin-auth`'s identity import. Nothing to
+change for consumers.

.changeset/20919-service-automation-connector-pull.md

@@ -0,0 +1,31 @@
+---
+'@objectstack/service-automation': minor
+---
+
+feat(service-automation): the connector sync executor pulls a `mapping`'s `connectorSource` and writes it through the import runner (#20919)
+
+`AutomationServicePlugin.pullConnectorSource({ mapping, context })` (and the
+exported `pullConnectorSource(deps, opts)`) reads the mapping through the
+protocol's `getMetaItem`, resolves `connectorSource.connector` to a declared
+(`connectors[]`) `rest` or `openapi` instance, makes ONE call to its read action,
+takes the array at `recordsPath` (default `body`), projects it through the
+mapping's `fieldMapping` and writes it with `@objectstack/core`'s `runImport` —
+the import door's coercion, `mode` / `upsertKey` matching and per-row verdicts,
+with the door's defaults for every knob `connectorSource` does not declare.
+
+- **Watermark, read from the target.** For `connectorSource.watermark`, the
+  starting point sent as `query[watermark.param]` is the highest value already
+  stored in the target field a `fieldMapping` entry copies `watermark.field` onto
+  (transform `none`). Nothing else stores it.
+- **One response per pull.** The connector's paging is not followed.
+- **Loud refusals,** each a `ConnectorPullError` (`code`, `status`, `reason`)
+  raised before anything is written: a plugin-registered or unregistered
+  connector, a degraded instance, a provider other than `rest` / `openapi`, an
+  undeclared action, `update` / `upsert` with an empty `upsertKey`, a
+  `javascript` transform, an unmapped `watermark.field`, an `ok: false` answer,
+  a non-array at `recordsPath` and a non-object record.
+- **Nothing schedules a pull** — a `job` will drive it; the caller supplies the
+  execution context.
+
+The plugin now records the provider of each declared connector instance it
+materializes, which the executor reads.

.changeset/20919-spec-connector-source-live.md

@@ -0,0 +1,18 @@
+---
+'@objectstack/spec': patch
+---
+
+docs(spec): `mapping.connectorSource` is pulled when a job drives it — the describes say where the watermark is read from and that a pull reads one response (#20919)
+
+The `connectorSource` describe no longer says the pull is not executed: the
+connector sync executor in `@objectstack/service-automation` reads the binding,
+and nothing schedules a pull until the `job` stage lands. `watermark.field` now
+states that the next pull's starting point is read from the TARGET field a
+`fieldMapping` entry copies it onto (an unmapped one is refused at pull time), and
+`watermark` states the one-response limit: the connector's paging is not followed,
+so a paged endpoint yields its first page only. The liveness ledger's
+`connectorSource` rows are `live` and keep `authorWarn` (nothing schedules a pull
+yet). The retired `connector.syncConfig` prescription and the
+`connector-sync-keys-retired` upgrade entry say the same, and the entry's
+acceptance criterion no longer claims the connector is validated at authoring.
+No key, value or default changed.

.changeset/20919-types-data-error-classification.md

@@ -0,0 +1,17 @@
+---
+'@objectstack/types': minor
+---
+
+feat(types): the data-error classification table (`mapDataError` and its judgements) is exported from `@objectstack/types` (#20919)
+
+The classification half of `@objectstack/rest`'s error boundary — `mapDataError`
+(a thrown error → the `{ status, body }` ADR-0112 answer, without emitting it),
+`declaredHttpStatus`, `declaredServerFaultAnswer`, `sandboxBusinessMessage`,
+`boundedDeclaredUserMessage`, `boundedDeclaredRefusalMessage` and
+`isEngineDuplicateRecordEnvelope` — moved here unchanged, beside the primitives it
+composes, so the bulk-import runner in `@objectstack/core` judges a failed row with
+the same table the REST door answers with. `@objectstack/rest` keeps the emitters
+and re-exports every name it exported before. The module also exports the eight
+helpers the REST emitters compose (`truncateClientMessage`, `thrownCodeFields`,
+`withoutDeclaredCodePrefix`, `withDeclaredUserMessage`, `isSandboxOrigin`,
+`isSandboxCrash`, `fiveXxArmDisplacesDeclared4xx`, `structuredCodeAnswer`).

content/docs/references/data/mapping.mdx

@@ -49,7 +49,7 @@ const result = ImportFieldMappingSchema.parse(data);
 | **fieldMapping** | `{ source: string \| string[]; target: string \| string[]; transform: Enum<'none' \| 'constant' \| 'lookup' \| 'split' \| 'join' \| 'javascript' \| 'map'>; params?: object }[]` | ✅ |  |
 | **mode** | `Enum<'insert' \| 'update' \| 'upsert'>` | optional (default: `"insert"`) |  |
 | **upsertKey** | `string[]` | optional | Fields to match for upsert (e.g. email) |
-| **connectorSource** | `{ connector: string; action: string; input?: Record<string, any>; recordsPath?: string; … }` | optional | Pull binding: the rest/openapi connector this mapping pulls rows from (one-way, full or timestamp-incremental; a `job` sets the cadence). Declared; the pull is not executed yet |
+| **connectorSource** | `{ connector: string; action: string; input?: Record<string, any>; recordsPath?: string; … }` | optional | Pull binding: the rest/openapi connector this mapping pulls rows from (one-way, full or timestamp-incremental; a `job` sets the cadence). Pulled when a job drives it; nothing schedules it yet |
 | **_lock** | `Enum<'none' \| 'no-overlay' \| 'no-delete' \| 'full'>` | optional | Item-level lock — controls overlay & delete (ADR-0010). |
 | **_lockReason** | `string` | optional | Human-readable reason shown when a write is refused by _lock. |
 | **_lockSource** | `Enum<'artifact' \| 'package' \| 'env-forced'>` | optional | Layer that set _lock (artifact \| package \| env-forced). |
@@ -75,7 +75,7 @@ const result = ImportFieldMappingSchema.parse(data);
 | **action** | `string` | ✅ | Key of the connector action that reads the records: `request` on a `rest` connector, an operation key on an `openapi` connector |
 | **input** | `Record<string, any>` | optional | Fixed input passed to the action on every pull (e.g. `{ method: 'GET', path: '/contacts' }` for `rest`) |
 | **recordsPath** | `string` | optional | Dot path, within the action's result, to the array of records (e.g. `body.results`); omitted ⇒ the result's `body` is the array |
-| **watermark** | `{ field: string; param: string }` | optional | Timestamp-incremental pull; omitted ⇒ every pull reads the full set |
+| **watermark** | `{ field: string; param: string }` | optional | Timestamp-incremental pull; omitted ⇒ every pull reads the full set. Either way a pull makes ONE action call and reads ONE response: the connector's paging is not followed, so a paged endpoint yields its first page only — and an incremental pull over a newest-first paged endpoint moves its starting point past the pages it never read |
 
 
 ---