Skip to content

Commit 72860fc

Browse files
committed
Merge remote-tracking branch 'origin/main' into claude/issue-20901-inline-grid-column-carriers
2 parents 30067c4 + def279a commit 72860fc

72 files changed

Lines changed: 3299 additions & 1221 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
---
2+
'@objectstack/spec': minor
3+
---
4+
5+
feat(spec)!: connector-attached sync leaves the connector — `syncConfig` and `fieldMappings` are retired, and a `mapping` gains the `connectorSource` pull binding (#20281)
6+
7+
**BREAKING** — `connector.syncConfig` (the `DataSyncConfig` block: `strategy`,
8+
`direction`, `realtimeSync`, `timestampField`, `conflictResolution`, `batchSize`,
9+
`deleteMode`, `filters`) and `connector.fieldMappings` (the `ConnectorFieldMapping`
10+
list: `source`, `target`, `defaultValue`, `dataType`, `required`, `syncMode`) are
11+
removed from `ConnectorSchema` and `DeclarativeConnectorEntrySchema` — so from
12+
`defineConnector`, `stack.connectors[]`, the `PUT /api/v1/meta/connector/:name` door
13+
and `AutomationEngine.registerConnector`. The `DataSyncConfigSchema`,
14+
`SyncStrategySchema`, `ConnectorConflictResolutionSchema` and
15+
`ConnectorFieldMappingSchema` exports (and their `DataSyncConfig`, `SyncStrategy`,
16+
`ConnectorConflictResolution`, `ConnectorFieldMapping` types and `…Parsed` aliases)
17+
leave `@objectstack/spec/integration` with them. ADR-0049, ruled ENFORCE on the
18+
maintainer's criterion for a declared-but-unenforced family, with the definition
19+
MOVED: every mainstream platform binds a sync to its TARGET, not to the connection.
20+
21+
Measured before removal: no engine ever ran a connector-attached sync or moved a
22+
value through a connector field mapping — outside the spec package `syncConfig`
23+
appeared only in two comments and `fieldMappings` nowhere, the automation service's
24+
declared-connector item carried neither key, and the def a provider registers is its
25+
own. The `latest_wins` and `soft_delete` defaults read as configured policy and did
26+
nothing; the platform has no soft delete.
27+
28+
**Added (declared, not yet executed):** `mapping.connectorSource` — the pull
29+
binding on the target side, beside the mapping's existing `targetObject`,
30+
`fieldMapping`, `mode` and `upsertKey`: `connector` (a `rest` or `openapi`
31+
connector instance), `action` (the action that reads the records), optional
32+
`input`, optional `recordsPath`, and an optional `watermark` (`field` on the
33+
record, `param` on the request) for a timestamp-incremental pull. Version 1 is a
34+
one-way pull. It carries no cadence (a `job` sets that), no credential (the
35+
connector instance holds it) and no delete or conflict policy. Nothing executes it
36+
in this release, and `os validate` / `os build` warn when it is authored.
37+
38+
### FROM → TO
39+
40+
| removed | what to write instead |
41+
| --- | --- |
42+
| `connector.syncConfig` | delete the key. A sync you still want is a `mapping` on its target object, with `connectorSource` naming the connector and its read action, `watermark` for an incremental pull, and a `job` for the cadence. `direction`, `conflictResolution` and `deleteMode` have no counterpart: the pull is one-way and writes through `mode` / `upsertKey`. |
43+
| `connector.fieldMappings` | delete the key, and carry its `source` → `target` pairs into that mapping's `fieldMapping` (a `defaultValue` becomes a `constant` transform). |
44+
| `DataSyncConfigSchema`, `SyncStrategySchema`, `ConnectorConflictResolutionSchema`, `ConnectorFieldMappingSchema` and their types | no replacement — nothing parsed a sync or a connector field mapping into anything that ran. |
45+
46+
**The one-line fix: delete `syncConfig:` and `fieldMappings:` from every connector.**
47+
`os migrate meta --from 17` lists the mechanical edits for existing sources.
48+
49+
⚠️ Runtime behaviour is deliberately **unchanged**: no connector sync ever ran.
50+
What changes is the answer an author gets — both keys are refused at parse with a
51+
prescription naming the target-side binding, and in `tsc` (their input type is
52+
`never`), instead of being saved with no effect.
53+
54+
### The retirement kit
55+
56+
- **Tombstones.** `syncConfig` and `fieldMappings` are `retiredKey()` tombstones on
57+
the private `ConnectorBaseSchema` both published carriers wrap (the schema is not
58+
`.strict()`, so a bare deletion would be a silent strip, ADR-0104).
59+
`RETIRED_KEYS_BY_MAJOR[18]`: `integration/Connector:syncConfig`,
60+
`integration/Connector:fieldMappings`,
61+
`integration/DeclarativeConnectorEntry:syncConfig` and
62+
`integration/DeclarativeConnectorEntry:fieldMappings`. Neither key had a default,
63+
so no retired-default residue is owed.
64+
- **Four defs leave whole** (`RETIRED_DEFS_BY_MAJOR[18]`): `integration/DataSyncConfig`,
65+
`integration/SyncStrategy`, `integration/ConnectorConflictResolution`,
66+
`integration/ConnectorFieldMapping`. The two `RENAMED_DEFS` entries that targeted
67+
them left the rename table.
68+
- **D2 conversion `connector-sync-keys-removed`** (step 18, retired from the load
69+
path): strips both keys from `connectors[]` and from stored `sys_metadata`
70+
connector rows (the rehydration seam replays it), one notice per key, as a
71+
lossless delete. It never writes a `mapping`: a pulled mapping would start writes
72+
that never happened.
73+
- **D3 entry `connector-sync-keys-retired`** carries the family's judgement: which
74+
syncs should now exist as target-side mappings, and what an author who relied on
75+
`export`, `bidirectional`, `soft_delete` or a conflict policy does without them.
76+
- **No deprecation window**, per the project's startup-stage posture.
77+
78+
⚠️ **The out-of-repo consumer population is NOT MEASURED.** `@objectstack/spec` is
79+
published, so this is breaking for consumers no telemetry was consulted for.
80+
81+
Clause-②: yes (narrowing)
82+
83+
<!-- adr-0087: registered connector-sync-keys-removed, connector-sync-keys-retired -->
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
"@objectstack/objectql": minor
3+
"@objectstack/spec": minor
4+
"@objectstack/lint": patch
5+
"@objectstack/service-analytics": patch
6+
---
7+
8+
fix(objectql,spec)!: a `groupBy` on a multi-value field and a `count_distinct` on a JSON-stored field are refused with `INVALID_FIELD` / 400 at the engine's `aggregate`, on every driver, and the aggregate × field-type table stops accepting `count_distinct` over the JSON-stored types
9+
10+
Clause-②: no (narrowing)
11+
12+
<!-- adr-0087: not-required (already-registered dataset-measure-aggregate-field-type-refused) the one metadata-facing half of this change is a row of AGGREGATE_FIELD_TYPE_COMPATIBILITY narrowing, and that family's hand-migration is already registered under protocol major 18 by this id: "an aggregate the field's type accepts, per AGGREGATE_FIELD_TYPE_COMPATIBILITY", with every refused pair of the table refused at the compile door. The non-temporal sum / avg narrowing rode the same id the same way; this diff amends that entry's surface and acceptance prose to name the count_distinct rider, and corrects the min / max entry's route that called count_distinct valid over every type. The engine-door halves refuse a query shape, not a stored one: no authorable key, export or stored row moves. -->
13+
14+
**BREAKING** (`@objectstack/objectql`): this narrows what `aggregate` accepts, in two positions, on every driver and for every caller that reaches the engine (the REST query door, a flow or hook, and the analytics strategy that lowers a cube query onto `engine.aggregate`). Shipped as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes.
15+
16+
- A `groupBy` entry that names a **multi-value** field: an inherently-multi option type (`multiselect`, `checkboxes`, `tags`), or a `select`, `lookup`, `user`, `file` or `image` field declared `multiple: true`. Both entry spellings are judged, the field name and the `{ field }` object.
17+
- A `count_distinct` aggregation over a **JSON-stored** field: a structured-JSON type (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`), an inherently-multi option type, or a multi-capable field declared `multiple: true`.
18+
19+
**BREAKING** (`@objectstack/spec`): `AGGREGATE_FIELD_TYPE_COMPATIBILITY.count_distinct` no longer lists the ten JSON-stored types (the structured-JSON seven and `multiselect`, `checkboxes`, `tags`), so `isAggregateCompatibleWithFieldType('count_distinct', type)` answers `false` for them. Every reader of the table refuses those pairs now: the dataset-measure lint rule (`measure-aggregate-field-type-refused`, run by `os validate` and at a runtime dataset save), the analytics dataset compile leg (`400 DATASET_INVALID`), and the engine door above. The `count` row is unchanged.
20+
21+
**What an author sees now.** `400 INVALID_FIELD`, naming the position (`groupBy[0]`, `groupBy[0].field`, or `aggregations[0].field`), the field and its declaration, saying the query was not run, and naming the route inside the first 500 characters the REST door keeps. For a multi-value field the route is to filter by one member: `where` with `$contains` on the field, one query per member. For a structured-JSON field it is to store the part you count in a field of its own, or to count rows with `count`. The thrown error carries `field`, `fields`, `object` and `param` (`groupBy` or `aggregations`).
22+
23+
**Why a refusal.** Every SQL driver stores these values in a JSON column, and the drivers share no meaning for one as a group key or a distinct key. Measured through `POST /api/v1/data/:object/query` over three rows: grouping by any of the eight multi-value declarations answered one group per array on the in-memory driver, one group per serialized array on SQLite, and 500 `DATABASE_ERROR` on PostgreSQL 16. `count_distinct` over any structured-JSON or multi-value field answered 3 on the in-memory driver (equal values counted apart), 2 on SQLite (serialized text compared), and 500 on PostgreSQL (no equality operator for `json`). No example app and no published stack groups by a multi-value field or counts one distinct, so no meaning is defined for either here.
24+
25+
**What to write instead.** A dataset measure or a query that counted a JSON-stored field distinct: use `count` over it, or store the scalar part you meant to count in a field of its own and `count_distinct` that field. A grouping by a multi-value field: filter by each member with `$contains` and count.
26+
27+
**Who is affected.** A caller that grouped by a multi-value field, or counted a JSON-stored field distinct, on the in-memory driver or on SQLite and read the answer as a real one; on PostgreSQL both were already a 500. A dataset whose measure pairs `count_distinct` with a JSON-stored field is refused by the lint rule and the compile leg.
28+
29+
**Unchanged.** (Two shapes the structured-JSON `groupBy` entry of this same release lists as unchanged are narrowed here: a `multiple: true` select as a group key, and `count_distinct` over a structured-JSON field. This entry is the later word on both.) A `groupBy` or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; the `having`, filter and sort positions; and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached.
30+
31+
`@objectstack/lint`: the dataset-measure refusal's hint no longer says `count_distinct` accepts every type.
32+
33+
`@objectstack/service-analytics`: the dataset compile leg's refusal of a `count_distinct` measure over a JSON-stored field says why it diverges (the drivers compare the values for equality three ways) and prescribes `count`, or a scalar field for the part being counted; its other refusals no longer say `count_distinct` accepts every type.
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
---
2+
'@objectstack/cli': patch
3+
---
4+
5+
fix(cli): `os package publish` sends `visibility` only when `--visibility` is passed, so re-publishing no longer moves a `marketplace` package to `org`
6+
7+
Clause-②: no
8+
9+
The `--visibility` flag defaulted to `org`, and the CLI always sent it in the package
10+
upsert (`POST /api/v1/cloud/packages`). That upsert is also the re-publish path, and
11+
the control plane updates an existing package's visibility whenever the request carries
12+
one. So re-publishing a new version of a `marketplace` package without repeating
13+
`--visibility marketplace` silently set it to `org` and took it out of the marketplace.
14+
15+
The flag no longer has a default, and an omitted flag is an omitted key:
16+
17+
- **Re-publish without the flag:** the package keeps its current visibility.
18+
- **First publish without the flag:** the control plane applies its own default
19+
(`org` on ObjectStack Cloud), so a new package gets the same visibility as before.
20+
- **With the flag:** unchanged. `--visibility private|org|marketplace` is sent and
21+
applied, as before.
22+
23+
The publish summary's `Visibility` line shows the control plane's answer. When the
24+
control plane does not report it and no flag was given, the line says
25+
`not reported by the control plane`. The "visibility is marketplace but the version is
26+
still draft" hint now follows that answer too, so it also fires when a `marketplace`
27+
package is re-published without the flag. The `--submit` help text now states its
28+
precondition as the package's visibility being `marketplace` (already stored, or set
29+
with `--visibility marketplace`), since the flag is no longer sent on every publish.

‎content/docs/deployment/cli.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2061,13 +2061,13 @@ when both files' `url`s agree.
20612061
| `--display-name <name>` | — | Name shown in the Marketplace (default `artifact.manifest.name`) |
20622062
| `--description <text>` | — | Short package description |
20632063
| `--category <slug>` | — | Marketplace category slug (`crm`, `hr`, `devtools`, …) |
2064-
| `--visibility <level>` | — | `org` (default, installable across your organization) · `private` (explicit grants only) · `marketplace` (public after review) |
2064+
| `--visibility <level>` | — | `org` (installable across your organization) · `private` (explicit grants only) · `marketplace` (public after review). Omitted: not sent, so the control plane decides — an existing package keeps its visibility, a new one gets the control plane's default (`org` on ObjectStack Cloud) |
20652065
| `--org <id>` | `OS_ORG_ID` | `owner_org_id`. Required with a bearer key in service mode; ignored in user mode |
20662066
| `--env <id>` | `OS_ENVIRONMENT_ID` | Environment to install the new version into. Defaults to the environment [`os environments switch`](#cloud-environments) — or `os environments create --activate` — recorded **for this control plane** in `~/.objectstack/cloud.json` |
20672067
| `--install` | — | Auto-install into `--env` after publishing. With no `--env`, no `$OS_ENVIRONMENT_ID` and no active environment for this control plane, it reports that and publishes without installing |
20682068
| `--seed-sample-data` | — | Include sample data in that auto-install |
20692069
| `--pre-release` | — | Mark the version as a pre-release (also inferred — see below) |
2070-
| `--submit` | — | Submit the new version for marketplace review. Needs `--visibility marketplace` and a complete listing |
2070+
| `--submit` | — | Submit the new version for marketplace review. Needs the package's visibility to be `marketplace` (already stored, or set with `--visibility marketplace`) and a complete listing |
20712071
| `--auto-approve` | — | Platform admin only: skip the review queue and publish straight to the public catalog |
20722072
| `--readme <markdown>` | — | Inline marketplace README. Mutually exclusive with `--readme-file` |
20732073
| `--readme-file <path>` | — | README file, read at publish time. Mutually exclusive with `--readme` |

‎content/docs/deployment/publish-and-preview.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,7 @@ Common flags:
165165
| `--server`, `-s` | `OS_CLOUD_URL` | Cloud control-plane URL |
166166
| `--token`, `-t` | `OS_CLOUD_API_KEY` | Bearer token (service mode) |
167167
| `--version`, `-v` | — | Semver version (default: `artifact.manifest.version`) |
168-
| `--visibility` | — | `org` (default) · `private` · `marketplace` |
168+
| `--visibility` | — | `org` · `private` · `marketplace`. Omitted: the control plane decides — an existing package keeps its visibility, a new one gets its default (`org` on ObjectStack Cloud) |
169169
| `--org` | `OS_ORG_ID` | Owner org id (service mode) |
170170
| `--env` | `OS_ENVIRONMENT_ID` | Environment to install the new version into. Defaults to the environment `os environments switch` (or `os environments create --activate`) recorded for this control plane in `~/.objectstack/cloud.json` |
171171
| `--install` | — | Auto-install the new version into `--env` after publishing |

‎content/docs/deployment/validating-metadata.mdx‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -206,9 +206,11 @@ measures: [{ name: 'avg_closed', aggregate: 'avg', field: 'closed_at' }]
206206
`avg` over a temporal column is where this bites hardest: one SQL family coerces
207207
the stored text and returns a plausible number (the average *year*), another has
208208
no such function and fails at query time. `min`/`max` over the same field are
209-
**accepted** — they return a real instant of the field's own type — and
210-
`count`/`count_distinct` are accepted over every type, because they read no
211-
arithmetic off the value. The analytics service refuses the same pair with
209+
**accepted** — they return a real instant of the field's own type — `count` is
210+
accepted over every type, because it reads no value, and `count_distinct` over
211+
every type but the JSON-stored ones (the structured-JSON types and
212+
`multiselect` / `checkboxes` / `tags`), whose values no two backends compare
213+
for equality alike. The analytics service refuses the same pair with
212214
`400 DATASET_INVALID` when a query is built; this is the identical verdict,
213215
from the identical table, one door earlier. The rule stays silent wherever the
214216
field's type cannot be resolved (an object this stack does not define, a

‎content/docs/references/data/mapping.mdx‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ const result = ImportFieldMappingSchema.parse(data);
4949
| **fieldMapping** | `{ source: string \| string[]; target: string \| string[]; transform: Enum<'none' \| 'constant' \| 'lookup' \| 'split' \| 'join' \| 'javascript' \| 'map'>; params?: object }[]` | ✅ | |
5050
| **mode** | `Enum<'insert' \| 'update' \| 'upsert'>` | optional (default: `"insert"`) | |
5151
| **upsertKey** | `string[]` | optional | Fields to match for upsert (e.g. email) |
52+
| **connectorSource** | `{ connector: string; action: string; input?: Record<string, any>; recordsPath?: string; … }` | optional | Pull binding: the rest/openapi connector this mapping pulls rows from (one-way, full or timestamp-incremental; a `job` sets the cadence). Declared; the pull is not executed yet |
5253
| **_lock** | `Enum<'none' \| 'no-overlay' \| 'no-delete' \| 'full'>` | optional | Item-level lock — controls overlay & delete (ADR-0010). |
5354
| **_lockReason** | `string` | optional | Human-readable reason shown when a write is refused by _lock. |
5455
| **_lockSource** | `Enum<'artifact' \| 'package' \| 'env-forced'>` | optional | Layer that set _lock (artifact \| package \| env-forced). |
@@ -66,6 +67,16 @@ const result = ImportFieldMappingSchema.parse(data);
6667
| **transform** | `Enum<'none' \| 'constant' \| 'lookup' \| 'split' \| 'join' \| 'javascript' \| 'map'>` | optional (default: `"none"`) | |
6768
| **params** | `{ value?: any; valueMap?: Record<string, any>; separator?: string }` | optional | |
6869

70+
### Nested Shape: `Mapping.connectorSource`
71+
72+
| Property | Type | Required | Description |
73+
| :--- | :--- | :--- | :--- |
74+
| **connector** | `string` | ✅ | Name of the connector instance the rows are pulled from — a `connectors[]` entry whose `provider` is `rest` or `openapi` (version 1) |
75+
| **action** | `string` | ✅ | Key of the connector action that reads the records: `request` on a `rest` connector, an operation key on an `openapi` connector |
76+
| **input** | `Record<string, any>` | optional | Fixed input passed to the action on every pull (e.g. `{ method: 'GET', path: '/contacts' }` for `rest`) |
77+
| **recordsPath** | `string` | optional | Dot path, within the action's result, to the array of records (e.g. `body.results`); omitted ⇒ the result's `body` is the array |
78+
| **watermark** | `{ field: string; param: string }` | optional | Timestamp-incremental pull; omitted ⇒ every pull reads the full set |
79+
6980

7081
---
7182

0 commit comments

Comments
 (0)