Merge remote-tracking branch 'origin/main' into claude/issue-20821-migrate-plan-deferred-reads

Author: claude

.changeset/20620-migrate-meta-review-pairing.md

@@ -0,0 +1,31 @@
+---
+'@objectstack/cli': patch
+---
+
+fix(cli): `os migrate meta --from N` prints the manual change that judges an applied edit beside that edit, marked review
+
+Clause-②: no
+
+Some applied mechanical edits are not the end of the job. A default flip such as
+`flow-decision-mode-inclusive-explicit` writes `mode: 'inclusive'` onto a decision so
+the flow keeps its old behaviour, and the manual change that judges it says the right
+edit is usually none: delete the key where the branch conditions partition. That
+judgment used to print hundreds of lines below the edit, among every other manual
+change of the major.
+
+A manual change can now declare which conversions' edits it judges. In the
+`Applied N mechanical change(s):` group, each run of edits by such a conversion is
+followed by one line:
+
+    ↳ review the N edits above against the manual change [protocol M] surface → replacement
+
+The line copies the headline the manual change prints in its own group, so its `why`
+and `verify` lines are found there under the same text. Two pairs are declared today:
+`flow-decision-mode-inclusive-explicit` with the decision-mode entry, and
+`time-default-utc-suffix-dropped` with the time-default entry. Only declared links
+pair; a manual change that merely mentions a conversion in its prose does not.
+
+Nothing else moves. The `N manual change(s) require your judgment:` group still lists
+every manual change, byte for byte, with the same count. An edit no manual change
+judges prints exactly as before. `--json`, `--out`, the exit code and the loader's
+stderr are unchanged, and `--stored` is not affected.

.changeset/20809-guidance-slot-docblocks.md

@@ -0,0 +1,16 @@
+---
+'@objectstack/spec': patch
+---
+
+docs(spec): the `strictObject` `guidance` option and the `ToolSchema` guidance table no longer describe `guidance` rows as tombstones for retired keys
+
+Clause-②: no
+
+`StrictObjectOptions.guidance` (in the published declarations) and the docblock above
+`TOOL_RETIRED_KEY_GUIDANCE` in `src/ai/tool.zod.ts` (shipped as source) called the slot a
+place for "tombstones for retired keys". A tombstone is `retiredKey()` in the shape, which
+keeps the key declared, and a `guidance` row for a declared key never fires. The docblocks
+now say what the slot is for: prescriptions for keys the shape does not declare —
+wrong-layer pointers, and the upgrade for a spelling removed from the shape.
+
+Text only: no schema, no guidance entry, no prescription and no parse behaviour changes.

.changeset/20896-template-import-door.md

@@ -0,0 +1,23 @@
+---
+'@objectstack/rest': patch
+---
+
+fix(rest): the import template (`GET /api/v1/data/:object/export?template=true`) is gated by the import door's permissions, not the export's
+
+Clause-②: no
+
+The template carries no records — only the columns the caller may write, one
+example row and an instructions sheet — so it answers to whoever may import, and
+the export permission (`allowExport`) neither admits nor refuses it:
+
+- A caller with the create permission on the object gets the template, with or
+  without `allowExport`.
+- A caller without the create permission gets `403 PERMISSION_DENIED`, with or
+  without `allowExport`.
+- An object whose `enable.apiMethods` exposes neither `create` nor `update`
+  answers `405 OBJECT_API_METHOD_NOT_ALLOWED`, as `POST /api/v1/data/:object/import`
+  does. An object that exposes `create` without `list` serves the template.
+- Without `template=true` the export is unchanged: the same two export checks
+  and the same bytes.
+
+To let a role download the template, grant it create on the object.

.changeset/20912-analytics-multi-value-distinct-refused.md

@@ -0,0 +1,27 @@
+---
+"@objectstack/service-analytics": minor
+---
+
+fix(service-analytics)!: a grouped dimension on a multi-value field and a `count_distinct` measure over a JSON-stored field are refused with `INVALID_FIELD` / 400 at the analytics door, before any SQL is built; a dataset `count_distinct` measure over a field declared `multiple: true` is refused at compile time
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (runtime-interface-only packages/services/service-analytics/src/dataset-compiler.ts#DatasetCompileOptions) a published options interface of `compileDataset`, whose one probe member is renamed and now answers the field's declaration instead of its type. It is not a Zod schema, a `packages/spec` declaration or an object definition, it is not a projection of a schema, and no metadata surface references it, so `objectstack migrate meta` has nothing to rewrite; the TypeScript host that passed the old member is told by its compiler at its own call site. The other halves refuse a QUERY shape at the analytics door (a grouping or distinct-count target) and one aggregate x declaration pair at the dataset compile door: no authorable key, spelling or stored row moves (`CubeSchema`, `DatasetSchema` and the analytics query body keep parsing every member), and which scalar a caller meant to group on or count is not something a ledger entry can rewrite. The other categories are closed on facts: the package publishes (not `unpublished`); no ADR-0087 id covers an analytics grouping or distinct-count target or this interface, and this diff adds none (not `registered` / `already-registered`); and the interface was concretely typed at the merge base, not erased (not `type-surface-only`). -->
+
+**BREAKING**: this narrows what the analytics query doors accept, in two positions, and what the dataset compiler accepts, in one. It holds on `POST /api/v1/analytics/query`, on its dry run `POST /api/v1/analytics/sql` and on `POST /api/v1/analytics/dataset/query`, on every driver and on both strategies. It ships as `minor` under the launch-window convention for accept-set narrowings.
+
+- A cube or dataset dimension whose column is a **multi-value** field, when it groups the result (a `dimensions` entry, or a `timeDimensions` entry with a `granularity`): an inherently multi option type (`multiselect`, `checkboxes`, `tags`), or a `select`, `radio`, `lookup`, `user`, `file` or `image` field declared `multiple: true`. The same types without the flag are served.
+- A `measures` entry that resolves to a `count_distinct` measure whose column is **JSON-stored**: a structured-JSON type (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`), an inherently multi option type, or a multi-capable field declared `multiple: true`. An authored cube measure, a suffix-inferred one (`tags_count_distinct`) and a compiled dataset's are judged alike; a `count` measure is not judged.
+- A dataset measure that pairs `count_distinct` with a base-object field declared `multiple: true` is refused `400 DATASET_INVALID` when the dataset compiles, at registration and on the request door, beside the type row that already refused `tags`.
+
+The column is judged where it is declared: on the cube's object, or, for a dotted path, on the object the cube's declared join names.
+
+**What an author sees now.** `400 INVALID_FIELD`, naming the member as the request wrote it, the column, the object, the declaration (`select with multiple: true`), saying the query was not run, and naming the route. For a multi-value field the route is a record query on the declaring object filtered by one member with `$contains`, one query per member. For a structured-JSON field it is to store the scalar part in a field of its own. The thrown error carries `member`, `param` (`dimensions`, `timeDimensions` or `measures`), `cube`, `field` and `object`. The dataset compile refusal names the measure, the field and its declaration with `multiple: true`.
+
+**Why a refusal.** The engine's aggregate door already refuses both shapes, and the native-SQL strategy compiled its own statement and never reached it. Measured through this service as `AnalyticsServicePlugin` composes it over a real engine: a dimension on a `tags`, `multiselect` or `multiple: true` select answered 200 with one group per serialized array on SQLite and 500 `DATABASE_ERROR` on PostgreSQL 16; an inferred `count_distinct` over a `json`, `tags` or `multiple: true` select field answered 2, 3 and 2 on SQLite and 500 on PostgreSQL; a dataset `count_distinct` over the `multiple: true` select registered, then answered 2 on SQLite and 500 on PostgreSQL. The engine-aggregate strategy answered 400 for every one of these, under the engine's position (`groupBy[0]`, `aggregations[0].field`) rather than the member the caller wrote. The predicates are `@objectstack/spec/data`'s, the ones the engine's doors read: `isMultiValueField`, and the aggregate × field-type table's `count_distinct` row.
+
+**Your fix (a host calling `compileDataset` directly).** `DatasetCompileOptions` no longer has `declaredFieldType`. Pass `declaredValueShape` instead: the same read of the field's metadata, answering `{ type, multiple }` (the type, and `multiple === true`) rather than the type alone, or `undefined` when nothing answers. A host that passes neither compiles exactly as before, with no aggregate × field-type refusal at all. `AnalyticsService` and `AnalyticsServicePlugin` wire it themselves from `sourceFieldMeta`.
+
+**Who is affected.** A dashboard, report or caller that grouped by a multi-value field, or counted a JSON-stored field distinct, through the native-SQL strategy on SQLite and read the serialized arrays or the text-compared count as real answers. On PostgreSQL the same queries were already a 500. A dataset that pairs `count_distinct` with a `multiple: true` field no longer registers.
+
+**Unchanged.** A dimension or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; a member naming a column the object does not have, which keeps its existing `INVALID_FIELD` answer first; a dotted path the cube declares no join for; a measure whose `sql` is an expression; and a host that wires no `sourceFieldMeta`, where the declaration cannot be read.

.changeset/20943-cube-member-sql-column-reference.md

@@ -0,0 +1,108 @@
+---
+'@objectstack/spec': minor
+'@objectstack/service-analytics': patch
+---
+
+feat(spec)!: an analytics cube member's `sql` is a column reference — a SQL expression there is refused at parse, and a derived value is declared on an ADR-0021 dataset (#20943)
+
+Clause-②: yes (narrowing)
+
+**BREAKING** — shipped as `minor` under the launch-window convention
+(`check-changeset-no-major` refuses `major` until GA; breaking-ness is carried by
+this banner, the `(narrowing)` arm above and the ADR-0087 disposition below,
+never by the level).
+
+`MetricSchema.sql` and `DimensionSchema.sql` — the `sql` of every member in an
+analytics cube's `measures` and `dimensions` — admit a column reference only: a
+field of the cube's object (`amount`), a relationship path of bare identifiers
+ending in one (`account.amount`, `account.owner.region`), or `'*'` for a count.
+Any other value — a `CASE WHEN …`, an aggregate or a ratio of aggregates, a quoted
+or `$`-prefixed spelling, an empty string — is refused at parse with a
+prescription. This is ADR-0021's "zero raw SQL / zero raw expressions" carried
+from the dataset layer to the cube members it compiles to (maintainer ruling D on
+the card): an expression names no single field, so no platform check can judge
+which fields it reads, and the two analytics strategies never agreed on it — the
+raw-SQL path ran it verbatim and the ObjectQL path refused it. The rule is a
+`pattern` in the published JSON Schema too, so a document validated against
+`json-schema/**` is judged as the parse judges it.
+
+## FROM → TO
+
+A derived value moves to an ADR-0021 dataset over the same object. A conditional
+count or sum is a dataset measure with its own structured `filter`; a ratio, sum,
+difference or product of measures is `derived: { op, of: [...] }` over measures
+named in the same dataset.
+
+```
+FROM  defineCube({ name: 'delivery', sql: 'task', measures: {
+        done_rate: { label: 'Done Rate (%)', type: 'number',
+                     sql: "SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) * 100.0 / COUNT(*)" },
+      } })
+      -> parsed; the expression ran verbatim on one strategy and was refused on the other
+TO    -> ZodError at measures.done_rate.sql (invalid_format): `measures.<metric>.sql` is a
+         column reference: a field of the cube's object (`amount`), a relationship path ending
+         in one (`account.amount`), or `'*'` for a count. A SQL expression there was retired …
+
+      defineDataset({ name: 'task_metrics', label: 'Task Metrics', object: 'task',
+        dimensions: [/* … */],
+        measures: [
+          { name: 'task_count', aggregate: 'count' },
+          { name: 'done_count', aggregate: 'count', filter: { status: 'done' } },
+          { name: 'done_rate', derived: { op: 'ratio', of: ['done_count', 'task_count'] }, format: '0.0%' },
+        ] })
+```
+
+**Mind the scale.** A `derived` ratio is a 0–1 fraction. An expression that
+multiplied by 100 returned percentage points; pair the ratio with a `%` numeral
+pattern (the server marks a ratio column's percent scale as a fraction) and
+re-check any consumer that read the old number raw.
+
+**A dimension that bucketed a column with a CASE expression** has no expression
+form in the cube layer or the dataset layer: group by the column itself, or keep
+the bucket as a field of the object and name that field.
+
+**The one-line fix:** parse each cube; every refusal at `…sql` is one member to
+move — replace it with the column it aggregates, or move the derived value to a
+dataset measure as above, and point the dashboards, reports and queries that named
+`<cube>.<member>` at the dataset measure.
+
+**What an author who still writes it sees.** `CubeSchema`, `defineCube()`,
+`defineStack({ analyticsCubes })` (`STACK_SCHEMA_INVALID` / 422) and the
+`analytics_cube` write door refuse the member at its `sql` path with the
+prescription. `tsc` does not: the key's type is still `string`.
+
+## The retirement kit
+
+- **Schema.** `MetricSchema.sql` / `DimensionSchema.sql` carry the pattern and
+  their prescriptions (`data/analytics.zod.ts`). A column reference parses
+  byte-identically to before. The retired metric `filters` guidance and the
+  analytics query's `filters` guidance no longer offer "fold the condition into
+  the metric's own `sql` expression" as a live channel; the `metric-filters-removed`
+  conversion summary and its D3 entry and step-18 rationale fragment say the same.
+- **ADR-0087.** The D3 entry `cube-member-sql-expression-retired`, with its
+  step-18 rationale fragment. No D2 conversion — an expression has no mechanical
+  rewrite into a dataset — and no `RETIRED_KEYS_BY_MAJOR` row: no key left the
+  shape, so the authorable-surface, api-surface and JSON-schema manifest
+  ratchets are unchanged.
+- **Liveness.** The `analytics_cube` ledger rows `measures.sql` and
+  `dimensions.sql` stay `live`, re-verified, with the narrowing recorded.
+- **Docs.** The `data/analytics` reference page is regenerated.
+- **Example.** The showcase cube's `done_rate` expression member moves to the
+  `showcase_task_metrics` dataset as `done_count` (a count filtered on
+  `status: 'done'`) and `done_rate` (`ratio` over `done_count` and `task_count`,
+  format `0.0%`).
+- **`@objectstack/service-analytics`** (README only): its query-body section no
+  longer tells a reader to fold a per-metric condition into the metric's own
+  `sql` expression. The runtime is unchanged: its expression branches remain for
+  a cube that reaches the service without meeting the parse, and their deletion
+  is a separate change.
+
+## Reach, measured
+
+- This repository: one authored expression member (the showcase `done_rate`),
+  moved here. Test fixtures in `@objectstack/service-analytics` that build
+  expression members WITHOUT the parse keep exercising the runtime's expression
+  branches, unchanged.
+- Out-of-repo authored cubes: NOT MEASURED.
+
+<!-- adr-0087: registered cube-member-sql-expression-retired -->

AGENTS.md

@@ -614,12 +614,12 @@ Even inside your own worktree, operate defensively:
      `git merge-tree` of any `merge=os-regen` path runs the same merge-ort machinery as
      `git merge` and therefore honours the custom driver, while GitHub runs none, so the
      two answer different questions about the same snapshot. Probe from a throwaway bare
-     clone that shares the object store and has no driver registered
-     (`git clone --bare --shared . PROBE.git`, then
-     `git --git-dir=PROBE.git merge-tree --write-tree --name-only BASE HEAD`), ⛔ never
-     with `-c merge.os-regen.driver=` or `-c merge.os-regen.driver=false`, neither of
-     which disables the driver: git runs the configured program, it fails, and every
-     routed path is reported conflicted, including ones that text-merge cleanly.
+     clone with no driver registered that FETCHES both commits (`--shared` may copy only
+     what `refs/heads/**` reach; recipe: `os-regen-merge.sh`'s header): exit 1 with no
+     tree id, `not something we can merge`, is a missing object, never a conflict. ⛔ Never
+     with `-c merge.os-regen.driver=` or `-c merge.os-regen.driver=false`, neither of which
+     disables the driver: git runs the configured program, it fails, and every routed path
+     is reported conflicted, including ones that text-merge cleanly.
    - **Registration is per clone** (`pnpm install` → `prepare` →
      `scripts/setup-git-hooks.mjs`); an unregistered clone falls back to git's default
      text merge — older behaviour, not breakage.

content/docs/permissions/permission-sets.mdx

@@ -95,7 +95,9 @@ opt-in. Grant it through an ordinary position-distributed set instead.
 Enforcement is server-side and covers both egress doors:
 
 - `GET /api/v1/data/:object/export` answers `403 EXPORT_NOT_PERMITTED` before it
-  reads the first row.
+  reads the first row. Its import template (`?template=true`) is not a copy of
+  data — it carries no records — so it is gated by the import instead: the create
+  permission on the object, never `allowExport`.
 - A **report** rendered as `csv` or `json` is the same bulk copy and is gated the
   same way, whether run interactively or delivered by a schedule. `html_table`
   is a rendered view and stays a read.