Skip to content

Commit 527030b

Browse files
committed
Merge remote-tracking branch 'origin/main' into claude/issue-20102-retire-saved-report-stack
2 parents ecb3bb5 + 6780e34 commit 527030b

47 files changed

Lines changed: 4362 additions & 242 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
"@objectstack/objectql": minor
3+
---
4+
5+
fix(objectql)!: `engine.aggregate({ having })` walks through the shared comparand-shape face, so `having: { total: [5] }` is refused exactly as the same shape in `where` is (#19974)
6+
7+
Clause-②: no (narrowing)
8+
9+
<!-- adr-0087: not-required (already-registered filter-equality-array-comparand-refused, filter-between-blank-endpoint-refused, filter-between-field-reference-endpoint-refused) this change adds no new transition. It puts the `having` clause of `engine.aggregate` behind refusals the shared comparand-shape face already makes for `where` and `aggregations[i].filter` on the same verb. The arms whose transitions are on the ledger are named: the equality-slot array, the blank `$between` endpoint and the `{ $field }` `$between` endpoint. The first two prescriptions apply to a `having` as written, and so does the third one's literal-bound half; its column-to-column half does not, because `having` resolves no `{ $field }` reference in any slot (a gap this change does not touch, stated in the table below). The null list member, the null `$between` endpoint and the null ordering comparand were ruled at the face with no ledger entry (their changesets declared no-migration-prescription), and the non-list `$in` / `$nin` / `$between` refusal is the face's original rule. `having` is a request-only key: no metadata type stores it, so there is no stored document for `objectstack migrate meta` to rewrite. The table below is the author-facing remedy for each arm, not a mechanical rewrite. -->
10+
11+
**BREAKING**: this narrows what `having` accepts on `engine.aggregate` (and on the REST aggregate query that forwards it there). A `having` carrying one of the shapes below used to answer; it is now refused with `INVALID_FILTER` / 400, before any driver is asked for a row. The refusal is the shared face's own message, byte for byte the refusal the same shape gets in `where`, with the path rooted at `having` instead of `where`. It ships as `minor` under the launch-window convention for accept-set narrowings.
12+
13+
The 2026-09-23 ruling on #19757 refuses an array in the equality slot at the shared comparand-shape face (`assertListComparandShapes` in `@objectstack/spec/data`) "for every driver at once". The face already ran on `where` and on each `aggregations[i].filter`. `having` never reaches a driver: the engine evaluates it itself after aggregation, on both the native `driver.aggregate()` path and the in-memory fallback. That evaluator answered every shape the face refuses. Measured on the base through `engine.aggregate` on `driver-memory` and `driver-sqlite-wasm`, over three groups with totals 500, 1250 and 20:
14+
15+
| you wrote in `having` | what it did before | write instead |
16+
|:--|:--|:--|
17+
| `{ total: [500] }` or `{ total: { $eq: [500] } }`, at any depth under `$and` / `$or` / `$not` | kept the 500 group, because JS `500 == [500]` is true; under `$not` it kept the complement, the 1250 and 20 groups | `{ total: 500 }`, or `{ total: { $in: [500, 1250] } }` for "one of these" |
18+
| `{ total: [] }` | kept no group | drop the condition, or write the value you meant |
19+
| `{ customer_id: { $in: 'c1' } }` / `{ customer_id: { $nin: 'c1' } }` | `$in` kept no group; `$nin` kept every group | `{ customer_id: 'c1' }` / `{ customer_id: { $ne: 'c1' } }`, or wrap the value in a list |
20+
| `{ customer_id: { $in: ['c1', null] } }` (or `$nin`) | the null member was compared as a value | `{ $or: [{ customer_id: { $in: ['c1'] } }, { customer_id: { $null: true } }] }` |
21+
| `{ total: { $gt: null } }` (or `$gte` / `$lt` / `$lte`) | `$gt` / `$gte` kept every group; `$lt` / `$lte` kept none | `{ total: { $eq: null } }` for "has no value", `{ total: { $ne: null } }` for "has a value" |
22+
| `{ total: { $between: 500 } }` or `{ total: { $between: [500] } }` | the scalar kept every group; the one-bound list kept the groups at or above it | `{ total: { $between: [min, max] } }` |
23+
| `{ total: { $between: [null, 1000] } }`, `['', 1000]` or `[undefined, 1000]` | the blank bound compared as a value | `{ total: { $lte: 1000 } }` for a one-sided range, or the bound you meant |
24+
| `{ total: { $between: [{ $field: 'order_count' }, 1000] } }` | the reference compared as a value | literal bounds. ⚠️ The refusal's own text suggests a two-bound `{ $field }` comparison, which `having` does not evaluate. In an operator slot the reference is compared as a value: under `$eq`, `$gt`, `$gte`, `$lt` or `$lte` it keeps no group, and under `$ne` it keeps every group. In the implicit slot (`{ total: { $field: 'order_count' } }`) it is refused as an unsupported operator (`INVALID_FILTER` / 400), though only when a grouped row carries that column: an empty grouped set evaluates nothing and comes back empty. That gap is not changed here |
25+
26+
The gate is ONE call in `engine.aggregate`, ahead of both `having` evaluations, so the two paths cannot disagree, and the verdict belongs to the filter rather than to the data: an empty grouped set refuses the same `having` a populated one does. Whatever arm the shared face gains later, `having` gains with it.
27+
28+
Who is affected: `having` is a request-only key (`QuerySchema.having`, `EngineAggregateOptions.having`), and no metadata type stores it. Every `having` in this repository's docs and published skills is a scalar comparison (`{ order_count: { $gt: 5 } }` and the like), and none authors a refused shape. Callers of `engine.aggregate` and of the REST aggregate query in a deployment were NOT measured.
29+
30+
Not changed: scalars, `null` in the equality slot (the has-no-value predicate), `$in` / `$nin` lists including the empty list, a two-bound `$between`, and scalar ordering bounds all answer exactly as before, on both paths. `$ne` with a list is not judged by the face yet, so `having` still answers it. Neither the comparand-TYPE door nor the unknown-field and declared-type gates that `where` also passes are run on `having`; this change adds the comparand-shape face only.
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
'@objectstack/driver-sql': patch
3+
'@objectstack/driver-sqlite-wasm': patch
4+
'@objectstack/driver-turso': patch
5+
---
6+
7+
fix(driver-sql, driver-turso): every filter-compile refusal stops naming a read scope's field or literal unless the refused predicate is marked as the caller's own (#20039)
8+
9+
Clause-②: no
10+
11+
A read scope is the RLS, sharing or tenant predicate that `plugin-security` (ordinary reads) and `service-analytics` (the ObjectQL analytics face) AND into the caller's `where`. Both merges mark the scope `'policy'` and the caller's own predicate `'author'` (`markFilterSubtreeProvenance`, `@objectstack/spec/data`). Nine more `SqlDriver` filter-compile refusals did not check the mark, so when one of them refused a scope, its `INVALID_FILTER` / 400 message named the scope's field, and for most of them its literal too. Measured through an ObjectQL `find` under a merge shaped like `plugin-security`'s, with the scope in the `'policy'` arm:
12+
13+
- an empty or non-string `$icontains` comparand;
14+
- a non-string `$like` / `$ilike` comparand;
15+
- a `$like` / `$ilike` pattern ending in a lone backslash;
16+
- an object or array comparand on `$contains`, `$notContains`, `$startsWith`, `$endsWith` or `$icontains`;
17+
- an `$in` / `$nin` / `$between` member that cannot be bound;
18+
- an `undefined` comparand, in any position;
19+
- an element of `$and` / `$or`, or the operand of `$not`, that is not a filter condition object;
20+
- a `$`-prefixed key in a node position that is not `$and`, `$or` or `$not`;
21+
- a `where` that reaches the driver as an array (the message printed the whole array).
22+
23+
Each of them now reads the mark on the node it was raised from, as the other compile refusals already did:
24+
25+
- **`'policy'`, unmarked or ambiguous:** same `INVALID_FILTER` / 400. The message says which kind of refusal fired, but names no field, operator variant, comparand, list position or filter path. Those go to the server log.
26+
- **`'author'`:** the full message, the same text the refusal answered before.
27+
28+
With these nine, every refusal on `SqlDriver`'s filter-compile path goes through the same seam.
29+
30+
`SqliteWasmDriver` (`@objectstack/driver-sqlite-wasm`) and `TursoDriver` in local mode extend `SqlDriver`, so they inherit this change from it: the same refusals answer the same way there.
31+
32+
**What an unmarked caller loses:** its own diagnostic from these refusals. That is every caller whose predicate reaches the driver unmarked, for example with no security plugin in the stack, in a system-context or anonymous call, or with a `where` that holds a `{placeholder}` token (the engine rewrites it before the merge). That caller gets the withheld wording with the same code and status, and the full text is in the server log. A member's plain `where` under `plugin-security` is marked `'author'` and keeps the full text.
33+
34+
The Turso REMOTE transport (`RemoteTransport`) compiles filters itself. Its copies of these refusals now read the mark the same way: the `$icontains`, `$like` / `$ilike`, lone-backslash, `undefined`, non-node element or operand, undeclared-key and non-object `where` refusals. So do its two other compile refusals that still named the field: an operator map with no operator in it, and a `$between` that reached the transport without being lowered. The operands go to its diagnostic sink. For six of these classes, the withheld sentence is the local one behind the `[RemoteTransport]` prefix. An object text comparand and an unbindable list member already answered there through its comparand refusal, which withholds. `TursoDriver`'s remote mode rebuilds every filter node before the transport sees it, so no mark reaches the transport there, and these refusals keep the withheld wording for every caller in that mode.
35+
36+
Not changed: which filters are refused, and the code and status of every refusal.
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
"@objectstack/service-analytics": minor
3+
---
4+
5+
fix(service-analytics)!: the analytics `where` door refuses a non-boolean `$null` / `$exists` flag, which it used to read as IS NOT NULL, the way every backend and the read-scope compiler already refuse it (#20040)
6+
7+
Clause-②: no (narrowing)
8+
9+
<!-- adr-0087: not-required (no-migration-prescription) no ADR-0087 ledger entry names the $null / $exists boolean domain: the refusals driver-sql makes (#5347, #5369) and the read-scope compiler makes (#6387) were never registered, and this change adds no new transition. It brings the analytics object-form `where`, its dataset and measure filters and the draft-data preview under the refusal every other face already makes. No ledger entry could carry it: a non-boolean flag has no boolean it can be mechanically rewritten to, because the backends disagreed on which one it meant and the string "false" is truthy. Which boolean the author meant is an authoring decision. The table below records the verdict each cell had and has; it prescribes no rewrite. -->
10+
11+
**BREAKING**: this narrows what the analytics faces of `@objectstack/service-analytics` accept. A `$null` or `$exists` flag whose value is not a boolean used to compile, in a caller `where`, a dataset `filter` or a measure `filter`, at any depth under `$and` / `$or` / `$not` or inside a nested relation. That covers a string such as `"false"`, a number, `null`, an array, a `Date` and a `{ $field }` reference. It is now refused with `INVALID_FILTER` / 400, in a message that names the operator, the field and the path, before any SQL statement runs or any `engine.aggregate` call is made. It ships as `minor` under the launch-window convention for accept-set narrowings.
12+
13+
| flag in an object-form `where` | before | now |
14+
|:--|:--|:--|
15+
| `$null` or `$exists` with a string, a number, `null`, an array, a `Date` or a `{ $field }` reference | read as IS NOT NULL on the native SQL path, the `/analytics/sql` echo and the ObjectQL engine path (the engine received `{ "$ne": null }`); `POST /api/v1/analytics/query` and `/analytics/dataset/query` answered 200 | refused 400: `Operator "$null" on field "stage" requires a boolean comparand (true or false). Received …` |
16+
| the same, under `$not` | the negation of IS NOT NULL: the rows with no value | refused 400 |
17+
| the same, in the draft-data preview | refused 400 as an operator the preview does not evaluate | refused 400, in the published door's words |
18+
| `true` or `false` | IS NULL or IS NOT NULL, per the contract | unchanged: the compiled tree, the SQL and the engine `where` are byte-identical |
19+
| `undefined`, or a plain object | refused 400 by the shared comparand-type face | unchanged, in that face's words |
20+
21+
`FieldOperatorsSchema` in `@objectstack/spec` declares both flags as booleans. The #5347 and #5369 rulings refuse a non-boolean one in every position and on every backend, because the backends read one in opposite directions: `driver-sql` compiled IS NULL for anything but `false`, and the JavaScript drivers compiled IS NOT NULL for anything but `true`. `driver-sql` refuses it, and so does this package's read-scope compiler. This door read every non-boolean as IS NOT NULL, so `"true"` and `"false"` asked for the same rows, and `{ "$null": "true" }` asked for the rows it excludes.
22+
23+
The repair is to write the boolean the filter means. `"$null": true` matches rows with no value and `"$null": false` rows with one; `$exists` is the exact inverse.
24+
25+
Over HTTP, both routes type the `where`, the `runtimeFilter` and the inline `dataset.filter` as `FilterConditionSchema`, whose field entries are open, so the body parse admitted the flag and the service served it. Both routes now answer 400 `INVALID_FILTER` from the service, with no statement run.
26+
27+
A `where` that carries a non-boolean flag and also a defect this compiler finds only while lowering (a field constraint mixing `$` and non-`$` keys, an operator outside the vocabulary, a zero-operator constraint) is now answered with the flag refusal. The code and status are the same 400 `INVALID_FILTER`. A shape or type defect elsewhere in the same `where` is still answered first.
28+
29+
Who is affected: nothing in this repository's examples, seeds, docs or package sources authors a non-boolean flag. A text scan over 4598 tracked non-test files found 22 matches: 21 are code comments, and one is an operator-name lookup table in `driver-memory`, not a filter. Stored datasets, dashboard widget filters and report runtime filters in a deployment were NOT measured.
30+
31+
Not changed: a `true` or `false` flag; the `FilterArray` spelling, whose `is_null` and `is_not_null` take their boolean from the operator name; the read-scope door, which already refused a non-boolean flag in its own withheld envelope.
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
---
2+
'@objectstack/runtime': patch
3+
---
4+
5+
fix(runtime): a self-hosted restart reads the kernel's own `sys_metadata` back into the registry, so objects authored at runtime keep serving
6+
7+
An object created and published at runtime on a self-hosted install (Studio,
8+
`PUT /api/v1/meta/object/…`, `POST /api/v1/packages/<id>/publish-drafts`)
9+
answered `404 OBJECT_NOT_FOUND` on the data API after the next restart, while
10+
its `sys_metadata` row was still there and
11+
`GET /api/v1/meta/object/<name>/published` still served it. Every `os dev` / `os serve` / `os start` boot was affected.
12+
13+
**Cause.** `createStandaloneStack` stamps `environmentId: 'env_local'` on every
14+
boot (or whatever `OS_ENVIRONMENT_ID` names), and `ObjectQLPlugin.start()` read
15+
any environment id as "a per-project kernel whose metadata comes from an
16+
artifact or a control-plane proxy", so it skipped reading `sys_metadata` at
17+
boot. It logged `Project kernel — skipping sys_metadata hydration (metadata
18+
sourced from artifact)`, which was false on this composition. This is the same
19+
deduction that `runPlatformMigrations` was declared out of.
20+
21+
**Fix: a declaration, not a deduction.** `createStandaloneStack` now declares
22+
`hydrateMetadataFromDb: true` to `ObjectQLPlugin`, where it used to be deduced
23+
from the environment-id stamp. The plugin option's own caution holds for every
24+
boot this function builds, for two reasons:
25+
26+
- the registry is per-instance: the function constructs a fresh
27+
`ObjectQLPlugin`, which builds its own `ObjectQL` and `SchemaRegistry`;
28+
- `sys_metadata` is on the kernel's own driver: it declares no datasource, so it
29+
routes to the one `default` datasource the function composes, and every
30+
database driver kind the function dispatches is a direct driver, never a
31+
control-plane proxy.
32+
33+
What an upgraded install sees at boot:
34+
35+
- every env-wide `sys_metadata` row (`organization_id` NULL), any metadata type,
36+
is registered again. Org-scoped rows are still served on demand and are not
37+
read at boot (ADR-0005);
38+
- a stored row that cannot register now says so at boot, on lines that were
39+
never reached here before: `[Protocol] [metadata_field_type_refused] …` at
40+
`error`, `[Protocol] Failed to hydrate <type>/<name>: …` and
41+
`[Protocol] [metadata_spec_invalid] …` at `warn`. The same boot also reports
42+
org-scoped rows of types that are not per-org overridable, on one aggregated
43+
line. Each line names its remedy;
44+
- the one-shot `os migrate *` / `os meta *` commands hydrate too, so a plan or
45+
a scan covers runtime-authored objects the way the serving boot registers
46+
them. The read itself writes nothing, and a deferred-DDL boot
47+
(`os migrate plan`, `os migrate duplicates`) still defers the tables of what
48+
it read.

0 commit comments

Comments
 (0)