Repository navigation
Commit 4b4ee88
Fixes #20745
Clause-②: no (narrowing)
## What this changes
A plain object with no `$`-operator key beneath a **relation** field
(the nested-relation form), a **structured-JSON** field (a whole-value
match), or the **platform-provisioned `id` column** that the declared
field map omits is now refused with `INVALID_FILTER` / 400, in the
engine's words, before any driver is asked. It holds on every driver, at
the three positions the engine judges: `where` (object form and
`FilterArray` sugar; `find` / `findOne` / `count` / `aggregate` /
`update` / `delete` and the judge-only `judgeFilter`),
`aggregations[i].filter`, and `having`.
**Landing site: the door PR #20744 opened for scalars, extended. There
is no second door and no second traversal.** The no-operator-object arm
of the number-comparand door's walk (`walkCondition`) already asked one
question per field key. It now classifies the column into one of three
kinds, each with its own words:
- `packages/objectql/src/no-operator-object-door.ts`:
`noOperatorObjectColumnKind` gives `scalar` (unchanged:
`SCALAR_FILTER_HEAD_TYPES` plus `MULTI_OPTION_TYPES`), `relation`
(spec's `REFERENCE_VALUE_TYPES`: `lookup`, `master_detail`, `user`,
`tree`), `json` (spec's `STRUCTURED_JSON_TYPES`), or `null` (never
judged). `provisionedNoOperatorObjectColumn` covers `id` / `created_at`
/ `updated_at` when the declared map omits them. The three word builders
live here too. ⛔ Nothing in it walks a filter.
- `number-comparand-declared-type-door.ts`: the per-key facts carry the
judged column instead of a scalar type. The walk, its positions and its
boundaries are unchanged.
- `engine.ts` / `having-filter.ts`: comments only.
- `packages/spec/src/data/filter.zod.ts`: prose only.
`FilterCondition`'s form 4 now states that the engine refuses it and
names the route. The `QueryFilter` example stops teaching it. The two
"Nested relation" type comments point at the refusal. ⛔ The type and the
schema are not narrowed.
- `content/docs/kernel/contracts/data-engine.mdx`: the `// Nested
relation filter` example is replaced by the route that works (query the
related object, then `$in` on its ids), with one paragraph on the
refusal and on multi-valued lookups.
**The words put the verdict and the route first.** The REST door
truncates a 4xx message at 500 characters (`CLIENT_MESSAGE_MAX` in
`packages/rest/src/error-response.ts`). The first draft of these words,
and the #20546 scalar words, put the route past that bound, so a REST
caller never saw it. Every kind now reads: position, then verdict, then
`The filter was NOT applied.`, then the route, then the reasoning. The
REST pins assert the route in the response body. The #20546 scalar words
were rewritten in the same shape because this change made their middle
sentence false: it said a nested-relation condition is something "only a
relation field … can carry". Examples of the words, as `engine.find`
throws them:
```text
find('rp_ledger'): filter on 'owner' puts an object with no operator key (keys "region") at where.owner, beneath the declared lookup field 'owner' — the nested-relation form, which the engine does not serve. The filter was NOT applied. Filter the related object 'rp_owner' first, then match 'owner' against the ids it returns: { "owner": { "$in": [ID, …] } }. An object with no "$" operator is filter structure, not a value: 'owner' stores the related record's id, no driver follows it into the related object, and an empty answer would read exactly like a real one.
find('rp_ledger'): filter on 'owners' puts an object with no operator key (keys "region") at where.owners, beneath the declared lookup field 'owners' — the nested-relation form, which the engine does not serve. The filter was NOT applied. Filter the related object 'rp_owner' first, then match 'owners' against the ids it returns: { "owners": { "$contains": ID } } for one id, an $or of those for several. …
find('rp_ledger'): filter on 'meta' puts an object with no operator key (keys "a") at where.meta, as the value of the declared json field 'meta' — a whole-value match, which the engine does not serve. The filter was NOT applied. Test the whole value's presence with { "meta": { "$null": false } }, or store the part you filter on in a field of its own and filter that field. …
find('rp_ledger'): filter on 'id' puts an object with no operator key (keys "a") at where.id, where a value of the platform-provisioned text column 'id' belongs. An object with no "$" operator is filter structure, not a value. The filter was NOT applied. Compare 'id' with a value ({ "id": VALUE }) or an operator ({ "id": { "$eq": VALUE } }). …
```
## Before, measured on `origin/main` `a51920f5fb`
The readings come through `POST /api/v1/data/:object/query` (the real
`RestServer` route over `ObjectStackProtocolImplementation` and
`ObjectQL`) on InMemoryDriver, SqlDriver on SQLite (better-sqlite3), and
SqlDriver on a live PostgreSQL 16.13 started for this run. Three rows:
owner `u1` (region NA) on `d1` and `d3`; `meta` `{a:1}` / `{a:2}` /
`{b:1}`.
| filter | InMemoryDriver | SQLite | PostgreSQL 16 |
|:--|:--|:--|:--|
| `where` `{ owner: { region: 'NA' } }` (lookup, the card) | 200, **no
rows** (`d1`, `d3` meant) | 400 `INVALID_FILTER`, the driver's words
("cannot be bound as a SQL parameter") | same as SQLite |
| the same under `master_detail`, a `multiple: true` lookup, `user`,
`tree` | 200, no rows | 400, the driver's words | 400, the driver's
words |
| `where` `{ meta: { a: 1 } }` (json, the card) | 200, `d1` (deep
equality) | 400, the driver's words | 400, the driver's words |
| `where` `{ ship_to: { city: 'Paris' } }` (address), `{ spec: { k: 1 }
}` (composite) | 200, `d1`, `d3` (deep equality) | 400 | 400 |
| `where` `{ id: { a: 1 } }` (the card) | 200, no rows | 400 | 400 |
| `where` `{ owner: {} }`, `{ meta: {} }` | 400, the driver's
zero-operator words | 400, the driver's words | same |
| `aggregations[1].filter` `{ owner: { region: 'NA' } }` | count 0 |
count 0 | count 0 |
| `aggregations[1].filter` `{ meta: { a: 1 } }` | count 1 | count 1 |
count 1 |
| `having` `{ owner: { region: 'NA' } }` over `groupBy: ['owner']` | no
group | no group | no group |
| `having` `{ meta: { a: 1 } }` over `groupBy: ['meta']` | one (wrong)
group | no group | **500 `DATABASE_ERROR`** (from the json groupBy
itself, see the notes) |
| route `{ owner: { $in: ['u1'] } }`; same under `master_detail`, `user`
| `d1`, `d3` | `d1`, `d3` | `d1`, `d3` |
| route `{ parent: { $in: ['d1'] } }` (tree) | `d2`, `d3` | `d2`, `d3` |
`d2`, `d3` |
| `{ owners: { $in: ['u1'] } }` (multiple lookup) | `d1`, `d3` |
**400**, the driver's JSON-column words | 400 |
| route `{ owners: { $contains: 'u1' } }`, and `$or` of `$contains` |
`d1`, `d3` | `d1`, `d3` | `d1`, `d3` |
| dotted `{ 'owner.region': 'NA' }` | 400 `INVALID_FIELD` (the dotted
verdict) | same | same |
| `{ meta: { $contains: 'a' } }` | 400 `INVALID_FILTER` (the
text-operator door: a JSON value is never a string) | same | same |
| `{ meta: { $null: false } }`, `{ meta: { $exists: true } }` | all 3
rows | all 3 rows | all 3 rows |
| control `{ photo: { url: 'x' } }` (image) | 200, no rows | 400, the
driver's words | same |
## After, the same run on this branch
Every refused row above answers `400 INVALID_FILTER` in the engine's
words, on all three drivers, at the path the object sits at
(`where.owner`, `where.$not.owner`, `aggregations[1].filter.owner`,
`having.owner`, …). No read of the object runs. The routes (`$in`,
`$contains`, `$null`), the dotted verdict, the `$contains`-on-json
refusal and the image control answer exactly as before.
## Hypotheses (zone 2): which held
- **H1: held, with two refinements.** The door classifies by declared
type, and relation and JSON became judged kinds of the same
classification, with their own words. (a) The classes are the spec's
closed sets: `REFERENCE_VALUE_TYPES` brings in `user` and `tree` beside
`lookup` / `master_detail`, and `STRUCTURED_JSON_TYPES` brings in
`address`, `composite`, `repeater`, `record`, `location` and `vector`
beside `json`. See the scope note below. (b) **Left unjudged:** file and
media types (the #8371 carve-out: a legacy stored value is an inline
object that memory can still match) and `formula` (refused one door
earlier, `INVALID_FIELD`). Triage's text covers neither.
- **H2: held, in the "not served" direction.** A dotted relation path
`'owner.region'` is served by no driver: it answers `400 INVALID_FIELD`
(the #8371 dotted verdict) on all three. So the refusal names only the
related-object query plus ids. **One refinement:** `$in` on ids does not
work for a **multi-valued** lookup on SQL, where the driver refuses
`$in` on its JSON column. So the words name `$contains` per id there
(measured `d1`, `d3` on all three).
- **H3: held.** All three positions take the new kinds through the same
walk. At `having`, relation and JSON columns **do** appear: a groupBy of
the field, or a `min` / `max` of it, carries that field's type
(`aggregatedRowColumnTypes`). A nested-relation `having` kept no group
on every driver before. `having` has no field declaration to read, so
its relation words name "the related object" and the `$in` spelling.
- **H4: held; the nested arm is not separable in `FilterCondition`
without narrowing another form.** Its index signature is `any |
FieldOperators | FilterCondition`, which TypeScript collapses to `any`.
So removing the `FilterCondition` member is a no-op, not a separation.
At the schema, the nested-relation form and a JSON object comparand are
the same shape (a plain object with no `$` key beneath a key), and only
the column's declared type tells them apart. The generic `Filter` type's
nested arm (a recursive `Filter` over an object-typed property's own
type) **is** a separate union member. Removing it would narrow `Filter`
for every object-typed property, so it is a `Clause-②: yes (narrowing)`
change for its own card. ⛔ Neither is separated here.
## Where this departs from the order or the ruling (named, not silently
chosen)
- **`{ id: { a: 1 } }`.** Triage says the `id` row "answers the door's
existing unknown-field verdict on every driver". The **Pins** ruling
says every row of the card's table answers the same `INVALID_FILTER`. On
`origin/main` the door has no unknown-field verdict that refuses. Its
verdict for an undeclared key is tolerance (the engine's registry-less
rule, pinned by `GUARD an UNKNOWN field …`), which leaves `id` to the
drivers: memory 200 with no rows, SQL 400. Both readings cannot hold at
once. `id` is not an unknown field by the engine's own definitions:
`find` / `findOne` add it to their known set, the write gate admits it
(`PLATFORM_PROVISIONED_COLUMNS`), and so do the REST ingress
(`resolveQueryFields`) and the per-aggregation reference names. So this
PR judges the three platform-provisioned columns by the type they store,
and only when the declared map omits them. That keeps `id`, `created_at`
and `updated_at` in the scalar words, keeps every other undeclared key
tolerated, and makes the Pins row true. Reported to the PM as an open
question.
- **Scope of the JSON kind.** Triage names `json`. `address` and
`composite` were measured with the identical split (memory deep-equal
rows, SQL 400). The spec publishes one class for them, so the arm judges
the class, not one member of it. This is the bounded in-place fix: it is
the same defect class, the same mechanical classification as the card,
and the same file under this claim, and it adds no new gate family.
- **Triage's `$contains` example for a JSON field.** It is not a route:
the text-operator door refuses `$contains` over a JSON value on every
driver (measured above). The JSON words name `$null` and a stored field
instead.
- **The per-aggregation `filter` with a JSON object.** It was the one
cell that already answered alike on every driver (count 1, the engine's
own deep equality). It is refused now, so that one filter has one answer
at every position. The changeset names it.
- **InMemoryDriver cells.** The memory cell of each refusal is
`engine-nested-object-door.test.ts`'s recording driver by construction:
the arm answers before a driver is resolved. The memory readings of the
**routes** were measured (the table above) but are not pinned in a new
suite: `check:driver-memory-census` refuses a new test consumer of the
in-memory driver without a maintainer ruling. It caught a first draft
that put one in `packages/runtime`, which was dropped.
## Tests
The final HEAD is `ee17b18fde`, a merge of `origin/main` `eead9dcf40`
into the branch.
- `pnpm --filter @objectstack/objectql test` on `ee17b18fde`: **339
files / 6719 passed**.
- `pnpm --filter @objectstack/rest test` on `ee17b18fde`: **231 files /
4469 passed / 71 skipped**.
- `pnpm --filter @objectstack/spec test` on `ee17b18fde`: **577 files /
17007 passed / 1 todo**.
- `test:repo` on `e08fd6883f`: spec 45 files / 794, objectql 1 / 5, rest
1 / 8.
- `typecheck` for objectql, rest and spec on `ee17b18fde`: exit 0,
including each `check:test-typecheck` with its ledger held.
- `pnpm --filter @objectstack/spec check:generated`: all 15 artifacts up
to date.
- New pin `packages/objectql/src/engine-nested-object-door.test.ts` (15
tests). It covers every relation type (single and multiple, with the
route per multiplicity and the related object's name), every
structured-JSON type, the provisioned `id`, `{}`, every verb and the
judge, `$and` / `$or` / `$not` and sugar, the per-aggregation filter,
`having` (a lookup groupBy, a `max` of a master-detail, a json groupBy),
the three REST doors into `findData`, the controls (the routes, a file
field, an unknown key), and the classification GUARDs over every
`FieldType`.
- New pin `packages/rest/src/data-nested-object-door.test.ts`. SQLite
always runs; PostgreSQL and MySQL run where `OS_TEST_POSTGRES_URL` /
`OS_TEST_MYSQL_URL` are set. It covers every row of the card's table,
and the **route asserted inside the REST body** (so it must land inside
the 500-character bound). It also covers the per-aggregation filter and
`having`, the routes answering the rows the nested form meant (`$in` on
the related object's ids gives `d1`, `d3`; `$contains` on a multiple
lookup gives `d1`, `d3`; tree gives `d2`, `d3`), and the file control.
Local run with a live PostgreSQL 16.13: **8 passed (sqlite 4, live
postgres 4) / 4 skipped (mysql, no URL)**. ⚠️ As with the sibling door
suites, no CI job sets these URLs for `@objectstack/rest`, so the live
cells run only locally.
- Fixture triage for the removed "accepted" branch. The #20546 pins
(`engine-no-operator-object-door.test.ts`,
`data-no-operator-object-door.test.ts`) keep a file field as their only
control. Two name-gate controls pinned "the nested-relation form still
passes the doors": `protocol-explicit-filter-field-gate.test.ts` (#7534)
and `query-expression-conformance.test.ts` (#8371). They now pin what
they were for: the answer is the engine's `INVALID_FILTER` in the
nested-relation words, never the name gate's `INVALID_FIELD`.
- Downstream consumers (`...@objectstack/objectql` direction), on
`e08fd6883f`:
- `@objectstack/metadata-protocol`: 190 files passed, 3 skipped / 2792
passed, 19 skipped.
- `@objectstack/service-analytics`: 140 / 3266 passed. Its
nested-relation `where` is flattened to cube members before any engine
call, and it passed unchanged.
- `@objectstack/plugin-security`: 147 files / 3202 passed, 23 skipped.
- The other downstream consumers are declared to CI.
**Reverse verification (ablation), from the committed fix.** It ran
through `scripts/ablation-replace.mjs` in WRAP mode, trap-restored. The
walk's gate `if (facts.column !== null && isNoOperatorObject(value)) {`
was narrowed back to the #20546 behaviour with `facts.column.kind ===
'scalar' && facts.column.provisioned !== true && facts.column.type !==
'__ablated_20745__'`. On disk the anchor went 1 → 0 and the marker 0 →
1, with blob `ea19d1959255` → `3134e87031b1`. Then objectql was rebuilt,
and `ablation-dist-preflight` found the marker in 4 built files.
- Predicted direction: red. Observed: red.
- objectql pins: **13 failed / 236 passed**. Every new refusal case
failed, and so did the two rewritten name-gate controls. Every control,
GUARD and #20546 scalar pin stayed green.
- rest pins: **4 failed / 12 passed / 8 skipped**. The `where` and
aggregate refusals failed on SQLite and live PostgreSQL. The routes, the
controls and the #20546 file stayed green.
- Restore leg: blob equals HEAD (`ea19d1959255`), `git diff HEAD` is
empty, and whole-tree `git status --porcelain` is empty. After a
rebuild, the `--absent` preflight found the marker absent from all 14
built files. Then both pin sets were green again (249 passed; 16 passed
+ 8 skipped).
## Gates
`node scripts/pm/dispatch-gates.mjs --commands` at `ee17b18fde` (after
`git fetch`, so not stale) derived 110 commands. All 110 were run on
`ee17b18fde`. `--ran` reconciles them: **110 derived, 110 run, 0
NOT-MEASURED, 0 UNRUN**, and all exit 0. Among them are these gates:
- `check:adr-0087-registration --base origin/main`: `not-required
(no-migration-prescription)` accepted.
- `check:changeset-no-major`, `check:empty-changeset`,
`check:doc-authoring`, `check:nul-bytes`.
- `check:engine-double-contract` (904 pinned), `check:where-matcher`
(440 matchers, 0 silently wrong), `check:driver-memory-census`,
`check:cross-package-test-inputs`, `check:test-source-alias`,
`check:type-check-coverage`, `check:type-check-debt`,
`check:query-options-erasure`.
- `check:dual-build-cjs-loads` (105 entry points, 66 packages).
- spec's `check:api-surface` / `check:docs` / `check:authorable-surface`
/ `check:skill-examples`.
Lint, narrowed and proven: `pnpm exec eslint --no-inline-config --format
json` over the 11 changed `.ts` files at `ee17b18fde` found **11 files,
0 errors, 0 warnings**. Three facts make this narrowing a measurement:
- The checked population comes from eslint's own config: `isPathIgnored`
answers `false` for all 11.
- The file count comes from the JSON output: 11 results.
- Untouched files cannot change verdict: `parserOptions.project` and
`projectService` are `null` for every file, so type-aware linting is not
enabled.
## Changesets
- `.changeset/20745-nested-object-door.md`: `@objectstack/objectql`
`minor`, a BREAKING banner, `Clause-②: no (narrowing)`, and the ADR-0087
marker `not-required (no-migration-prescription)`, as the #20546
changeset has. It states what an author sees now and the route that
works, per kind, with the table. It says it supersedes the "Unchanged"
paragraph of the pending scalar-field entry
(`20546-no-operator-object-on-scalar`) for relation and structured-JSON
fields.
- `.changeset/20745-nested-relation-prose.md`: `@objectstack/spec`
`patch` for the shipped JSDoc.
- No export or published type changes: the door module is internal, and
`@objectstack/objectql`'s root and `./core` exports are unchanged.
`check:api-surface` is green.
## Acceptance notes
- **Out of scope, reported to the PM, not filed:**
- The **published skill** `skills/objectstack-query` teaches the
nested-relation form as working: `SKILL.md` "Nested Relation Filters",
the "Filter parent by child conditions" row and the `search` paragraph,
and `rules/filters.md` "Nested Relation Filters". It was already untrue
before this change (memory answered no rows, SQL 400). It is a governed
Tier H surface outside this claim's file surface, so it is not touched
here.
- A **`groupBy` of a `json` field** answers 500 `DATABASE_ERROR` on
PostgreSQL. On InMemoryDriver it merges every row into one group (`n:
3`), and on SQLite it gives one group per serialized value.
- File and media fields keep the #8371 carve-out and stay unjudged. `{
photo: { url: 'x' } }` still answers memory 200 with no rows and SQL
400.
- At `having` there is no field declaration, so a relation column's
words say "the related object" and give the `$in` spelling. A `having`
over a multi-valued relation groupBy would get that single-valued
spelling.
- `referenceTargetOf` names the related object in the relation words.
For a field whose `reference` carrier never went through the schema's
parse (a non-string), it throws its own `TypeError` instead of the
refusal. Parse refuses that shape at the contract door.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
1 parent 085ca6b commit 4b4ee88
14 files changed
Lines changed: 1008 additions & 94 deletions
File tree
- .changeset
- content/docs/kernel/contracts
- packages
- objectql/src
- rest/src
- spec/src/data
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
178 | 178 | | |
179 | 179 | | |
180 | 180 | | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
185 | 185 | | |
186 | 186 | | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
187 | 194 | | |
188 | 195 | | |
189 | 196 | | |
| |||
0 commit comments