Skip to content

fix(objectql)!: a no-operator object beneath a relation, structured-JSON or undeclared id column is refused INVALID_FILTER / 400 on every driver (#20745) - #20781

Merged
objectstack-fleet[bot] merged 8 commits into
mainfrom
claude/issue-20745-nested-object-door
Sep 30, 2026
Merged

objectstack-fleet[bot] merged 8 commits into
mainfrom
claude/issue-20745-nested-object-door

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #20745
Clause-②: no (narrowing)

What this changes

A plain object with no $-operator key beneath a relation field (the nested-relation form), a structured-JSON field (a whole-value match), or the platform-provisioned id column that the declared field map omits is now refused with INVALID_FILTER / 400, in the engine's words, before any driver is asked. It holds on every driver, at the three positions the engine judges: where (object form and FilterArray sugar; find / findOne / count / aggregate / update / delete and the judge-only judgeFilter), aggregations[i].filter, and having.

Landing site: the door PR #20744 opened for scalars, extended. There is no second door and no second traversal. The no-operator-object arm of the number-comparand door's walk (walkCondition) already asked one question per field key. It now classifies the column into one of three kinds, each with its own words:

  • packages/objectql/src/no-operator-object-door.ts: noOperatorObjectColumnKind gives scalar (unchanged: SCALAR_FILTER_HEAD_TYPES plus MULTI_OPTION_TYPES), relation (spec's REFERENCE_VALUE_TYPES: lookup, master_detail, user, tree), json (spec's STRUCTURED_JSON_TYPES), or null (never judged). provisionedNoOperatorObjectColumn covers id / created_at / updated_at when the declared map omits them. The three word builders live here too. ⛔ Nothing in it walks a filter.
  • number-comparand-declared-type-door.ts: the per-key facts carry the judged column instead of a scalar type. The walk, its positions and its boundaries are unchanged.
  • engine.ts / having-filter.ts: comments only.
  • packages/spec/src/data/filter.zod.ts: prose only. FilterCondition's form 4 now states that the engine refuses it and names the route. The QueryFilter example stops teaching it. The two "Nested relation" type comments point at the refusal. ⛔ The type and the schema are not narrowed.
  • content/docs/kernel/contracts/data-engine.mdx: the // Nested relation filter example is replaced by the route that works (query the related object, then $in on its ids), with one paragraph on the refusal and on multi-valued lookups.

The words put the verdict and the route first. The REST door truncates a 4xx message at 500 characters (CLIENT_MESSAGE_MAX in packages/rest/src/error-response.ts). The first draft of these words, and the #20546 scalar words, put the route past that bound, so a REST caller never saw it. Every kind now reads: position, then verdict, then The filter was NOT applied., then the route, then the reasoning. The REST pins assert the route in the response body. The #20546 scalar words were rewritten in the same shape because this change made their middle sentence false: it said a nested-relation condition is something "only a relation field … can carry". Examples of the words, as engine.find throws them:

find('rp_ledger'): filter on 'owner' puts an object with no operator key (keys "region") at where.owner, beneath the declared lookup field 'owner' — the nested-relation form, which the engine does not serve. The filter was NOT applied. Filter the related object 'rp_owner' first, then match 'owner' against the ids it returns: { "owner": { "$in": [ID, …] } }. An object with no "$" operator is filter structure, not a value: 'owner' stores the related record's id, no driver follows it into the related object, and an empty answer would read exactly like a real one.

find('rp_ledger'): filter on 'owners' puts an object with no operator key (keys "region") at where.owners, beneath the declared lookup field 'owners' — the nested-relation form, which the engine does not serve. The filter was NOT applied. Filter the related object 'rp_owner' first, then match 'owners' against the ids it returns: { "owners": { "$contains": ID } } for one id, an $or of those for several. …

find('rp_ledger'): filter on 'meta' puts an object with no operator key (keys "a") at where.meta, as the value of the declared json field 'meta' — a whole-value match, which the engine does not serve. The filter was NOT applied. Test the whole value's presence with { "meta": { "$null": false } }, or store the part you filter on in a field of its own and filter that field. …

find('rp_ledger'): filter on 'id' puts an object with no operator key (keys "a") at where.id, where a value of the platform-provisioned text column 'id' belongs. An object with no "$" operator is filter structure, not a value. The filter was NOT applied. Compare 'id' with a value ({ "id": VALUE }) or an operator ({ "id": { "$eq": VALUE } }). …

Before, measured on origin/main a51920f5fb

The readings come through POST /api/v1/data/:object/query (the real RestServer route over ObjectStackProtocolImplementation and ObjectQL) on InMemoryDriver, SqlDriver on SQLite (better-sqlite3), and SqlDriver on a live PostgreSQL 16.13 started for this run. Three rows: owner u1 (region NA) on d1 and d3; meta {a:1} / {a:2} / {b:1}.

filter InMemoryDriver SQLite PostgreSQL 16
where { owner: { region: 'NA' } } (lookup, the card) 200, no rows (d1, d3 meant) 400 INVALID_FILTER, the driver's words ("cannot be bound as a SQL parameter") same as SQLite
the same under master_detail, a multiple: true lookup, user, tree 200, no rows 400, the driver's words 400, the driver's words
where { meta: { a: 1 } } (json, the card) 200, d1 (deep equality) 400, the driver's words 400, the driver's words
where { ship_to: { city: 'Paris' } } (address), { spec: { k: 1 } } (composite) 200, d1, d3 (deep equality) 400 400
where { id: { a: 1 } } (the card) 200, no rows 400 400
where { owner: {} }, { meta: {} } 400, the driver's zero-operator words 400, the driver's words same
aggregations[1].filter { owner: { region: 'NA' } } count 0 count 0 count 0
aggregations[1].filter { meta: { a: 1 } } count 1 count 1 count 1
having { owner: { region: 'NA' } } over groupBy: ['owner'] no group no group no group
having { meta: { a: 1 } } over groupBy: ['meta'] one (wrong) group no group 500 DATABASE_ERROR (from the json groupBy itself, see the notes)
route { owner: { $in: ['u1'] } }; same under master_detail, user d1, d3 d1, d3 d1, d3
route { parent: { $in: ['d1'] } } (tree) d2, d3 d2, d3 d2, d3
{ owners: { $in: ['u1'] } } (multiple lookup) d1, d3 400, the driver's JSON-column words 400
route { owners: { $contains: 'u1' } }, and $or of $contains d1, d3 d1, d3 d1, d3
dotted { 'owner.region': 'NA' } 400 INVALID_FIELD (the dotted verdict) same same
{ meta: { $contains: 'a' } } 400 INVALID_FILTER (the text-operator door: a JSON value is never a string) same same
{ meta: { $null: false } }, { meta: { $exists: true } } all 3 rows all 3 rows all 3 rows
control { photo: { url: 'x' } } (image) 200, no rows 400, the driver's words same

After, the same run on this branch

Every refused row above answers 400 INVALID_FILTER in the engine's words, on all three drivers, at the path the object sits at (where.owner, where.$not.owner, aggregations[1].filter.owner, having.owner, …). No read of the object runs. The routes ($in, $contains, $null), the dotted verdict, the $contains-on-json refusal and the image control answer exactly as before.

Hypotheses (zone 2): which held

  • H1: held, with two refinements. The door classifies by declared type, and relation and JSON became judged kinds of the same classification, with their own words. (a) The classes are the spec's closed sets: REFERENCE_VALUE_TYPES brings in user and tree beside lookup / master_detail, and STRUCTURED_JSON_TYPES brings in address, composite, repeater, record, location and vector beside json. See the scope note below. (b) Left unjudged: file and media types (the [finding] The FILTER axis has no DOTTED-path verdict — where: { project_id.name: 'x' } rides its head segment past both doors, where SORT refuses the same spelling (#4256) #8371 carve-out: a legacy stored value is an inline object that memory can still match) and formula (refused one door earlier, INVALID_FIELD). Triage's text covers neither.
  • H2: held, in the "not served" direction. A dotted relation path 'owner.region' is served by no driver: it answers 400 INVALID_FIELD (the [finding] The FILTER axis has no DOTTED-path verdict — where: { project_id.name: 'x' } rides its head segment past both doors, where SORT refuses the same spelling (#4256) #8371 dotted verdict) on all three. So the refusal names only the related-object query plus ids. One refinement: $in on ids does not work for a multi-valued lookup on SQL, where the driver refuses $in on its JSON column. So the words name $contains per id there (measured d1, d3 on all three).
  • H3: held. All three positions take the new kinds through the same walk. At having, relation and JSON columns do appear: a groupBy of the field, or a min / max of it, carries that field's type (aggregatedRowColumnTypes). A nested-relation having kept no group on every driver before. having has no field declaration to read, so its relation words name "the related object" and the $in spelling.
  • H4: held; the nested arm is not separable in FilterCondition without narrowing another form. Its index signature is any | FieldOperators | FilterCondition, which TypeScript collapses to any. So removing the FilterCondition member is a no-op, not a separation. At the schema, the nested-relation form and a JSON object comparand are the same shape (a plain object with no $ key beneath a key), and only the column's declared type tells them apart. The generic Filter type's nested arm (a recursive Filter over an object-typed property's own type) is a separate union member. Removing it would narrow Filter for every object-typed property, so it is a Clause-②: yes (narrowing) change for its own card. ⛔ Neither is separated here.

Where this departs from the order or the ruling (named, not silently chosen)

  • { id: { a: 1 } }. Triage says the id row "answers the door's existing unknown-field verdict on every driver". The Pins ruling says every row of the card's table answers the same INVALID_FILTER. On origin/main the door has no unknown-field verdict that refuses. Its verdict for an undeclared key is tolerance (the engine's registry-less rule, pinned by GUARD an UNKNOWN field …), which leaves id to the drivers: memory 200 with no rows, SQL 400. Both readings cannot hold at once. id is not an unknown field by the engine's own definitions: find / findOne add it to their known set, the write gate admits it (PLATFORM_PROVISIONED_COLUMNS), and so do the REST ingress (resolveQueryFields) and the per-aggregation reference names. So this PR judges the three platform-provisioned columns by the type they store, and only when the declared map omits them. That keeps id, created_at and updated_at in the scalar words, keeps every other undeclared key tolerated, and makes the Pins row true. Reported to the PM as an open question.
  • Scope of the JSON kind. Triage names json. address and composite were measured with the identical split (memory deep-equal rows, SQL 400). The spec publishes one class for them, so the arm judges the class, not one member of it. This is the bounded in-place fix: it is the same defect class, the same mechanical classification as the card, and the same file under this claim, and it adds no new gate family.
  • Triage's $contains example for a JSON field. It is not a route: the text-operator door refuses $contains over a JSON value on every driver (measured above). The JSON words name $null and a stored field instead.
  • The per-aggregation filter with a JSON object. It was the one cell that already answered alike on every driver (count 1, the engine's own deep equality). It is refused now, so that one filter has one answer at every position. The changeset names it.
  • InMemoryDriver cells. The memory cell of each refusal is engine-nested-object-door.test.ts's recording driver by construction: the arm answers before a driver is resolved. The memory readings of the routes were measured (the table above) but are not pinned in a new suite: check:driver-memory-census refuses a new test consumer of the in-memory driver without a maintainer ruling. It caught a first draft that put one in packages/runtime, which was dropped.

Tests

The final HEAD is ee17b18fde, a merge of origin/main eead9dcf40 into the branch.

  • pnpm --filter @objectstack/objectql test on ee17b18fde: 339 files / 6719 passed.
  • pnpm --filter @objectstack/rest test on ee17b18fde: 231 files / 4469 passed / 71 skipped.
  • pnpm --filter @objectstack/spec test on ee17b18fde: 577 files / 17007 passed / 1 todo.
  • test:repo on e08fd6883f: spec 45 files / 794, objectql 1 / 5, rest 1 / 8.
  • typecheck for objectql, rest and spec on ee17b18fde: exit 0, including each check:test-typecheck with its ledger held.
  • pnpm --filter @objectstack/spec check:generated: all 15 artifacts up to date.
  • New pin packages/objectql/src/engine-nested-object-door.test.ts (15 tests). It covers every relation type (single and multiple, with the route per multiplicity and the related object's name), every structured-JSON type, the provisioned id, {}, every verb and the judge, $and / $or / $not and sugar, the per-aggregation filter, having (a lookup groupBy, a max of a master-detail, a json groupBy), the three REST doors into findData, the controls (the routes, a file field, an unknown key), and the classification GUARDs over every FieldType.
  • New pin packages/rest/src/data-nested-object-door.test.ts. SQLite always runs; PostgreSQL and MySQL run where OS_TEST_POSTGRES_URL / OS_TEST_MYSQL_URL are set. It covers every row of the card's table, and the route asserted inside the REST body (so it must land inside the 500-character bound). It also covers the per-aggregation filter and having, the routes answering the rows the nested form meant ($in on the related object's ids gives d1, d3; $contains on a multiple lookup gives d1, d3; tree gives d2, d3), and the file control. Local run with a live PostgreSQL 16.13: 8 passed (sqlite 4, live postgres 4) / 4 skipped (mysql, no URL). ⚠️ As with the sibling door suites, no CI job sets these URLs for @objectstack/rest, so the live cells run only locally.
  • Fixture triage for the removed "accepted" branch. The [finding] a plain object with no $ key as a scalar field's filter value answers 200 with no rows on the memory driver and INVALID_FILTER 400 on SQLite and PostgreSQL #20546 pins (engine-no-operator-object-door.test.ts, data-no-operator-object-door.test.ts) keep a file field as their only control. Two name-gate controls pinned "the nested-relation form still passes the doors": protocol-explicit-filter-field-gate.test.ts (Data query: an unknown field inside where / $filter answers 200/0 instead of 400 INVALID_FIELD — the bare-key door disagrees (#4134's uncovered sibling) #7534) and query-expression-conformance.test.ts ([finding] The FILTER axis has no DOTTED-path verdict — where: { project_id.name: 'x' } rides its head segment past both doors, where SORT refuses the same spelling (#4256) #8371). They now pin what they were for: the answer is the engine's INVALID_FILTER in the nested-relation words, never the name gate's INVALID_FIELD.
  • Downstream consumers (...@objectstack/objectql direction), on e08fd6883f:
    • @objectstack/metadata-protocol: 190 files passed, 3 skipped / 2792 passed, 19 skipped.
    • @objectstack/service-analytics: 140 / 3266 passed. Its nested-relation where is flattened to cube members before any engine call, and it passed unchanged.
    • @objectstack/plugin-security: 147 files / 3202 passed, 23 skipped.
    • The other downstream consumers are declared to CI.

Reverse verification (ablation), from the committed fix. It ran through scripts/ablation-replace.mjs in WRAP mode, trap-restored. The walk's gate if (facts.column !== null && isNoOperatorObject(value)) { was narrowed back to the #20546 behaviour with facts.column.kind === 'scalar' && facts.column.provisioned !== true && facts.column.type !== '__ablated_20745__'. On disk the anchor went 1 → 0 and the marker 0 → 1, with blob ea19d1959255 → 3134e87031b1. Then objectql was rebuilt, and ablation-dist-preflight found the marker in 4 built files.

Gates

node scripts/pm/dispatch-gates.mjs --commands at ee17b18fde (after git fetch, so not stale) derived 110 commands. All 110 were run on ee17b18fde. --ran reconciles them: 110 derived, 110 run, 0 NOT-MEASURED, 0 UNRUN, and all exit 0. Among them are these gates:

  • check:adr-0087-registration --base origin/main: not-required (no-migration-prescription) accepted.
  • check:changeset-no-major, check:empty-changeset, check:doc-authoring, check:nul-bytes.
  • check:engine-double-contract (904 pinned), check:where-matcher (440 matchers, 0 silently wrong), check:driver-memory-census, check:cross-package-test-inputs, check:test-source-alias, check:type-check-coverage, check:type-check-debt, check:query-options-erasure.
  • check:dual-build-cjs-loads (105 entry points, 66 packages).
  • spec's check:api-surface / check:docs / check:authorable-surface / check:skill-examples.

Lint, narrowed and proven: pnpm exec eslint --no-inline-config --format json over the 11 changed .ts files at ee17b18fde found 11 files, 0 errors, 0 warnings. Three facts make this narrowing a measurement:

  • The checked population comes from eslint's own config: isPathIgnored answers false for all 11.
  • The file count comes from the JSON output: 11 results.
  • Untouched files cannot change verdict: parserOptions.project and projectService are null for every file, so type-aware linting is not enabled.

Changesets

  • .changeset/20745-nested-object-door.md: @objectstack/objectql minor, a BREAKING banner, Clause-②: no (narrowing), and the ADR-0087 marker not-required (no-migration-prescription), as the [finding] a plain object with no $ key as a scalar field's filter value answers 200 with no rows on the memory driver and INVALID_FILTER 400 on SQLite and PostgreSQL #20546 changeset has. It states what an author sees now and the route that works, per kind, with the table. It says it supersedes the "Unchanged" paragraph of the pending scalar-field entry (20546-no-operator-object-on-scalar) for relation and structured-JSON fields.
  • .changeset/20745-nested-relation-prose.md: @objectstack/spec patch for the shipped JSDoc.
  • No export or published type changes: the door module is internal, and @objectstack/objectql's root and ./core exports are unchanged. check:api-surface is green.

Acceptance notes

  • Out of scope, reported to the PM, not filed:
    • The published skill skills/objectstack-query teaches the nested-relation form as working: SKILL.md "Nested Relation Filters", the "Filter parent by child conditions" row and the search paragraph, and rules/filters.md "Nested Relation Filters". It was already untrue before this change (memory answered no rows, SQL 400). It is a governed Tier H surface outside this claim's file surface, so it is not touched here.
    • A groupBy of a json field answers 500 DATABASE_ERROR on PostgreSQL. On InMemoryDriver it merges every row into one group (n: 3), and on SQLite it gives one group per serialized value.
  • File and media fields keep the [finding] The FILTER axis has no DOTTED-path verdict — where: { project_id.name: 'x' } rides its head segment past both doors, where SORT refuses the same spelling (#4256) #8371 carve-out and stay unjudged. { photo: { url: 'x' } } still answers memory 200 with no rows and SQL 400.
  • At having there is no field declaration, so a relation column's words say "the related object" and give the $in spelling. A having over a multi-valued relation groupBy would get that single-valued spelling.
  • referenceTargetOf names the related object in the relation words. For a field whose reference carrier never went through the schema's parse (a non-string), it throws its own TypeError instead of the refusal. Parse refuses that shape at the contract door.

Generated by Claude Code

…tured-JSON or provisioned column at the engine door (WIP)

Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY
Co-authored-by: Claude <noreply@anthropic.com>
…, live dialects and InMemoryDriver (WIP)

Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY
Co-authored-by: Claude <noreply@anthropic.com>
…; spec prose, docs and changesets

Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY
Co-authored-by: Claude <noreply@anthropic.com>
…te cell is a measured reading

Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY
Co-authored-by: Claude <noreply@anthropic.com>
… engine's own refusal

Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/objectql, @objectstack/spec, touching 25 documentable anchor(s). ⚠️ 1 changed file(s) yielded no anchor (packages/objectql/src/having-filter.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

21 hand-written doc(s) name something this change touched — list omitted above 15 rows. Re-derive on the tree named below: node scripts/docs-audit/affected-docs.mjs --json 1bcba27d2e73242c6169d24e12031b3a9f52b44d.

⛔ 5 release-owned page(s) also affected — read-only, see AGENTS.md Documentation Guardrails.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/objectql/src/having-filter.ts) — pages documenting those are invisible to this run
  • 1 anchor(s) matched too much of the corpus to be a work list: created_at (literal, 35 pages)
  • 5 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 54 of 206 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 138 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 1bcba27d2e73242c6169d24e12031b3a9f52b44d → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 870fcbb63c7b6939f306f79d8f31c36188595b71 — the merge of head ee17b18fdec9355e633b8f651518d2f30d98a984 into base 1bcba27d2e73242c6169d24e12031b3a9f52b44d, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 870fcbb63c7b6939f306f79d8f31c36188595b71 && git checkout 870fcbb63c7b6939f306f79d8f31c36188595b71
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 1bcba27d2e73242c6169d24e12031b3a9f52b44d ee17b18fdec9355e633b8f651518d2f30d98a984 && git checkout -B drift-repro 1bcba27d2e73242c6169d24e12031b3a9f52b44d && git merge --no-ff ee17b18fdec9355e633b8f651518d2f30d98a984

node scripts/docs-audit/affected-docs.mjs --json 1bcba27d2e73242c6169d24e12031b3a9f52b44d

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 1bcba27d2e73242c6169d24e12031b3a9f52b44d → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: ee17b18fdec9355e633b8f651518d2f30d98a984
Local-runs: none

PR #20781 for card #20745, branch claude/issue-20745-nested-object-door: 14 files, +1008 / -94, the net diff against origin/main from merge-base eead9dcf40 (one merge of origin/main on top, no other history). The PR file list and the os-dev-report's files_changed name the same 14 files. PR #20744 (#20546, merged as 97005aed0) is the door this extends. The PR head read ee17b18fde at the start and at the end of this review; it did not move. Read-only throughout: the diff, the card and its four comments (triage 5902573237, the claim, the os-dev-report, the seat answer 5904634845), the PR body, PR #20744, the spec and driver sources on the head, and the head's check-runs. Nothing built, run or re-run.

Check-runs on the head, the newer of two readings this round (this comment posted 2026-09-30T05:31Z): 34 runs, 27 success, 2 skipped (Console Pin Gate, Packed-tarball smoke — both not applicable to this diff), 5 still in_progress (Test Core 1/6, 3/6, 5/6, 6/6 and Lint & Repo Gates), 0 failed. Green already: Check Changeset (the job that runs check-empty-changeset, check-adr-0087-registration and check-changeset-no-major on a PR), all four Type Check jobs, Build Core, Build Docs, Temporal Conformance (live PG + MySQL), Dogfood Regression Gate and Verify CLI, Spec property liveness, Governed Surface Queue Guard, the claim and single-writer guards. Not waited for or polled; landing still waits on every check green, which the queue guard enforces, not this record.

① Derived judgments

Every accept-set and public-surface change the diff implies, each judged:

  1. Relation kind narrowed — RIGHT. A no-operator object beneath a column whose declared type is in the spec's REFERENCE_VALUE_TYPES (lookup, master_detail, user, tree, single or multiple: true) is refused INVALID_FILTER / 400 at the engine, before any driver, at where (object form and FilterArray sugar; find / findOne / count / aggregate / update / delete / judgeFilter), aggregations[i].filter and having. It lands as a third kind in the existing walkCondition arm (number-comparand-declared-type-door.ts); the diff adds no walk and touches no driver file, so triage's "no second door" and the claim's "no driver edit" hold. The form no driver serves: the column stores the related record's id, so memory's equality on that id answers no rows, the SQL driver refuses the bind, and the mongodb driver's embedded-document equality on an id string answers no rows — none follows the relation.
  2. Structured-JSON kind narrowed — RIGHT. The same object beneath STRUCTURED_JSON_TYPES (json, composite, repeater, record, location, address, vector) is refused alike. The drivers share no meaning for it (memory deep-equals the document, SQL refuses the bind), so one loud answer is the only shared one.
  3. Platform-provisioned id / created_at / updated_at when the declared map omits them — RIGHT (seat answer A). declaredFactsOf now returns a scalar column for exactly those three names (text, datetime, datetime), read from the same set engine.ts calls PLATFORM_PROVISIONED_COLUMNS at the write gate and find / findOne add to their known set. Every other undeclared key keeps the registry-less tolerance (pinned: not_a_field, owner_id, organization_id, _id are never judged). After the arm, a provisioned key with a scalar or operator value falls to facts.number === null and continue, so { id: 'd1' } and { id: { $in: [...] } } pass untouched (pinned). One nuance, not a defect: on an object whose declared map is empty the write gate stands down entirely, while this arm still judges an object beneath id — an object can match no id anywhere, so refusing it is the truthful answer.
  4. Per-aggregation filter with a JSON object — RIGHT. It was the one cell that answered alike before (count 1, the engine's own deep equality — the dev's measurement, not re-measured here) and is refused now, so one filter has one answer at every position. The changeset names it in its own table row.
  5. {} beneath a relation or JSON column — RIGHT. Refused in the arm's words instead of each driver's zero-operator words; the code and status do not move.
  6. having over a relation or JSON column — RIGHT. aggregatedRowColumnTypes carries a groupBy's declared type and a min / max field's type, and the arm judges by that type; pinned for a lookup groupBy, a max of a master-detail and a json groupBy. (The spec's AGGREGATE_FIELD_TYPE_COMPATIBILITY refuses max over a reference type, but its consumers are packages/lint, not engine.aggregate, so the pin is reachable and the words fire first.)
  7. The [finding] a plain object with no $ key as a scalar field's filter value answers 200 with no rows on the memory driver and INVALID_FILTER 400 on SQLite and PostgreSQL #20546 scalar words rewritten — RIGHT, a words change only. The middle sentence ("only a relation field ... can carry") became false with this change and is gone; every kind now reads position, verdict, The filter was NOT applied., route, reasoning. CLIENT_MESSAGE_MAX = 500 in packages/rest/src/error-response.ts truncates a 4xx message, describeObject caps the shown keys at three, and the REST pin asserts the route text inside the body for every refused row. The existing [finding] a plain object with no $ key as a scalar field's filter value answers 200 with no rows on the memory driver and INVALID_FILTER 400 on SQLite and PostgreSQL #20546 pins asserted none of the removed sentence (their diff changes only the controls).
  8. The unjudged remainder is closed and correct. Reading the FieldType roster against the judged sets leaves exactly the file and media types (image, file, avatar, video, audio, the [finding] The FILTER axis has no DOTTED-path verdict — where: { project_id.name: 'x' } rides its head segment past both doors, where SORT refuses the same spelling (#4256) #8371 carve-out — a legacy stored value is an inline object) and formula (not in SCALAR_FILTER_HEAD_TYPES because it is SEARCH_VIRTUAL_TYPES; refused one door earlier with INVALID_FIELD) as null; plus every undeclared non-provisioned key, { $field } references, operator bags, and arrays / Date / Map / class instances (not isNoOperatorObject). The GUARD pins the classification over every FieldType option and the controls pin the routes, a file field and an unknown key reaching the driver as written.
  9. The routes the words name, judged per driver. (a) $in on the related object's ids for a single-valued relation: a scalar id column on every SQL dialect, memory and mongodb — works. (b) $contains per id (an $or of those for several) for a multi-valued relation: on SQL the only working spelling — $in is in the driver-sql: a declared operator on a multiple: true (JSON array) column silently answers wrong — $in/$eq always zero rows, $nin returns the rows it was asked to EXCLUDE #7398 JSON_COLUMN_INCOMPATIBLE_OPERATORS set and refused, while $contains lowers through the driver-sql: the $contains MEMBERSHIP spelling on any multi-valued / JSON column is a DATABASE_ERROR 500 on live PostgreSQL (SQLSTATE 42883, operator does not exist: json ~~ text) — it has only ever been executed on SQLite #17590 jsonMembershipPredicate to exact element membership on SQLite, PostgreSQL and MySQL; on memory and mongodb $contains lowers to a regex over each element, which matches a full id (measured d1, d3) — that it is a substring test rather than exact membership there is a pre-existing residue of $contains on the document-shaped drivers, not this card's. (c) { field: { $null: false } } for JSON: IS NOT NULL on SQL, a != null arm on memory and mongodb — works. (d) The dotted path owner.region is correctly not offered: the [finding] The FILTER axis has no DOTTED-path verdict — where: { project_id.name: 'x' } rides its head segment past both doors, where SORT refuses the same spelling (#4256) #8371 dotted door classifies a reference head as relation and refuses it INVALID_FIELD on every driver. (e) Triage's $contains example for a JSON field is correctly not offered: the [Decision] refuse a text operator ($contains family) over a field whose DECLARED type is not textual — INVALID_FILTER 400 at the engine's field-aware door (option C of #14079); the textual-type vocabulary is the question #15661 text-operator door refuses every text operator over STRUCTURED_JSON_TYPES.
  10. Public surface: none moves — RIGHT. no-operator-object-door.ts and number-comparand-declared-type-door.ts are exported from neither index.ts nor core.ts, so the new functions and the changed NoOperatorObjectRefusal shape are internal. FilterCondition, Filter, FilterConditionSchema and QueryFilter change in JSDoc and comments only (the diff to filter.zod.ts is 14 / 8 lines of comment text; no Zod builder or type member moves), so the generated spec artifacts are unaffected (JSDoc is not .describe()); Type Check · source gates is green. @objectstack/objectql's . and ./core exports are unchanged.
  11. The review faces, sentence by sentence. content/docs/kernel/contracts/data-engine.mdx: "refused ... on every driver" true by construction; "no driver follows a relation into the related object" true; "a whole-value match on a JSON value means something different on each backend" true; the $contains-per-id sentence for multiple: true true (on SQL it is the only spelling; elsewhere it works too); the replacement example uses engine.find with fields: ['id'], the page's own idiom (its opening example). The objectql changeset: every type it names is in the spec set it cites; "the SQL driver refuses $in on a multi-valued column" true; "A dotted path ... already refused with INVALID_FIELD on every driver" true; "$contains ... already refused over a JSON value on every driver" true; the provisioned types (text, datetime) true; "inside the first 500 characters the REST door keeps" true for the pinned shapes; "Supersedes the Unchanged paragraph of 20546-no-operator-object-on-scalar" true, both entries pending. The spec changeset: "stays in the type and the schema (nothing is narrowed)" true; the routes true; the QueryFilter example and Filter comment sentences true. filter.zod.ts form 4 prose true throughout, and the inline comment in checkFilterConditionComparands ("refuses both at query time") true. The refusal words: the relation words' "stores the related record's id, no driver follows it" true; the JSON words' "one compares the documents, another refuses the bind" true; the provisioned id words' "holds scalar values" true. The PR body: "no second door and no second traversal" true; the H4 reasoning true (an index-signature union with any collapses to any; the generic Filter nested arm is a separate member whose removal would narrow every object-typed property); "no export or published type changes" true; the ablation arithmetic matches the pin structure (11 new refusal tests + the 2 rewritten controls = the 13 objectql failures; the 4 REST failures are the two refusal tests on SQLite and live PostgreSQL); the "Before" tables are the dev's measurements on the base, not re-measured here, and every cell agrees with the driver code read above. No false sentence found on any face.

② Semver level

  • Clause-②: no (narrowing) on line 2 of the PR body and in the objectql changeset — right: nothing widens a published payload; the accept set narrows, which is BREAKING under the closed pair.
  • @objectstack/objectql minor with a BREAKING banner and one ADR-0087 marker not-required (no-migration-prescription) — right. minor is the launch-window convention check-changeset-no-major enforces (breaking-ness carried by the banner plus the ADR-0087 disposition, not the level); the marker is the same category the merged [finding] a plain object with no $ key as a scalar field's filter value answers 200 with no rows on the memory driver and INVALID_FILTER 400 on SQLite and PostgreSQL #20546 entry carries for the same door, and its facts hold (no authorable key, export or stored shape moves; no mechanical rewrite exists because which related records or JSON part the caller meant is not in the object). Check Changeset, which runs check-adr-0087-registration and check-changeset-no-major on this PR, is success on the head. The body states what an author sees now, the route per kind, who is affected and the measured table — the migration text an upgrading agent needs.
  • @objectstack/spec patch — right. The FilterCondition / Filter / QueryFilter JSDoc ships in the published .d.ts, so the diff publishes and skip-changeset would be wrong; it widens no public surface, so patch is the level the Check Changeset step's WHICH LEVEL prose gives (precedent: 20288-service-realtime-emitted-event-examples.md, a docs-only patch). The diff to packages/spec is prose only, as the changeset says. Two changesets rather than one is right: a JSDoc patch must not put a BREAKING banner in spec's CHANGELOG.

③ Boundary flags

Every dev flag from the os-dev-report and the PR body, and both open_questions:

  1. open_questions[0], the id row — seat ruled A; the implementation is A; judged right. Triage's "existing unknown-field verdict" is tolerance on origin/main (declaredFactsOf returned null for any key outside the map), which is per-driver, so it and the Pins ruling could not both hold; reading id as platform-provisioned, as the write gate, find / findOne and the REST ingress already do, makes the Pins row true and adds no second opinion about names. Answered.
  2. open_questions[1], the kinds' scope — seat ruled A; implemented as the spec's classes; judged right. One closed classification from REFERENCE_VALUE_TYPES and STRUCTURED_JSON_TYPES, the same shape [finding] a plain object with no $ key as a scalar field's filter value answers 200 with no rows on the memory driver and INVALID_FILTER 400 on SQLite and PostgreSQL #20546 took from SCALAR_FILTER_HEAD_TYPES; the changeset names every type each set reaches, so it states the ruling truthfully. Answered.
  3. $contains refused on JSON, so the words name $null and a stored field — verified against the [Decision] refuse a text operator ($contains family) over a field whose DECLARED type is not textual — INVALID_FILTER 400 at the engine's field-aware door (option C of #14079); the textual-type vocabulary is the question #15661 text-operator door; the route named works on every driver (①.9c). Answered.
  4. The [finding] a plain object with no $ key as a scalar field's filter value answers 200 with no rows on the memory driver and INVALID_FILTER 400 on SQLite and PostgreSQL #20546 scalar words rewritten for the 500-character bound — verified (①.7). Answered.
  5. The per-aggregation JSON filter, consistent before and refused now — a narrowing inside the declared arm, named in the changeset table (①.4). Answered.
  6. Two changesets — right (②). Answered.
  7. The runtime memory pin dropped for check:driver-memory-census — the gate is real (a ledger plus a gate, a maintainer-ruled closed set of in-memory-driver test consumers), so a new consumer needs a ruling this card does not carry. The InMemoryDriver refusal cells are pinned by construction (the objectql suite's recording driver: the arm answers before any driver is resolved); the memory route readings are measured in the PR body and the module header, not pinned. Accepted as is; no new gate family opened.
  8. Two existing controls rewritten (protocol-explicit-filter-field-gate.test.ts Data query: an unknown field inside where / $filter answers 200/0 instead of 400 INVALID_FIELD — the bare-key door disagrees (#4134's uncovered sibling) #7534, query-expression-conformance.test.ts [finding] The FILTER axis has no DOTTED-path verdict — where: { project_id.name: 'x' } rides its head segment past both doors, where SORT refuses the same spelling (#4256) #8371) — they pinned that the name gates never descend into the nested form, and they still pin exactly that: the answer is the engine's INVALID_FILTER in the nested-relation words with err.field undefined, never the name gate's INVALID_FIELD. Inside the claim's "tests in packages/objectql" surface. Answered.
  9. origin/main merged once — one merge commit at the head, the net diff unchanged by it. Answered.
  10. data-engine.mdx replaced rather than deleted — the working route plus one paragraph; the triage aim (the page stops promising what nothing serves) is met and the example is the page's idiom. Answered.
  11. Out-of-scope, reported not filed — escalated to the seat, not this PR's: (a) the published skill skills/objectstack-query teaches the form the engine now refuses (verified on origin/main: SKILL.md lines 74, 86, 185 to 191, 332, 341; rules/filters.md line 132); it is Tier H and correctly untouched here, and it should be filed as a contract-text defect with the dev's dedupe words. (b) groupBy over a json field answering 500 DATABASE_ERROR on PostgreSQL: not re-measurable read-only; the dev reports a measured reach and dedupe words, so it should be filed. (c) File and media objects still answer per driver — the ruled [finding] The FILTER axis has no DOTTED-path verdict — where: { project_id.name: 'x' } rides its head segment past both doors, where SORT refuses the same spelling (#4256) #8371 carve-out, pinned as the control; accepted. (d) The pending 20546-no-operator-object-on-scalar changeset's "Unchanged" paragraph is now superseded and the 20745 entry says so explicitly; both are unreleased, so a changeset-only edit to the 20546 entry before the release consumes them would give one coherent CHANGELOG text. Recommended, not blocking.
  12. Acceptance notes — a having over a multiple: true relation groupBy gets the single-valued $in spelling because no declaration is readable there: a bounded imprecision in words at a rare position, accepted. referenceTargetOf throws the spec's own TypeError for a non-string reference carrier that never went through parse: the contract door refuses that shape, and the throw is the total-accessor rule the spec already keeps; accepted.

Implemented-by: claude/issue-20745-nested-object-door
Reviewed-by: session_01DEvba2nBuD4tWzfq8r8NFY

VERDICT: PASS


Generated by Claude Code

akarma-synetal pushed a commit to akarma-synetal/framework that referenced this pull request Oct 7, 2026
…ID_FIELD / 400 at the engine aggregate door, on every driver (objectstack-ai#20783) (objectstack-ai#20804)

Fixes objectstack-ai#20783
Clause-②: no (narrowing)

## What this changes

A `groupBy` entry that names a declared **structured-JSON** field
(`json`, `composite`, `repeater`, `record`, `location`, `address`,
`vector`) is now refused by `engine.aggregate` with `INVALID_FIELD` /
400, in the engine's words, before any driver is asked. It holds on
every driver and for every caller that reaches the engine: the REST
query door, a flow or hook calling the engine in-process, and the
analytics strategy that lowers a cube query onto `engine.aggregate`.
Both entry spellings are judged: the field name (`groupBy[0]`) and the
`{ field }` object (`groupBy[0].field`), a `dateGranularity` bucket
included.

The words, as `POST /api/v1/data/:object/query` returns them (the route
is inside the 500 characters the REST door keeps, and the REST pin
asserts it there):

```text
aggregate('rest_group_by_json_20783'): groupBy[0] names 'meta', a declared json field — a structured-JSON value, which the engine does not group by. The query was NOT run. Group by a field that stores one scalar value: store the part you group on in a field of its own and group by that field. A JSON document is no group key the drivers share: one merged every row into a single group, one grouped each serialized document apart, one refused the statement.
```

The thrown error carries `code: 'INVALID_FIELD'`, `status` and
`httpStatus` 400, `field`, `fields` (every offending entry), `object`
and `param: 'groupBy'`.

**Landing site, as the order expected: `packages/objectql`'s aggregate
admission. No driver file, no serialization rule.**

- `packages/objectql/src/group-by-structured-json-door.ts` (new):
`assertGroupByNamesNoStructuredJsonField(object, schema, groupBy)`. The
class is the spec's `STRUCTURED_JSON_TYPES` (the set PR objectstack-ai#20781's JSON
arm judges), never a list minted here. Not judged: an undeclared name
(the engine's registry-less tolerance; the REST ingress answers an
unknown name `INVALID_FIELD` first), a host with no field map, and every
other type.
- `packages/objectql/src/engine.ts`: one call at the entry of
`aggregate`, right after `rejectCredentialAggregation` (which reads the
same `groupBy` entries), so a protected field keeps that refusal's
words. `aggregate` is the only engine verb that takes `groupBy`: `find`
refuses the key (`ENGINE_FIND_OPTION_KEYS`).
- `packages/objectql/src/number-comparand-declared-type-door.ts`:
comment only. Its sentence "a `json` … groupBy … is judged now" at
`having` became false for a json groupBy, which no longer reaches
`having`.

**Why `INVALID_FIELD`, an existing code.** The verdict is about the
named field's type at a position. That is the question the REST ingress
answers with `INVALID_FIELD` for an unknown `groupBy` name
(`assertGroupByFieldsExist`), the search axis answers with
`INVALID_FIELD` for a field whose type it cannot scan, and the engine's
`assertFilterIsMaterializable` answers with `INVALID_FIELD` for a
virtual field ("this verdict is about the NAME's type").
`INVALID_FILTER` is the engine's value-shape envelope and `groupBy` is
not a filter; `INVALID_QUERY` is the ingress's malformed-shape code, and
the entry here is well formed.

## Before, measured on `origin/main` `7a09eee1b1`

Through `POST /api/v1/data/:object/query` (the real `RestServer` route
over `ObjectStackProtocolImplementation` and `ObjectQL`) with `{
groupBy: [FIELD], aggregations: [{ function: 'count', alias: 'n' }] }`,
and through `engine.aggregate` directly (same answers). Drivers:
InMemoryDriver, SqlDriver on SQLite (better-sqlite3), and SqlDriver on a
private PostgreSQL 16.13 started for this run. Three rows: `title` x, x,
y; `meta` `{a:1}`, `{a:2}`, `{b:1}`; and one differing value per row
under every other structured-JSON field.

| `groupBy` | InMemoryDriver | SQLite | PostgreSQL 16 |
|:--|:--|:--|:--|
| `title` (text, the control) | 200, `x` 2 · `y` 1 | same | same |
| `meta` (json, the card) | 200, **one group** `{a:1}`, `n` 3 | 200, one
group per serialized document (3) | **500 `DATABASE_ERROR`** ("could not
identify an equality operator for type json") |
| `composite`, `repeater`, `record`, `location`, `address` | 200, one
group, `n` 3 | 200, one group per serialized document | 500 |
| `vector` | 200, one group per array (`[1,2]` 2 · `[3,4]` 1) | 200, one
group per serialized array | 500 |
| `{ field: 'meta' }` | 200, one group, `n` 3 | 200, 3 groups | 500 |
| `{ field: 'meta', dateGranularity: 'month' }` | 200, one `null`
bucket, `n` 3 | 200, one `null` bucket, `n` 3 | 500 ("cannot cast type
json to timestamp with time zone") |
| `['title', 'meta']` | 200, 2 groups | 200, 3 groups | 500 |

## After, the same run on this branch (`43ae6c1fcc`)

Every structured-JSON row above answers `400 INVALID_FIELD` in the
engine's words on all three drivers, naming the position (`groupBy[0]`,
`groupBy[0].field`, `groupBy[1]` for the mixed entry), the field and its
declared type. No read of the object runs. The `title` control answers
`x` 2 · `y` 1 on all three, unchanged. The InMemoryDriver cells of this
run came from a scratch script over the built packages. They are not
committed: `check:driver-memory-census` refuses a new test consumer of
that driver without a ruling, so the committed memory cell is the
recording driver below, by construction.

## Hypotheses (zone 2): which held

- **H1: held.** `aggregate` resolves `groupBy` entries against the
declared field map before the driver in `rejectCredentialAggregation`
(credential and `internal` fields) and, for `having`, in
`aggregatedRowColumnTypes`. The refusal sits beside the first, at the
verb's entry, so it runs before the per-aggregation `filter` and
`having` doors and before any driver is resolved. Code: `INVALID_FIELD`,
reasons above. The REST ingress also resolves `groupBy` names
(`assertGroupByFieldsExist` in `metadata-protocol`), but only for the
REST path. The engine is the one door every caller shares, as triage
directed.
- **H2: held, refined.** Every member of `STRUCTURED_JSON_TYPES` answers
per driver today, and none answers one way. Six members split exactly
like `json` (memory one merged group, SQLite per serialized document,
PostgreSQL 500). `vector` splits differently on memory (one group per
array, not one merged group) and still 500 on PostgreSQL, so it does not
answer one way either. So the class is refused, not `json` alone.
- **H3: held, and it was NOT already refused.** A date-bucketed `{
field, dateGranularity }` over a `json` field passed the REST ingress (a
known field, a valid granularity) and answered one `null` bucket on
memory and SQLite and 500 on PostgreSQL. It is refused now with the `{
field }` form's words, since no granularity makes a JSON document a
date.
- **H4: measured. The analytics face is partly the same door.** Measured
through `AnalyticsService.query` wired with `AnalyticsServicePlugin`'s
own auto-bridges (`executeAggregate` to `engine.aggregate`,
`executeRawSql` to `engine.execute`), with a cube dimension `sql:
'meta'` on a `json` field:

  | cube query | InMemoryDriver | SQLite | PostgreSQL 16 |
  |:--|:--|:--|:--|
| `dimensions: [meta]`, before | 200, one merged group (raw SQL
unsupported, fell back to `engine.aggregate`) | 200, one group per
serialized document (**native SQL**, engine not reached) | 500
`DATABASE_ERROR` (**native SQL**) |
| `dimensions: [meta]`, after | **400 `INVALID_FIELD`** (this door) |
unchanged | unchanged |
| `timeDimensions: [{ meta, granularity: month }]`, before | 200, one
`null` bucket | 200, one `null` bucket | 500 |
| the same, after | **400 `INVALID_FIELD`** (the native strategy
declines a granularity, so `engine.aggregate` serves it) | **400** |
**400** |

So the ObjectQL strategy (any cube query on a driver without raw SQL,
and any bucketed time dimension) reaches this door and is folded in with
no `packages/services` edit. **`NativeSQLStrategy` does not**: on SQL
drivers it compiles `GROUP BY "meta"` by hand and bypasses the engine.
That half is a sibling card (reported to the PM below, not filed here).
The dataset compiler only checks that a dimension's field is declared
(`assertDeclared`), so a dataset dimension on a `json` field compiles
into the same cube dimension and splits the same way. The memory cube
face (`MemoryAnalyticsService`) has **zero** production constructors
(`git grep "new MemoryAnalyticsService"` outside tests: 0 hits at
`7a09eee1b1`), so it reaches a driver only in tests. It was not edited
and not measured for this shape.

## Producers measured (for "refuse, don't define")

Zero producers group by a structured-JSON field. Every `grouping` /
`groupBy` / `groupByField` / `dimensions` target in `examples/` at
`7a09eee1b1` is one of `status`, `priority`, `created_at`, `account`,
`stage`, `total`, `region`, `sales_region`, `progress`, `issued_on`,
`industry`, `category`, `signed_on`, `health`, `completed_date`,
`close_date`, and none is structured-JSON (the showcase's
structured-JSON fields are `field-zoo`'s `f_*`, `account.hq`,
`account.support_config` and `task.location`). The same count over the
platform packages names `object_name`, `user_id`, `id`, `action`,
`topic`, `provider_id`, `organization_id`, `namespace`, `kind`,
`actor_id` and `phone`, plus two dynamic engine callers in
`service-analytics` (see the acceptance notes).

## Tests

- New `packages/objectql/src/engine-group-by-json-door.test.ts` (6
tests, recording driver, so the in-memory cell by construction). It
covers every structured-JSON type with the full envelope (`code`,
`status`, `httpStatus`, `field`, `fields`, `object`, `param`) and the
position, and asserts zero reads. It covers the `{ field }` form, a date
bucket, an entry after a scalar one and two offenders, and the REST door
into `findData`. Controls: `text`, `number`, a `multiple: true` select,
an image field and an undeclared name all reach the driver, and a
structured-JSON field as an aggregated column is unchanged. GUARDs: the
judged types equal `STRUCTURED_JSON_TYPES` over every `FieldType`, and
there is no verdict without a field map, for an undeclared name, or for
an entry naming no field.
- New `packages/rest/src/data-group-by-json-door.test.ts`: SQLite
always, PostgreSQL / MySQL where `OS_TEST_POSTGRES_URL` /
`OS_TEST_MYSQL_URL` are set. Every structured-JSON type answers 400
`INVALID_FIELD` with the route in the REST body and zero reads. The
object and bucket forms and the mixed entry answer the same 400. The
control `title` answers `x` 2 · `y` 1 from the driver. ⚠️ As with the
sibling door suites, no CI job sets those URLs for `@objectstack/rest`,
so the live cells run only locally. Local run with the private
PostgreSQL 16.13: **6 passed (sqlite 3, live postgres 3) / 3 skipped
(mysql, no URL)**.
- Fixture triage in `engine-nested-object-door.test.ts` (PR objectstack-ai#20781's
pin): its `having` case over `groupBy: ['meta']` pinned the branch this
PR closes, because a json groupBy never reaches `having` now. It is
**replaced**, not respelled. The JSON arm at `having` is still reached
through a `max` of a json field (`having: { top_meta: { a: 1 } }`),
which answers the same `INVALID_FILTER` whole-value words. No other
fixture groups by a structured-JSON field. Measured by a grep over every
test that both groups and declares a structured-JSON type; the
downstream suites below stayed green.
- `pnpm --filter @objectstack/objectql test` at `43ae6c1fcc`: **342
files / 6739 passed**.
- `pnpm --filter @objectstack/rest test` at `43ae6c1fcc` (with the live
PostgreSQL URL set): **232 files / 4516 passed / 33 skipped**.
- `typecheck` for `@objectstack/objectql` and `@objectstack/rest` at
`43ae6c1fcc`: exit 0, including each `check:test-typecheck` (objectql's
ledger held at 40 files / 234 errors / 65 signatures; rest 0), so both
new test files compile.
- Downstream consumers (`...@objectstack/objectql` direction), at
`cafaf885d8`: `@objectstack/service-analytics` **141 files / 3267
passed**; `@objectstack/metadata-protocol` **190 files passed, 3 skipped
/ 2792 passed, 19 skipped**. The other consumers are declared to CI.

**Reverse verification (ablation), from the committed fix at
`1eacad5296`.** It ran through `scripts/ablation-replace.mjs` in WRAP
mode, trap-restored. The engine's call
`assertGroupByNamesNoStructuredJsonField(object,
this._registry.getObject(object), query.groupBy);` was fed `(query as {
__ablated_20783__?: unknown }).__ablated_20783__` (always undefined)
instead of `query.groupBy`. On disk the anchor went 1 to 0 and the
marker 0 to 1, with blob `67198fc4a8fc` to `837fc9257e37`. objectql was
rebuilt, and `ablation-dist-preflight` found the marker in 4 built
files.

- Predicted direction: red. Observed: red.
- objectql pins: **3 failed / 18 passed**. Every refusal case failed.
The controls, both GUARDs and the whole
`engine-nested-object-door.test.ts` stayed green.
- rest pins: **4 failed / 2 passed / 3 skipped**. The refusal cases
failed on SQLite and live PostgreSQL, and the `title` control stayed
green.
- Restore leg: blob equals HEAD (`67198fc4a8fc`), `git diff HEAD` is
empty, and whole-tree `git status --porcelain` is empty. After a
rebuild, the `--absent` preflight found the marker absent from all 14
built files, and the tree reading was clean. Then both pin sets were
green again (21 passed; 6 passed + 3 skipped).

## Gates

`node scripts/pm/dispatch-gates.mjs --commands` at `43ae6c1fcc` (the
merge of `origin/main` `96e724475c`, which touches none of this diff's
packages) derived **65** commands over the 7 changed paths. All 65 were
run on `43ae6c1fcc`, and `--ran` reconciles them: **65 derived, 65 run,
0 NOT-MEASURED, 0 UNRUN**, all exit 0. Among them:

- `check:adr-0087-registration --base origin/main`: `not-required
(no-migration-prescription)` accepted.
- `check:changeset-no-major`, `check:empty-changeset`,
`check:doc-authoring`, `check:nul-bytes`, `check:issue-citations`.
- `check:engine-double-contract`, `check:where-matcher`,
`check:driver-memory-census`, `check:cross-package-test-inputs`,
`check:test-source-alias`, `check:type-check-coverage`,
`check:type-check-debt`.
- `check:query-options-erasure`. It caught a first draft of the REST pin
that erased an `engine.aggregate` options bag to `any` (test surface 236
to 237). Fixed by typing it (`cafaf885d8`), and the surface is back at
236.
- `check:dual-build-cjs-loads`, after a full `turbo run build` of
`./packages/*` and `./packages/*/*`.

Lint, narrowed and proven: `pnpm exec eslint --no-inline-config --format
json` over the 6 changed `.ts` files at `43ae6c1fcc` found **6 files, 0
errors, 0 warnings**. Three facts make this narrowing a measurement:

- The checked population comes from eslint's own config: `isPathIgnored`
answers `false` for all 6.
- The file count comes from the JSON output: 6 results.
- Untouched files cannot change verdict: `parserOptions.project` and
`projectService` are `null` for every file, so type-aware linting is not
enabled.

## Changeset

`.changeset/20783-groupby-structured-json-refused.md`:
`@objectstack/objectql` `minor`, a BREAKING banner, `Clause-②: no
(narrowing)`, and exactly one ADR-0087 marker, `not-required
(no-migration-prescription)`, in the form PR objectstack-ai#20781's changeset uses. It
carries no rewrite table and no arrow: the body states what an author
sees now, why, who is affected and what is unchanged. No export or
published type changes: the door module is internal, and
`@objectstack/objectql`'s root and `./core` exports are unchanged.

## Acceptance notes

- **Reported to the PM, not filed here (same family, measured):**
- **The native-SQL analytics face** (triage's H4 sibling). A cube or
dataset dimension on a structured-JSON field through `NativeSQLStrategy`
answers one group per serialized document on SQLite and 500 on
PostgreSQL. It never reaches this door. The fix lands in
`packages/services/service-analytics`, which this claim does not touch.
- **`groupBy` on a `multiple: true` select** through `POST
/api/v1/data/:object/query`: memory answers one group per array, SQLite
one per serialized array, and PostgreSQL 500 (json equality). Not this
card's class, and a multi-value field has a plausible other meaning (a
bucket per member), so it is not refused here.
- **`count_distinct` over a `json` field** through the same door: memory
3, SQLite 3, PostgreSQL 500. `AGGREGATE_FIELD_TYPE_COMPATIBILITY`
accepts `count_distinct` for every `FieldType` on the ground that every
backend gives one answer. PostgreSQL does not for json.
- `service-analytics` has two dynamic `engine.aggregate` callers:
`resolveFkAttr` (`groupBy: ['id', attr]`, a cross-object dimension
attribute) and the display-label pass (`groupBy: ['id', displayField]`).
If that attribute or display field is structured-JSON, they now answer
this 400. Before, by reading and not measured: memory and SQLite grouped
under `id` first, so one row per record, and PostgreSQL would have
answered the same json-equality 500. No example or platform dataset
names such a dimension (the census above).
- A refusal relayed through the analytics engine path names the engine's
position (`groupBy[0]`), not the cube member the caller wrote
(`c20783.meta`). Carrier: the native-SQL sibling card, which touches the
same face.
- The native-SQL analytics path returned PostgreSQL's `count` as a
string (`"2"`) where SQLite returned a number, in the stand-in wiring
above. Observed only; not measured through the HTTP door.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
akarma-synetal pushed a commit to akarma-synetal/framework that referenced this pull request Oct 7, 2026
…d record's column, not the refused nested form (objectstack-ai#20811)

Fixes objectstack-ai#20782
Clause-②: no

## What this changes

The published skill `skills/objectstack-query` taught `{ relation: {
field: value } }` beneath a lookup as a working `where` form. On `main`
the engine refuses that form on every driver — `INVALID_FILTER` / 400,
in the words of `relationWords()` in
`packages/objectql/src/no-operator-object-door.ts` — and names the route
it serves: filter the related object first, then match the relation
column against the ids it returns (`$in` on a single-valued column;
`$contains` per id, an `$or` of those for several, on a `multiple: true`
one, whose JSON column the SQL driver refuses `$in` on). The skill now
states that refusal and that route at every site that taught or pointed
at the form. It anticipates neither letter of the open v18 decision
(objectstack-ai#20802 is not addressed here): it states today's behaviour.

Six landing sites, all in `skills/objectstack-query`, measured on
`origin/main` `96e72447`:

| Site (at `96e72447`) | Before | After |
|:--|:--|:--|
| `SKILL.md:76`, Removed-key row `query.joins` | "`expand`, or a nested
relation filter" | "`expand` (display), or filter the related object and
`$in` its ids" |
| `SKILL.md:88`, rules index | "nested relations" | "filtering by a
related record" |
| `SKILL.md:187-:194`, subsection | "Nested Relation Filters" with the
`{ contact: { profile: { verified: true } } }` example | "Filtering by a
related record": the refusal, the route, a pointer to the rule |
| `SKILL.md:334` | "use a nested relation filter" | "`$in` ids from its
own query" |
| `SKILL.md:343`, Cross-Object row | "Filter parent by child conditions
— Nested relation filter" | two rows, one per direction (below) |
| `rules/filters.md:132-:152`, section | "Nested Relation Filters", two
✅ examples of the refused form | "Relation Filters": the refusal, one
two-step example, the multi-valued spelling, the reverse direction |

**The `:343` row was mislabelled.** "Filter parent by child conditions"
names the reverse direction (a parent by its children's fields), while
the form it pointed at, even had it been served, expresses only the
forward one (rows by their lookup target's column). The row is now two:
"Filter rows by their lookup target's column" → query the target object,
then `{ lookup: { $in: ids } }` (`$contains` per id when `multiple`);
"Filter parent by child conditions" → query the child with `fields:
[lookup]`, then `{ id: { $in: those ids } }` on the parent. Both routes
are served: the engine's own `expand` batch-loads with `{ id: { $in } }`
(`engine.ts`, the "Batch-load related records using $in query" block),
and the REST ingress admits `id` as a filter key (`protocol.ts`
`resolveQueryFields`, `known.add('id')`).

**No dotted alternative is taught.** `'owner.region'` is refused one
door earlier by the objectstack-ai#8371 dotted verdict (`filter-comparand-shape.ts`,
`INVALID_FIELD` / 400); a grep of the skill for a dotted `where` path
finds none.

**Census.** `grep -rn -i -E 'nested relation|relation filter|nested
relations' skills/` on `96e72447` finds exactly the six sites; a
multi-line shape grep for a `where` example nesting a no-operator object
under a relation key finds the same three (`SKILL.md:193`,
`rules/filters.md:138`, `:145`) and nothing else in `skills/**`.

## Paying for it inside `rules/filters.md`

The file sat at 2148 / 2149 tokens. The section rewrite (881 bytes) is
paid for by deleting three examples whose rule already has a home in the
same package, so the file lands at 2149 / 2149 (headroom 0, ceiling
untouched):

| Deleted from `rules/filters.md` | Home |
|:--|:--|
| `## Implicit Equality (Shorthand)` (`:40-:51`) | `SKILL.md` "Implicit
Equality (Shorthand)" (`:94-:101`), and the `$eq` row of this file's
Operator Reference |
| `### NOT` (`:93-:100`) | `SKILL.md` Logical Operators, `{ where: {
$not: { status: 'closed' } } }` (`:183-:184`) |
| `### Combining Logical Operators` (`:102-:113`) | `SKILL.md` "AND + OR
combined" (`:173-:181`); sibling-keys-are-AND is this file's first
Common Mistake |

The `role: 'admin'` literal in the deleted third example moved
`check:role-word`'s count for the file 8 → 7; the gate prescribes the
ratchet-down ("run `--update` and commit the baseline"), and commit
`b893787b` is that `--update` output: one row of
`scripts/role-word-baseline.json`. That file is outside the claim's
declared surface; declared here and in the report.

## `skills/**` readings (lines and tokens; tokens are the ratchet's
`ceil(utf8 bytes / 4)`)

| | Before (`96e72447`) | After (`b893787b`) | Δ |
|:--|--:|--:|--:|
| `SKILL.md` | 401 lines · 3990 tokens (ceiling 5552) | 399 lines · 4055
tokens | −2 lines · +65 tokens |
| `rules/filters.md` | 251 lines · 2148 tokens (ceiling 2149) | 214
lines · 2149 tokens | −37 lines · +1 token |
| Whole package `skills/objectstack-query` (6 files) | 1184 lines ·
10461 tokens | 1145 lines · 10527 tokens | −39 lines · +66 tokens |

Line budget (PM-set, net 0 at most): −39. No untouched line was
re-wrapped; no ceiling row moved.

## Verification

Gates, all at head `b893787b`, exit codes captured by redirect and
verdict lines quoted from each log: the 31 families `dispatch-gates
--commands --repo objectstack-ai/objectstack` derives from the change
set, reconciled with `--ran` ("31 derived, 31 run, 0 NOT-MEASURED, 0
UNRUN") — every one exit 0, including `check-skills-token-ratchet` ("54
authored bundle file(s) within their ceilings") and its `--self-test`
(65 cases), `check:role-word` after the ratchet-down ("OK, no new
occurrences of the reserved word"), `check:corpus-claim-drift`,
`check:skill-identifier-liveness`, `check:doc-authoring`,
`check:nul-bytes`, `check:doc-formula-expressions`
(`@objectstack/formula` and `@objectstack/lint` built first), spec
`check:skill-docs` ("Skill docs in sync") and spec `check:skill-refs`
("9 generated files in sync"). `check:skill-examples` does not apply: no
block in this skill carries an `os:check` marker.

Pin: the skill's example validation does not cover these snippets (no
`os:check` marker), so the pin is PR objectstack-ai#20781's, on `main`:
`packages/objectql/src/engine-nested-object-door.test.ts` (the refusal
envelope `{ code: 'INVALID_FILTER', status: 400 }` beneath `lookup`,
`master_detail`, `multiple: true` lookup, `user` and `tree`, and the
CONTROL that `{ owner: { $in: [...] } }`, `{ owners: { $contains: ... }
}`, its `$or`, and `{ id: { $in: [...] } }` reach the driver as written)
and `packages/rest/src/data-nested-object-door.test.ts` (400 over `POST
/api/v1/data/:object/query`; the routes answer `d1`, `d3`). Run
first-hand here: `pnpm --filter @objectstack/objectql exec vitest run
--maxWorkers=2 src/engine-nested-object-door.test.ts` → "Test Files 1
passed (1), Tests 15 passed (15)". No package is touched, so no package
build or test suite is owed beyond that.

## Acceptance notes

- `content/docs/protocol/objectql/query-syntax.mdx` (`:602-:610`,
read-only here): the "Filtering Across Relationships" callout's headline
is true (neither the nested form nor a dotted path is served), but its
mechanism paragraph is stale — it says `SqlDriver.applyFilters()`
compiles the nested object as a single-column comparison and emits the
dotted key verbatim to Knex, while on `main` both are refused at the
engine before any driver (`INVALID_FILTER` / 400 for the nested form,
`INVALID_FIELD` / 400 for the dotted path), and it names no served
route. Noted, not filed: documentation drift under a true headline; no
carrier known.
- `check:pm-governed-merges` as the package script spells it is the
`--self-test` alone (exit 0 here); the live sweep is CI's.
- The report's `api_writes` lists every relay write of this run.

## 维护者速读(草稿)

**改了什么。** 发布的查询技能包 `skills/objectstack-query` 原先把「在关联字段下直接写条件」(`{
customer: { country: 'US' }
}`)当作能用的过滤写法来教。现在六处教它或指向它的句子都改成平台今天的真实行为:这种写法被引擎拒绝(`INVALID_FILTER` /
400),可行的路是先查关联对象拿到 id,再对关联字段用 `$in`(多值关联用 `$contains` 逐个 id)。规则文件里删掉了三段在
SKILL.md 已有同样规则的重复示例,用来支付这段改写;整个包净减 39 行,每个文件的 token 上限都没动。

**为什么改。** AI 作者照着技能包写,写出来的查询在内存驱动上静默返回 0 行、在 SQL 驱动上报 400;PR objectstack-ai#20781
合并后所有驱动都统一拒绝。技能包是客户项目里 AI 的教材,教错一句就是每个客户项目里的错误查询。本 PR 不预判 v18
决策卡(objectstack-ai#20802)的任一方向,只陈述今天的行为。

**风险与代价(含回滚)。** 只改文档文字与一行门禁基线(`role-word` 计数 8 →
7,门禁自己要求的下调),不改任何代码或发布包。风险在措辞:若维护者裁定 v18 支持关联过滤,这几句还要再改一次(决策卡已注明)。回滚即
revert 本 PR 的两个提交。

**席位意见。**

**你要做的。** 复核六处措辞与三处删除各有归宿;批准后由席位落地(Tier H)。

---
_Generated by [Claude
Code](https://claude.ai/code/session_01KTZmMfzVzjNvyaLyQ8mHvg)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
akarma-synetal pushed a commit to akarma-synetal/framework that referenced this pull request Oct 7, 2026
…t the engine seam, the related object read as the caller, a loud cap, drivers untouched (objectstack-ai#20802) (objectstack-ai#20872)

Part of objectstack-ai#20802
Clause-②: yes (widening)

The engine half of ruling 5907789183 (letter A). The analytics cube read
and the analytics read-scope face are the second half, after objectstack-ai#5930 step
3 (objectstack-ai#20810); `skills/objectstack-query` (objectstack-ai#20782) belongs to the skills
seat, who is told after this lands. So this PR does not complete the
card.

## What this does

**The nested-relation form `{ relation: { field: value } }` is served in
`where`.** It is lowered at the engine's filter seam that objectstack-ai#5930 step 2
built (`cfa931535`). The drivers receive `$in` / `$contains` and are not
changed (ADR-0053 D-D1 item 5, D4 (b)).

- **Where the lowering runs.** Stage 2 of `where` admission now has
three steps: resolve the placeholders, then lower each nested-relation
condition, then run the shared `lowerFilterCondition`.
`ObjectQL.resolveRelateThenLowerWhere` holds that order, so a verb
cannot resolve without it. `resolveWhereTokens` and `withResolvedWhere`
became async to carry it.
- **How the lowering works.** `lowerRelationConditions` reads the
related object with the engine's own `find`: `fields: ['id']`, `limit:
RELATION_FILTER_ID_CAP + 1`, and the caller's execution context. The ids
it returns become `{ relation: { $in: ids } }` on a single-valued
relation. On a multi-valued one they become an `$or` of one `$contains`
per id, which matches on any member. That is the spec's own any-of
spelling, and the SQL family refuses `$in` over the JSON column. No
related record matching gives `$in: []` or `$or: []`: FALSE, never an
absent predicate.
- **No second walk.** A condition is found by the one filter walk the
engine already runs with each column's declaration in hand,
`walkCondition` in `number-comparand-declared-type-door.ts`.
- At the door (stage 1), that walk admits a condition structurally, from
declarations alone (`admitRelationCondition`), instead of refusing it.
- The lowering calls the same walk (`mapRelationConditions`) twice: once
to collect the conditions, and once, after the reads, to replace them in
the same order.
- The door and the lowering therefore find a condition at the same
boundaries by construction. No driver-local guard exists.
- **As the caller.** The related read goes through the middleware chain
like any read. The related object's CRUD gate, row scope and field
permissions apply, and its own doors judge the condition's comparands. A
refusal from any of them is the answer, loudly.
- **Bounded.** The cap is `RELATION_FILTER_ID_CAP` = 1000, one named
constant, exported from `@objectstack/objectql`. Past it the filter is
refused with `INVALID_FILTER` / 400. The refusal names the cap, the
related object and the two-step route. The filter is never run over a
cut-off list.

**The accept set that widens.** It is the engine's `where` (`find`,
`findOne`, `count`, `aggregate`, `update`, `delete`) plus `judgeFilter`,
and the REST query doors that reach `findData` (`POST
/api/v1/data/:object/query` and the `filter` / `$filter` spellings). Per
relation kind:

- `lookup`, `master_detail`, `user`, `tree`, single-valued: refused, now
served (`$in`).
- the same kinds with `multiple: true`: refused, now served (any
member).

**Nothing served today narrows.**

- The admission runs only where the arm refused before: a no-operator
object beneath a relation column at `where`.
- The cap refusal is new, but it applies to a form that was refused.
- Every filter without a nested-relation condition reaches the drivers
by reference, byte-identical to before (pinned).

**What stays refused.** Each is refused in the engine's words, before
any read:

- a `json` field's object comparand, and the provisioned `id`
(unchanged);
- a second level (a relation condition beneath the related object's own
relation field);
- a dotted key inside the condition;
- a key the related object does not declare;
- `{}`;
- a related object that is not registered;
- the dotted path `{ 'owner.region': 'NA' }`, still `INVALID_FIELD`
(objectstack-ai#8371);
- the form in an aggregation's own `filter` and in `having`. The engine
evaluates both itself, and its evaluator has no member test for a stored
list. Their words now say `where` serves it.

**Text.**

- `FilterCondition`'s docblock item 4 now states the served semantics.
- The `QueryFilter` example shows the form again.
- The `data-engine.mdx` example that PR objectstack-ai#20781 removed comes back, with
the cut, the cap and the two-step route.

**The dotted-path words, made true again (a bounded in-place fix, named
here).**

- Both dotted relation refusals said "a filter reaches only columns of
'…' itself". This change makes that false.
- The two refusals are the engine's (`filter-comparand-shape.ts`) and
the query-parameter door's (`metadata-protocol` `protocol.ts`, outside
the claim's declared file surface).
- Both now name the nested form to write instead (`{ "owner": {
"region": VALUE } }`), in the same words, and keep the shared
denormalise remedy.
- A conformance pin holds the two routes equal.

## Measured

On this branch at `56da9b6d50` through `POST
/api/v1/data/:object/query`. Owner `u1` is region NA on `d1` and `d3`,
and `d4` has no owner. Before, on `origin/main` after PR objectstack-ai#20781, every
relation row answered `INVALID_FILTER` / 400 on every driver.

| `where` | SQLite | PostgreSQL 16.13 (live, local) | InMemoryDriver
(measured, not pinned) |
|:--|:--|:--|:--|
| `{ owner: { region: 'NA' } }` (lookup), `boss` (master_detail) | `d1`,
`d3` | `d1`, `d3` | `d1`, `d3` |
| `{ owners: { region: 'NA' } }` (multiple lookup) | `d1`, `d3` | `d1`,
`d3` | `d1`, `d3` (see the note) |
| `{ parent: { title: 'a' } }` (tree) | `d2`, `d3` | `d2`, `d3` | `d2`,
`d3` |
| `{ $not: { owner: { region: 'NA' } } }` | `d2`, `d4` | `d2`, `d4` |
`d2`, `d4` |
| `{ $or: [{ owner: { region: 'EU' } }, { title: 'a' }] }` | `d1`, `d2`
| `d1`, `d2` | `d1`, `d2` |
| `{ owner: { region: 'APAC' } }` (no match) | none | none | none |
| a condition matching 1001 related records | 400 `INVALID_FILTER`, the
cap words | same | not measured |

- **The memory note.** The in-memory driver matches `$contains` over a
stored array by substring per element. That is the gap
`FILTER_OPERATORS`' `$contains` docblock records for that driver. So
with ids `u1` and `u10`, a multi-valued condition meaning `u1` also
matches the row holding `['u10']`: measured memory `d1`, `d3`, `d5`,
against SQL `d1`, `d3`. Single-valued relations are exact everywhere.
- **Why memory is not pinned.** `check:driver-memory-census` refuses a
new test consumer of that driver without a ruling.
- **H7 (RLS), measured.** A policy `record.owner.region == 'NA'` is
refused at compile: "cross-object/nested field path … is not
pushdown-able". The policy is dropped to the deny sentinel, so it
answers zero rows. The RLS compile seam is untouched, and no async read
was added there.

## Mechanism hypotheses: which held

- **H1: held, refined.** `lowerFilterCondition` is pure and synchronous,
and the relation step needs the engine. So the step lives in objectql,
between token resolution and the shared lowering, and each engine filter
position reaches it at most once.
- Served: `where` on `find`, `findOne`, `count`, `aggregate`, `update`
and `delete` (the multi and by-predicate paths alike), and the judge.
  - Not served (refused, named): `aggregations[i].filter` and `having`.
- **H2: held, with one widening of the kind list.** The door admits the
form under every `REFERENCE_VALUE_TYPES` kind, not only `lookup` /
`master_detail`.
- Reason: `user` and `tree` point at a related object the same way, and
the objectstack-ai#20745 seat answer ruled that one class gets one answer.
- `user` needs `sys_user` registered; where it is not, it is refused
loudly ("no object 'sys_user' is registered here").
  - Every refusal the order listed is kept, as listed above.
- **H3: measured "yes": the one check exists and is reused.**
- A direct filter on a field the caller cannot read is refused today:
`403 PERMISSION_DENIED`, the security layer's filter-oracle guard
`assertReadableQueryFields`.
- The related read reaches that same check, so the nested form answers
the same 403, naming the field.
- There is no second copy of the rule. ⚠️ This is not the ruling's
literal `INVALID_FILTER`; see the open question in the report.
- **H4: `$in` does not mean "any member" everywhere; `$or` of
`$contains` does on SQL.**
- `$in` over a multi-valued lookup is refused on SQL (JSON column) and
is any-member on memory.
- `$contains` is membership on SQLite and PostgreSQL, and
substring-per-element on memory (the note above).
- The `$or` of `$contains` is the spec's declared any-of spelling, so it
is the lowered form.
- **H5: held, and no bound to reuse.** `expand`'s batch loader bounds
nothing: it deliberately forwards no limit. The cap is a new named
constant.
- **H6: held.** `$and` / `$or` compose as written. `$not` over a
relation condition takes the shared lowering's NULL-safe negation, so a
row with no relation satisfies it. The driver input is pinned equal to
the hand-written two-step route's, and `$not` + no match gives every
row. An empty inner result is FALSE, never "no filter". `$nor` is not in
the vocabulary.
- **H7: held; out of scope.** Measured above.

## Tests (all on `56da9b6d50`)

- `@objectstack/objectql` test: 345 files / 6786 passed. typecheck exit
0, `check:test-typecheck` OK.
- `@objectstack/rest` test, with `OS_TEST_POSTGRES_URL` set to a local
PostgreSQL 16.13: 237 files / 4706 passed / 35 skipped (MySQL cells and
suites with no URL). typecheck exit 0.
- `@objectstack/metadata-protocol` test: 191 files passed, 3 skipped /
2801 passed, 19 skipped. typecheck exit 0.
- `@objectstack/spec` test: 578 files / 17066 passed / 1 todo. typecheck
exit 0. `check:generated`: all 15 artifacts up to date (no regeneration
needed; docblock only).
- Downstream: `@objectstack/plugin-security` 149 files / 3227 passed, 23
skipped. `driver-memory` 65 / 1470 passed. `driver-sql` 201 files
passed, 11 skipped / 3254 passed, 188 skipped. The other
`...@objectstack/objectql` consumers are declared to CI.
- New pins:
- `packages/objectql/src/engine-nested-relation-lowering.test.ts` (13
tests, recording driver). It covers:
    - every relation type;
    - any-member on a multi-valued relation;
    - the empty id set;
- `$and` / `$or` / `$not` / sugar, pinned equal to the two-step route's
driver input;
    - every verb and the judge;
    - placeholder resolution;
- the related read as the caller, and a middleware refusal surfacing
with no outer read;
    - the cap at 1000 and at 1001;
- the kept refusals, with the judge answering execution's words
verbatim;
    - the related object's own doors;
    - the aggregation `filter` / `having` refusals;
    - a pass-through control.
- `packages/rest/src/data-nested-object-door.test.ts` (rewritten):
objectstack-ai#20745's table turned into rows on SQLite and live PostgreSQL, plus the
two-step equivalence, the kept refusals with the route inside the
500-character REST bound, the cap pin (1001 related records refused,
1000 served) and the controls.
- `packages/rest/src/data-nested-relation-permission.test.ts`: the
permission pin, with the real `SecurityPlugin` on a real engine and
SQLite, through the REST door. An unreadable related field is refused
403, never emptied, while a system read can filter by it. A hidden
related record matches nothing.
- Fixture triage for the removed refusal branch:
  - `engine-nested-object-door.test.ts` keeps only what still refuses.
- `query-expression-conformance.test.ts`: its nested-form control now
pins the served rows, and a new pin checks that both doors' dotted
refusals name the same route.
- `protocol-explicit-filter-field-gate.test.ts`: its GUARD still proves
the name gate never descends.

**Ablations, each from the committed fix.** Each ran through
`scripts/ablation-replace.mjs` in WRAP mode, trap-restored. After each
mutation objectql was rebuilt, and `ablation-dist-preflight` found the
marker in 4 built files.

- **A, the lowering disabled.** `if (sites.length === 0) return where;`
became an unconditional `return where` (marker
`__ablated_20802_lowering__`). Blob `9237c3dfc995` → `f9994599356f`.
Predicted red, observed red:
  - objectql pins: 10 failed / 235 passed;
  - rest pins: 11 failed / 4 passed / 6 skipped.
  - The structural refusals and controls stayed green.
- **B, the related read as the system.** `...(execCtx ? { context:
execCtx } : {}),` became `isSystem: true` (marker
`__ablated_20802_caller__`). Blob → `b0c1748c5b70`. Predicted red,
observed red:
  - objectql 1 failed / 244 passed (the as-the-caller pin);
- rest 2 failed / 13 passed / 6 skipped: the permission pin answered
rows instead of 403, and the row-scope pin returned `d4`.
- (A first run of B used a mutation that left `execCtx` unused, and its
DTS step failed on TS6133; the JS carried the marker. It was re-run
type-clean, and those numbers are the ones above.)
- **Restore.** Blob equals HEAD `9237c3dfc995`, and `git diff HEAD` is
empty. After a rebuild, the `--absent` preflight found both markers
absent from all 14 built files, and the whole tree was clean. The pins
were green again: 245 passed; 15 passed / 6 skipped.

## Gates

`node scripts/pm/dispatch-gates.mjs --commands` at `56da9b6d50` (fresh,
not stale) derived 120 commands, and all 120 were run. `--ran` with each
exit code recorded: **120 derived, 120 run, 0 NOT-MEASURED, 0 UNRUN**,
all exit 0.

Three gates refused first with `PREREQUISITE NOT MET` (exit 3), and none
of the three is counted as a failure:

- `check:skill-examples`
- `check:dual-build-cjs-loads`
- `check:type-check-debt`

They were re-run green after `turbo run build --filter='./packages/*'
--filter='./packages/*/*'`.

Lint, narrowed and proven: `eslint --no-inline-config --format json`
over the 15 changed `.ts` files gave **15 files, 0 errors, 0 warnings**.

- `isPathIgnored` is false for all 15.
- The population is `eslint.config.mjs`'s `**/*.{ts,…}` block.
- `parserOptions.project` / `projectService` are unset for every file.
Type-aware linting is off, so this diff cannot move a verdict on an
untouched file.

## Changesets

- `.changeset/20802-nested-relation-filter-served.md`:
`@objectstack/objectql` `minor`, `Clause-②: yes (widening)`. It says it
supersedes the relation-field paragraph of the pending
`20745-nested-object-door` entry, and it states the in-memory
`$contains` substring caveat.
- `.changeset/20802-nested-relation-prose.md`: `@objectstack/spec`
`patch` (shipped JSDoc).
- `.changeset/20802-dotted-relation-route.md`:
`@objectstack/metadata-protocol` `patch` (refusal words).
- ADR anchor:
`scripts/adr-anchors/packages__objectql__src__relation-filter-lowering.ts.json`
→ ADR-0053.

## Acceptance notes

- **The analytics half.** The cube read and the analytics read-scope
face are not touched here (objectstack-ai#20810 first). Until then the analytics face
still flattens the nested form to cube members, and the read scope still
refuses it.
- **The permission refusal's envelope.** It is `PERMISSION_DENIED` /
403, the one existing check, reused. It is not the ruling's
parenthetical `INVALID_FILTER`, and it is raised to the PM as an open
question.
- **Read order.** The related read runs at stage 2, before the outer
verb's own middleware. A caller with no read access to the queried
object still gets the outer 403, but the related read has already run as
that caller. It reads nothing the caller could not read directly.
- **Lint face.** `@objectstack/lint`'s list-view dotted-path hint still
says "Filter on a column of … itself". That is an instruction rather
than a claim this change made false. Carrier: none; not changed here.
- **The in-memory substring gap.** On the in-memory driver, the
multi-valued any-member lowering inherits that driver's
substring-per-element `$contains`. It is reported, not fixed here (no
driver file).

---
_Generated by [Claude
Code](https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation protocol:data size/xl tests tooling

Projects

None yet

2 participants