Skip to content

Commit 157baa7

Browse files
fix(objectql)!: a groupBy on a structured-JSON field is refused INVALID_FIELD / 400 at the engine aggregate door, on every driver (#20783) (#20804)
Fixes #20783 Clause-②: no (narrowing) ## What this changes A `groupBy` entry that names a declared **structured-JSON** field (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`) is now refused by `engine.aggregate` with `INVALID_FIELD` / 400, in the engine's words, before any driver is asked. It holds on every driver and for every caller that reaches the engine: the REST query door, a flow or hook calling the engine in-process, and the analytics strategy that lowers a cube query onto `engine.aggregate`. Both entry spellings are judged: the field name (`groupBy[0]`) and the `{ field }` object (`groupBy[0].field`), a `dateGranularity` bucket included. The words, as `POST /api/v1/data/:object/query` returns them (the route is inside the 500 characters the REST door keeps, and the REST pin asserts it there): ```text aggregate('rest_group_by_json_20783'): groupBy[0] names 'meta', a declared json field — a structured-JSON value, which the engine does not group by. The query was NOT run. Group by a field that stores one scalar value: store the part you group on in a field of its own and group by that field. A JSON document is no group key the drivers share: one merged every row into a single group, one grouped each serialized document apart, one refused the statement. ``` The thrown error carries `code: 'INVALID_FIELD'`, `status` and `httpStatus` 400, `field`, `fields` (every offending entry), `object` and `param: 'groupBy'`. **Landing site, as the order expected: `packages/objectql`'s aggregate admission. No driver file, no serialization rule.** - `packages/objectql/src/group-by-structured-json-door.ts` (new): `assertGroupByNamesNoStructuredJsonField(object, schema, groupBy)`. The class is the spec's `STRUCTURED_JSON_TYPES` (the set PR #20781's JSON arm judges), never a list minted here. Not judged: an undeclared name (the engine's registry-less tolerance; the REST ingress answers an unknown name `INVALID_FIELD` first), a host with no field map, and every other type. - `packages/objectql/src/engine.ts`: one call at the entry of `aggregate`, right after `rejectCredentialAggregation` (which reads the same `groupBy` entries), so a protected field keeps that refusal's words. `aggregate` is the only engine verb that takes `groupBy`: `find` refuses the key (`ENGINE_FIND_OPTION_KEYS`). - `packages/objectql/src/number-comparand-declared-type-door.ts`: comment only. Its sentence "a `json` … groupBy … is judged now" at `having` became false for a json groupBy, which no longer reaches `having`. **Why `INVALID_FIELD`, an existing code.** The verdict is about the named field's type at a position. That is the question the REST ingress answers with `INVALID_FIELD` for an unknown `groupBy` name (`assertGroupByFieldsExist`), the search axis answers with `INVALID_FIELD` for a field whose type it cannot scan, and the engine's `assertFilterIsMaterializable` answers with `INVALID_FIELD` for a virtual field ("this verdict is about the NAME's type"). `INVALID_FILTER` is the engine's value-shape envelope and `groupBy` is not a filter; `INVALID_QUERY` is the ingress's malformed-shape code, and the entry here is well formed. ## Before, measured on `origin/main` `7a09eee1b1` Through `POST /api/v1/data/:object/query` (the real `RestServer` route over `ObjectStackProtocolImplementation` and `ObjectQL`) with `{ groupBy: [FIELD], aggregations: [{ function: 'count', alias: 'n' }] }`, and through `engine.aggregate` directly (same answers). Drivers: InMemoryDriver, SqlDriver on SQLite (better-sqlite3), and SqlDriver on a private PostgreSQL 16.13 started for this run. Three rows: `title` x, x, y; `meta` `{a:1}`, `{a:2}`, `{b:1}`; and one differing value per row under every other structured-JSON field. | `groupBy` | InMemoryDriver | SQLite | PostgreSQL 16 | |:--|:--|:--|:--| | `title` (text, the control) | 200, `x` 2 · `y` 1 | same | same | | `meta` (json, the card) | 200, **one group** `{a:1}`, `n` 3 | 200, one group per serialized document (3) | **500 `DATABASE_ERROR`** ("could not identify an equality operator for type json") | | `composite`, `repeater`, `record`, `location`, `address` | 200, one group, `n` 3 | 200, one group per serialized document | 500 | | `vector` | 200, one group per array (`[1,2]` 2 · `[3,4]` 1) | 200, one group per serialized array | 500 | | `{ field: 'meta' }` | 200, one group, `n` 3 | 200, 3 groups | 500 | | `{ field: 'meta', dateGranularity: 'month' }` | 200, one `null` bucket, `n` 3 | 200, one `null` bucket, `n` 3 | 500 ("cannot cast type json to timestamp with time zone") | | `['title', 'meta']` | 200, 2 groups | 200, 3 groups | 500 | ## After, the same run on this branch (`43ae6c1fcc`) Every structured-JSON row above answers `400 INVALID_FIELD` in the engine's words on all three drivers, naming the position (`groupBy[0]`, `groupBy[0].field`, `groupBy[1]` for the mixed entry), the field and its declared type. No read of the object runs. The `title` control answers `x` 2 · `y` 1 on all three, unchanged. The InMemoryDriver cells of this run came from a scratch script over the built packages. They are not committed: `check:driver-memory-census` refuses a new test consumer of that driver without a ruling, so the committed memory cell is the recording driver below, by construction. ## Hypotheses (zone 2): which held - **H1: held.** `aggregate` resolves `groupBy` entries against the declared field map before the driver in `rejectCredentialAggregation` (credential and `internal` fields) and, for `having`, in `aggregatedRowColumnTypes`. The refusal sits beside the first, at the verb's entry, so it runs before the per-aggregation `filter` and `having` doors and before any driver is resolved. Code: `INVALID_FIELD`, reasons above. The REST ingress also resolves `groupBy` names (`assertGroupByFieldsExist` in `metadata-protocol`), but only for the REST path. The engine is the one door every caller shares, as triage directed. - **H2: held, refined.** Every member of `STRUCTURED_JSON_TYPES` answers per driver today, and none answers one way. Six members split exactly like `json` (memory one merged group, SQLite per serialized document, PostgreSQL 500). `vector` splits differently on memory (one group per array, not one merged group) and still 500 on PostgreSQL, so it does not answer one way either. So the class is refused, not `json` alone. - **H3: held, and it was NOT already refused.** A date-bucketed `{ field, dateGranularity }` over a `json` field passed the REST ingress (a known field, a valid granularity) and answered one `null` bucket on memory and SQLite and 500 on PostgreSQL. It is refused now with the `{ field }` form's words, since no granularity makes a JSON document a date. - **H4: measured. The analytics face is partly the same door.** Measured through `AnalyticsService.query` wired with `AnalyticsServicePlugin`'s own auto-bridges (`executeAggregate` to `engine.aggregate`, `executeRawSql` to `engine.execute`), with a cube dimension `sql: 'meta'` on a `json` field: | cube query | InMemoryDriver | SQLite | PostgreSQL 16 | |:--|:--|:--|:--| | `dimensions: [meta]`, before | 200, one merged group (raw SQL unsupported, fell back to `engine.aggregate`) | 200, one group per serialized document (**native SQL**, engine not reached) | 500 `DATABASE_ERROR` (**native SQL**) | | `dimensions: [meta]`, after | **400 `INVALID_FIELD`** (this door) | unchanged | unchanged | | `timeDimensions: [{ meta, granularity: month }]`, before | 200, one `null` bucket | 200, one `null` bucket | 500 | | the same, after | **400 `INVALID_FIELD`** (the native strategy declines a granularity, so `engine.aggregate` serves it) | **400** | **400** | So the ObjectQL strategy (any cube query on a driver without raw SQL, and any bucketed time dimension) reaches this door and is folded in with no `packages/services` edit. **`NativeSQLStrategy` does not**: on SQL drivers it compiles `GROUP BY "meta"` by hand and bypasses the engine. That half is a sibling card (reported to the PM below, not filed here). The dataset compiler only checks that a dimension's field is declared (`assertDeclared`), so a dataset dimension on a `json` field compiles into the same cube dimension and splits the same way. The memory cube face (`MemoryAnalyticsService`) has **zero** production constructors (`git grep "new MemoryAnalyticsService"` outside tests: 0 hits at `7a09eee1b1`), so it reaches a driver only in tests. It was not edited and not measured for this shape. ## Producers measured (for "refuse, don't define") Zero producers group by a structured-JSON field. Every `grouping` / `groupBy` / `groupByField` / `dimensions` target in `examples/` at `7a09eee1b1` is one of `status`, `priority`, `created_at`, `account`, `stage`, `total`, `region`, `sales_region`, `progress`, `issued_on`, `industry`, `category`, `signed_on`, `health`, `completed_date`, `close_date`, and none is structured-JSON (the showcase's structured-JSON fields are `field-zoo`'s `f_*`, `account.hq`, `account.support_config` and `task.location`). The same count over the platform packages names `object_name`, `user_id`, `id`, `action`, `topic`, `provider_id`, `organization_id`, `namespace`, `kind`, `actor_id` and `phone`, plus two dynamic engine callers in `service-analytics` (see the acceptance notes). ## Tests - New `packages/objectql/src/engine-group-by-json-door.test.ts` (6 tests, recording driver, so the in-memory cell by construction). It covers every structured-JSON type with the full envelope (`code`, `status`, `httpStatus`, `field`, `fields`, `object`, `param`) and the position, and asserts zero reads. It covers the `{ field }` form, a date bucket, an entry after a scalar one and two offenders, and the REST door into `findData`. Controls: `text`, `number`, a `multiple: true` select, an image field and an undeclared name all reach the driver, and a structured-JSON field as an aggregated column is unchanged. GUARDs: the judged types equal `STRUCTURED_JSON_TYPES` over every `FieldType`, and there is no verdict without a field map, for an undeclared name, or for an entry naming no field. - New `packages/rest/src/data-group-by-json-door.test.ts`: SQLite always, PostgreSQL / MySQL where `OS_TEST_POSTGRES_URL` / `OS_TEST_MYSQL_URL` are set. Every structured-JSON type answers 400 `INVALID_FIELD` with the route in the REST body and zero reads. The object and bucket forms and the mixed entry answer the same 400. The control `title` answers `x` 2 · `y` 1 from the driver. ⚠️ As with the sibling door suites, no CI job sets those URLs for `@objectstack/rest`, so the live cells run only locally. Local run with the private PostgreSQL 16.13: **6 passed (sqlite 3, live postgres 3) / 3 skipped (mysql, no URL)**. - Fixture triage in `engine-nested-object-door.test.ts` (PR #20781's pin): its `having` case over `groupBy: ['meta']` pinned the branch this PR closes, because a json groupBy never reaches `having` now. It is **replaced**, not respelled. The JSON arm at `having` is still reached through a `max` of a json field (`having: { top_meta: { a: 1 } }`), which answers the same `INVALID_FILTER` whole-value words. No other fixture groups by a structured-JSON field. Measured by a grep over every test that both groups and declares a structured-JSON type; the downstream suites below stayed green. - `pnpm --filter @objectstack/objectql test` at `43ae6c1fcc`: **342 files / 6739 passed**. - `pnpm --filter @objectstack/rest test` at `43ae6c1fcc` (with the live PostgreSQL URL set): **232 files / 4516 passed / 33 skipped**. - `typecheck` for `@objectstack/objectql` and `@objectstack/rest` at `43ae6c1fcc`: exit 0, including each `check:test-typecheck` (objectql's ledger held at 40 files / 234 errors / 65 signatures; rest 0), so both new test files compile. - Downstream consumers (`...@objectstack/objectql` direction), at `cafaf885d8`: `@objectstack/service-analytics` **141 files / 3267 passed**; `@objectstack/metadata-protocol` **190 files passed, 3 skipped / 2792 passed, 19 skipped**. The other consumers are declared to CI. **Reverse verification (ablation), from the committed fix at `1eacad5296`.** It ran through `scripts/ablation-replace.mjs` in WRAP mode, trap-restored. The engine's call `assertGroupByNamesNoStructuredJsonField(object, this._registry.getObject(object), query.groupBy);` was fed `(query as { __ablated_20783__?: unknown }).__ablated_20783__` (always undefined) instead of `query.groupBy`. On disk the anchor went 1 to 0 and the marker 0 to 1, with blob `67198fc4a8fc` to `837fc9257e37`. objectql was rebuilt, and `ablation-dist-preflight` found the marker in 4 built files. - Predicted direction: red. Observed: red. - objectql pins: **3 failed / 18 passed**. Every refusal case failed. The controls, both GUARDs and the whole `engine-nested-object-door.test.ts` stayed green. - rest pins: **4 failed / 2 passed / 3 skipped**. The refusal cases failed on SQLite and live PostgreSQL, and the `title` control stayed green. - Restore leg: blob equals HEAD (`67198fc4a8fc`), `git diff HEAD` is empty, and whole-tree `git status --porcelain` is empty. After a rebuild, the `--absent` preflight found the marker absent from all 14 built files, and the tree reading was clean. Then both pin sets were green again (21 passed; 6 passed + 3 skipped). ## Gates `node scripts/pm/dispatch-gates.mjs --commands` at `43ae6c1fcc` (the merge of `origin/main` `96e724475c`, which touches none of this diff's packages) derived **65** commands over the 7 changed paths. All 65 were run on `43ae6c1fcc`, and `--ran` reconciles them: **65 derived, 65 run, 0 NOT-MEASURED, 0 UNRUN**, all exit 0. Among them: - `check:adr-0087-registration --base origin/main`: `not-required (no-migration-prescription)` accepted. - `check:changeset-no-major`, `check:empty-changeset`, `check:doc-authoring`, `check:nul-bytes`, `check:issue-citations`. - `check:engine-double-contract`, `check:where-matcher`, `check:driver-memory-census`, `check:cross-package-test-inputs`, `check:test-source-alias`, `check:type-check-coverage`, `check:type-check-debt`. - `check:query-options-erasure`. It caught a first draft of the REST pin that erased an `engine.aggregate` options bag to `any` (test surface 236 to 237). Fixed by typing it (`cafaf885d8`), and the surface is back at 236. - `check:dual-build-cjs-loads`, after a full `turbo run build` of `./packages/*` and `./packages/*/*`. Lint, narrowed and proven: `pnpm exec eslint --no-inline-config --format json` over the 6 changed `.ts` files at `43ae6c1fcc` found **6 files, 0 errors, 0 warnings**. Three facts make this narrowing a measurement: - The checked population comes from eslint's own config: `isPathIgnored` answers `false` for all 6. - The file count comes from the JSON output: 6 results. - Untouched files cannot change verdict: `parserOptions.project` and `projectService` are `null` for every file, so type-aware linting is not enabled. ## Changeset `.changeset/20783-groupby-structured-json-refused.md`: `@objectstack/objectql` `minor`, a BREAKING banner, `Clause-②: no (narrowing)`, and exactly one ADR-0087 marker, `not-required (no-migration-prescription)`, in the form PR #20781's changeset uses. It carries no rewrite table and no arrow: the body states what an author sees now, why, who is affected and what is unchanged. No export or published type changes: the door module is internal, and `@objectstack/objectql`'s root and `./core` exports are unchanged. ## Acceptance notes - **Reported to the PM, not filed here (same family, measured):** - **The native-SQL analytics face** (triage's H4 sibling). A cube or dataset dimension on a structured-JSON field through `NativeSQLStrategy` answers one group per serialized document on SQLite and 500 on PostgreSQL. It never reaches this door. The fix lands in `packages/services/service-analytics`, which this claim does not touch. - **`groupBy` on a `multiple: true` select** through `POST /api/v1/data/:object/query`: memory answers one group per array, SQLite one per serialized array, and PostgreSQL 500 (json equality). Not this card's class, and a multi-value field has a plausible other meaning (a bucket per member), so it is not refused here. - **`count_distinct` over a `json` field** through the same door: memory 3, SQLite 3, PostgreSQL 500. `AGGREGATE_FIELD_TYPE_COMPATIBILITY` accepts `count_distinct` for every `FieldType` on the ground that every backend gives one answer. PostgreSQL does not for json. - `service-analytics` has two dynamic `engine.aggregate` callers: `resolveFkAttr` (`groupBy: ['id', attr]`, a cross-object dimension attribute) and the display-label pass (`groupBy: ['id', displayField]`). If that attribute or display field is structured-JSON, they now answer this 400. Before, by reading and not measured: memory and SQLite grouped under `id` first, so one row per record, and PostgreSQL would have answered the same json-equality 500. No example or platform dataset names such a dimension (the census above). - A refusal relayed through the analytics engine path names the engine's position (`groupBy[0]`), not the cube member the caller wrote (`c20783.meta`). Carrier: the native-SQL sibling card, which touches the same face. - The native-SQL analytics path returned PostgreSQL's `count` as a string (`"2"`) where SQLite returned a number, in the stand-in wiring above. Observed only; not measured through the HTTP door. --- _Generated by [Claude Code](https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4cc5bcd commit 157baa7

7 files changed

Lines changed: 574 additions & 2 deletions
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
"@objectstack/objectql": minor
3+
---
4+
5+
fix(objectql)!: a `groupBy` on a structured-JSON field is refused with `INVALID_FIELD` / 400 at the engine's `aggregate`, on every driver
6+
7+
Clause-②: no (narrowing)
8+
9+
<!-- adr-0087: not-required (no-migration-prescription) a refusal of a grouping TARGET at the engine's aggregate door: a groupBy entry naming a declared json, composite, repeater, record, location, address or vector field. No authorable key, spelling, export or stored shape moves (the door module is internal; `@objectstack/objectql` exports nothing new and nothing less, and `EngineAggregateOptions` / `QuerySchema.groupBy` keep parsing the entry), and no stored row is read or rewritten. The grouping had no shared meaning to preserve (one merged group on the in-memory driver, one group per serialized document on SQLite, a 500 on PostgreSQL), and which scalar part of the document a caller meant to group on is not something a ledger entry can rewrite. The other categories are closed on facts: the package publishes (not `unpublished`); no ADR-0087 id covers a grouping target (not `registered` / `already-registered`); and the change is runtime behaviour, not a declaration (not `runtime-interface-only` / `type-surface-only`). -->
10+
11+
**BREAKING**: this narrows what `aggregate` accepts as a grouping target. A `groupBy` entry that names a declared field of the structured-JSON class (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`) is refused by the engine before any driver is asked. Both entry spellings are judged, the field name and the `{ field }` object, a `dateGranularity` bucket included. It ships as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes.
12+
13+
**What an author sees now.** `400 INVALID_FIELD`, naming the position (`groupBy[0]`, or `groupBy[0].field` for the object form), the field and its declared type, saying the query was not run, and naming the route inside the first 500 characters the REST door keeps: group by a field that stores one scalar value, storing the part of the document you group on in a field of its own. The thrown error carries `field`, `fields`, `object` and `param: 'groupBy'`.
14+
15+
**Why a refusal.** The drivers share no meaning for a JSON document as a group key. Measured through `POST /api/v1/data/:object/query` over three rows with different documents under the grouped field: the in-memory driver answered 200 with one group holding every row, SQLite answered 200 with one group per serialized document, and PostgreSQL answered 500 `DATABASE_ERROR`. A `vector` field split the same three ways, and a date bucket over a `json` field answered one `null` bucket on memory and SQLite and 500 on PostgreSQL. No producer that groups by a structured-JSON field was found (no dataset, cube, view grouping or `groupBy` in the example apps names one), so no meaning is defined for it here.
16+
17+
**Who is affected.** A caller of `engine.aggregate` or of the REST query door that grouped by such a field on the in-memory driver or on SQLite and read the merged or per-serialization groups as real ones. On PostgreSQL the same query was already a 500. The analytics service's aggregate path (a cube query the native-SQL strategy declines, such as a time dimension with a granularity, or any cube query on the in-memory driver) reaches the engine and answers this refusal too.
18+
19+
**Unchanged.** A `groupBy` on any other type (`text`, `number`, a `multiple: true` select, a file field), a structured-JSON field as an AGGREGATED column (`count`, `count_distinct`, `min`, `max`), and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached.
Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#20783] A `groupBy` entry naming a STRUCTURED-JSON field (`json`,
5+
* `composite`, `repeater`, `record`, `location`, `address`, `vector`) is
6+
* refused `INVALID_FIELD` / 400 by `engine.aggregate`, naming the field, its
7+
* declared type and the position, before any driver is asked
8+
* (`group-by-structured-json-door.ts`).
9+
*
10+
* Measured on the base (`origin/main` `7a09eee1b1`) through
11+
* `POST /api/v1/data/:object/query`, `{ groupBy: [FIELD], aggregations:
12+
* [{ function: 'count', alias: 'n' }] }` over three rows:
13+
*
14+
* | `groupBy` | InMemoryDriver | SqlDriver, SQLite | SqlDriver, PostgreSQL 16 |
15+
* |:--|:--|:--|:--|
16+
* | `text` (the control) | 200, one group per value | same | same |
17+
* | `json`, and its `composite`, `repeater`, `record`, `location`, `address` twins | 200, one group holding every row | 200, one group per serialized document | 500 `DATABASE_ERROR` |
18+
* | `vector` | 200, one group per array | 200, one group per serialized array | 500 |
19+
* | `{ field: 'meta', dateGranularity: 'month' }` (json) | 200, one `null` bucket | 200, one `null` bucket | 500 |
20+
*
21+
* The InMemoryDriver cell is this suite's recording driver by construction:
22+
* the door answers before a driver is resolved, so no read runs. The SQL cells
23+
* over a real driver live in `@objectstack/rest`'s
24+
* `data-group-by-json-door.test.ts`; that driver's test consumers are a ruled,
25+
* closed census (`check:driver-memory-census`), so no new suite of it here.
26+
*/
27+
28+
import { describe, it, expect, beforeEach } from 'vitest';
29+
import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol';
30+
import { FieldType, STRUCTURED_JSON_TYPES, type EngineAggregateOptions } from '@objectstack/spec/data';
31+
import { ObjectQL } from './engine.js';
32+
import { assertGroupByNamesNoStructuredJsonField } from './group-by-structured-json-door.js';
33+
34+
const OBJECT = 'group_by_json_probe';
35+
36+
/** field · declared type — one field of every structured-JSON type. */
37+
const JSONS: ReadonlyArray<readonly [string, string]> = [
38+
['meta', 'json'],
39+
['spec', 'composite'],
40+
['rep', 'repeater'],
41+
['rec', 'record'],
42+
['loc', 'location'],
43+
['ship_to', 'address'],
44+
['vec', 'vector'],
45+
];
46+
47+
const PROBE = {
48+
name: OBJECT,
49+
label: 'Group-by JSON probe',
50+
fields: {
51+
title: { name: 'title', type: 'text' },
52+
amount: { name: 'amount', type: 'number' },
53+
tags: { name: 'tags', type: 'select', multiple: true, options: [{ label: 'A', value: 'a' }] },
54+
photo: { name: 'photo', type: 'image' },
55+
...Object.fromEntries(JSONS.map(([name, type]) => [name, { name, type }])),
56+
},
57+
};
58+
59+
const COUNT = [{ function: 'count', alias: 'n' }] as EngineAggregateOptions['aggregations'];
60+
61+
interface SeenRead { ast: any }
62+
63+
/** Minimal recording driver — the same witness shape as the sibling door suites. */
64+
function makeRecordingDriver() {
65+
const rows = new Map<string, Record<string, unknown>>();
66+
const reads: SeenRead[] = [];
67+
const run = (_ast: any) => [...rows.values()];
68+
const driver: any = {
69+
name: 'recording', version: '0.0.0', supports: {},
70+
async connect() {}, async disconnect() {}, async checkHealth() { return true; }, async execute() { return null; },
71+
async find(_o: string, ast: any) { reads.push({ ast }); return run(ast); },
72+
async findOne(_o: string, ast: any) { reads.push({ ast }); return run(ast)[0] ?? null; },
73+
async count(_o: string, ast: any) { reads.push({ ast }); return run(ast).length; },
74+
async create(_o: string, data: Record<string, unknown>) {
75+
const id = (data.id as string) ?? `r_${rows.size + 1}`;
76+
const row = { ...data, id }; rows.set(id, row); return row;
77+
},
78+
async update(_o: string, id: string, data: Record<string, unknown>) {
79+
const up = { ...(rows.get(id) ?? {}), ...data, id }; rows.set(id, up); return up;
80+
},
81+
async delete(_o: string, id: string) { return rows.delete(id); },
82+
async beginTransaction() { return { commit: async () => {}, rollback: async () => {} }; },
83+
async commit() {}, async rollback() {},
84+
};
85+
return { driver, reads };
86+
}
87+
88+
type Thrown = (Error & {
89+
code?: string; status?: number; httpStatus?: number; field?: string; fields?: string[]; object?: string; param?: string;
90+
}) | null;
91+
92+
const refusalOf = async (p: Promise<unknown>): Promise<Thrown> => p.then(() => null, (e: any) => e);
93+
94+
const ENVELOPE = { code: 'INVALID_FIELD', status: 400, httpStatus: 400 };
95+
const envelopeOf = (err: Thrown) => ({ code: err?.code, status: err?.status, httpStatus: err?.httpStatus });
96+
97+
/** The door's own words, in every refusal it raises — a control must never be answered in them. */
98+
const DOOR_WORDS = 'which the engine does not group by';
99+
100+
describe('[#20783] a groupBy on a structured-JSON field, at the engine\'s aggregate door', () => {
101+
let engine: ObjectQL;
102+
let reads: SeenRead[];
103+
104+
beforeEach(async () => {
105+
const rec = makeRecordingDriver();
106+
reads = rec.reads;
107+
engine = new ObjectQL();
108+
engine.registerDriver(rec.driver, true);
109+
await engine.init();
110+
engine.registry.registerObject(PROBE as any, 'test');
111+
reads.length = 0;
112+
});
113+
114+
it('refuses a groupBy on every structured-JSON type with INVALID_FIELD / 400, naming the field, its type and the position — no read', async () => {
115+
for (const [field, type] of JSONS) {
116+
const err = await refusalOf(engine.aggregate(OBJECT, { groupBy: [field], aggregations: COUNT }));
117+
expect(envelopeOf(err), field).toEqual(ENVELOPE);
118+
expect({ field: err?.field, fields: err?.fields, object: err?.object, param: err?.param }, field)
119+
.toEqual({ field, fields: [field], object: OBJECT, param: 'groupBy' });
120+
expect(err!.message, field).toMatch(
121+
new RegExp(`^aggregate\\('${OBJECT}'\\): groupBy\\[0\\] names '${field}', a declared ${type} field `),
122+
);
123+
expect(err!.message, field).toContain('The query was NOT run.');
124+
}
125+
expect(reads, 'every refusal precedes the driver').toHaveLength(0);
126+
});
127+
128+
it('refuses the { field } object form, a date bucket over it, and an entry after a scalar one, at the entry\'s own position — no read', async () => {
129+
const cases: ReadonlyArray<readonly [EngineAggregateOptions['groupBy'], string, string[]]> = [
130+
[[{ field: 'meta' }], 'groupBy[0].field', ['meta']],
131+
[[{ field: 'meta', dateGranularity: 'month' }], 'groupBy[0].field', ['meta']],
132+
[['title', 'ship_to'], 'groupBy[1]', ['ship_to']],
133+
[['meta', 'title', { field: 'loc' }], 'groupBy[0]', ['meta', 'loc']],
134+
] as ReadonlyArray<readonly [EngineAggregateOptions['groupBy'], string, string[]]>;
135+
for (const [groupBy, position, fields] of cases) {
136+
const label = JSON.stringify(groupBy);
137+
const err = await refusalOf(engine.aggregate(OBJECT, { groupBy, aggregations: COUNT }));
138+
expect(envelopeOf(err), label).toEqual(ENVELOPE);
139+
expect(err?.fields, label).toEqual(fields);
140+
expect(err!.message, label).toContain(`): ${position} names '${fields[0]}',`);
141+
}
142+
expect(reads).toHaveLength(0);
143+
});
144+
145+
it('CONTROL a text, number, multi-value select, file or undeclared groupBy reaches the driver, never this refusal', async () => {
146+
for (const field of ['title', 'amount', 'tags', 'photo', 'not_declared']) {
147+
const before = reads.length;
148+
const out = await engine.aggregate(OBJECT, { groupBy: [field], aggregations: COUNT }).then(
149+
() => null,
150+
(e: Error) => e.message,
151+
);
152+
expect(out ?? '', field).not.toContain(DOOR_WORDS);
153+
expect(reads.length - before, `${field}: the driver was asked`).toBe(1);
154+
}
155+
// A structured-JSON field as an AGGREGATED column is not this door's.
156+
await expect(engine.aggregate(OBJECT, {
157+
groupBy: ['title'],
158+
aggregations: [{ function: 'count', field: 'meta', alias: 'n' }],
159+
} as EngineAggregateOptions)).resolves.toBeDefined();
160+
});
161+
162+
it('the REST door into findData answers the same refusal — no read', async () => {
163+
const protocol = new ObjectStackProtocolImplementation(engine);
164+
const err = await refusalOf(protocol.findData({
165+
object: OBJECT,
166+
query: { groupBy: ['meta'], aggregations: COUNT },
167+
} as any));
168+
expect(envelopeOf(err)).toEqual(ENVELOPE);
169+
expect(err!.message).toContain(`groupBy[0] names 'meta', a declared json field`);
170+
expect(reads).toHaveLength(0);
171+
});
172+
173+
it('GUARD the judged types are exactly the spec\'s STRUCTURED_JSON_TYPES, over every FieldType', () => {
174+
for (const type of FieldType.options) {
175+
const thrown = (() => {
176+
try {
177+
assertGroupByNamesNoStructuredJsonField(OBJECT, { fields: { f: { type } } }, ['f']);
178+
return null;
179+
} catch (e) {
180+
return e as Thrown;
181+
}
182+
})();
183+
expect(thrown === null ? null : envelopeOf(thrown), type)
184+
.toEqual(STRUCTURED_JSON_TYPES.has(type) ? ENVELOPE : null);
185+
}
186+
});
187+
188+
it('GUARD no verdict without a field map, for an undeclared name, or for an entry that names no field', () => {
189+
const judge = (schema: unknown, groupBy: unknown) => () => assertGroupByNamesNoStructuredJsonField(OBJECT, schema, groupBy);
190+
expect(judge(undefined, ['meta'])).not.toThrow();
191+
expect(judge({}, ['meta'])).not.toThrow();
192+
expect(judge(PROBE, ['nope', { field: 'nope' }, { dateGranularity: 'month' }, 7, null])).not.toThrow();
193+
expect(judge(PROBE, 'meta')).not.toThrow();
194+
expect(judge(PROBE, [])).not.toThrow();
195+
});
196+
});

‎packages/objectql/src/engine-nested-object-door.test.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -294,7 +294,9 @@ describe('[#20745] a no-operator object beneath a relation, structured-JSON or p
294294
const cases: ReadonlyArray<readonly [EngineAggregateOptions, string, string, string]> = [
295295
[{ groupBy: ['owner'], aggregations: [{ function: 'count', alias: 'n' }], having: { owner: { region: 'NA' } } }, 'owner', 'lookup', 'nested-relation form'],
296296
[{ groupBy: ['title'], aggregations: [{ function: 'max', field: 'boss', alias: 'top' }], having: { top: { region: 'NA' } } }, 'top', 'master_detail', 'nested-relation form'],
297-
[{ groupBy: ['meta'], aggregations: [{ function: 'count', alias: 'n' }], having: { meta: { a: 1 } } }, 'meta', 'json', 'whole-value match'],
297+
// [#20783] A JSON column reaches `having` as a `max` of a json field: a
298+
// json GROUPBY is refused one door earlier (`engine-group-by-json-door.test.ts`).
299+
[{ groupBy: ['title'], aggregations: [{ function: 'max', field: 'meta', alias: 'top_meta' }], having: { top_meta: { a: 1 } } }, 'top_meta', 'json', 'whole-value match'],
298300
] as ReadonlyArray<readonly [EngineAggregateOptions, string, string, string]>;
299301
for (const [query, column, type, words] of cases) {
300302
reads.length = 0;

‎packages/objectql/src/engine.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,7 @@ import {
200200
EmptyCredentialWriteError,
201201
SECRET_MASK,
202202
} from './secret-fields.js';
203+
import { assertGroupByNamesNoStructuredJsonField } from './group-by-structured-json-door.js';
203204
import { pluralToSingular, ExternalWriteForbiddenError } from '@objectstack/spec/shared';
204205
import { SchemaRegistry, computeFQN, type ArtifactInstallScope } from './registry.js';
205206
import { expandSearchToFilter } from './search-filter.js';
@@ -16365,6 +16366,14 @@ export class ObjectQL implements IObjectQLEngine {
1636516366
// the `where` doors above, before the AST is built and tokens resolve).
1636616367
query = this.expandSearchOnAggregateOptions(object, query);
1636716368
this.rejectCredentialAggregation(object, query);
16369+
// [#20783] …and a `groupBy` entry naming a structured-JSON field (`json`,
16370+
// `composite`, `address`, …) is refused `INVALID_FIELD` / 400 here, before
16371+
// any driver is asked: the drivers share no meaning for a JSON document as
16372+
// a group key (memory merged every row into one group, SQLite grouped each
16373+
// serialized document apart, PostgreSQL answered 500). After the
16374+
// credential refusal, which reads the same entries, so a protected field
16375+
// keeps that refusal's words.
16376+
assertGroupByNamesNoStructuredJsonField(object, this._registry.getObject(object), query.groupBy);
1636816377
// [#10576] The per-aggregation `filter` (`AggregationNodeSchema.filter`,
1636916378
// the contract half of #10413) is a second filter position on this verb,
1637016379
// so it walks through the same refusal doors `where` does at this seam:

0 commit comments

Comments
 (0)