Skip to content

Commit 4727fcb

Browse files
fix(service-analytics)!: refuse a JSON-stored dimension or count_distinct over a relationship path the cube declares no join for, located through the one hop resolver (#21247)
Fixes #21232 Clause-②: no (narrowing) ## What changed The structured-JSON door (`structured-json-dimension-door.ts`) now locates a dotted path's column through the one hop resolver, the way every other reader in the package does. - **`columnOf`** asks `columnObjectOf` (`hop-object.ts`, consumed unchanged) with the host's hop reference. The answer is the cube's declared join at that path, else the relationship field's declared `reference`, else the alias. That is the object both strategies join and read for the path. Before, `columnOf` read `cube.joins` alone and stood down on a path the cube declares no join for. ⛔ No second resolver: the hop walk is `hop-object.ts`'s, as for the measure side in PR #21230. - **`assertNoStructuredJsonDimension`** takes one more argument, `referenceOf` (the `HopReference` type from `hop-object.ts`). Its one caller, `AnalyticsService.assertDimensionsGroupScalarColumns`, passes `this.hopReference`. That is the same function the field gate, the admitted and scoped set, and both strategies resolve a hop with. The function is internal to the package (not exported from `index.ts`). - The refusal words and the envelope are unchanged. A member over an undeclared-join path now gets the refusal a member over a declared join already got: `INVALID_FIELD` / 400, `member`, `param`, `cube`, `field` (the path), `object` (the object the lookup declares as its target). ## Measured: `POST /api/v1/analytics/query` and `/sql` on the real dispatcher route Setup: `AnalyticsServicePlugin` over a real `ObjectQL` engine and `SqlDriver`, a signed-in caller, the real `dispatcher-plugin` mount, SQLite in memory and a private PostgreSQL 16.14. A configured cube over `os21232_deal` declares a join for `account` (`hq` `json`, `name` `text`) and none for `owner`. `owner` is a lookup whose `reference` is `os21232_person` (`prefs` `json`, `labels` `tags`, `email` `text`). Before: `origin/main` at `3a7b6eb0`. After: this branch's `service-analytics` build. There are 80 cells (2 drivers x 2 faces x 2 doors x 10 members). 32 changed and 48 are byte-identical. The scratch probe was deleted. | member | face | SQLite before → after | PostgreSQL 16.14 before → after | |:--|:--|:--|:--| | dimension `owner.prefs` (json) / `owner.labels` (tags) | native | 200, one group per serialized value → **400 `INVALID_FIELD`** | **500 `DATABASE_ERROR`** → **400** | | the same | ObjectQL | 400 `INVALID_FIELD` from the engine, `groupBy[1]` → **400 from this door, naming the member** | the same | | `count_distinct` over `owner.prefs` / `owner.labels` | native | 200, `2` / `2` → **400** | **500** → **400** | | the same | ObjectQL | 400, the cross-object refusal (no `field` / `object`) → **400 from this door** | the same | | any of the above on `/sql` (dry run) | both | 200 (native, and the ObjectQL dimension) → **400** | the same | | control: the same members over `account.hq` (declared join) | both | 400 `INVALID_FIELD`, this door → unchanged | unchanged | | control: `owner.email` dimension | both | 200 → unchanged | unchanged | | control: `owner.email` / `account.name` count_distinct | native / ObjectQL | 200 / 400 cross-object refusal → unchanged | unchanged | ## Pins **New file:** `packages/services/service-analytics/src/__tests__/json-stored-door-undeclared-join.test.ts`. It uses the plugin's own composition over a real engine, a SQLite cell and a PostgreSQL cell (a named skip without `OS_TEST_POSTGRES_URL`), and both faces. It has 10 tests, 5 per cell. - A dimension and a `count_distinct` over `owner.prefs` / `owner.labels` are refused `INVALID_FIELD` / 400 on both faces, with nothing read. The checked fields are `code`, `status`, `member`, `param`, `cube`, `field` (the path) and `object` (`os21232_person`), and the raw-SQL and engine-aggregate counters stay at 0. The two faces' envelopes must be equal. Neither face's own refusal carries this envelope, so equality shows the door answered. - An ad-hoc query's inferred cube declares no join at all. A dotted dimension on it (`owner.prefs`) is refused the same way. - The dry-run door refuses what the query door refuses. - Controls: the same members over the declared join `account.hq` get the same refusal (`object` the joined object). The scalar `owner.email` dimension is served on both faces (one group per owner), and its `count_distinct` answers `3`. **Unit file** `dimension-structured-json-door.test.ts`: a new block with 5 tests. The relationship field's declared reference names the object, on both faces (`crm_account`, not the alias). The reference wins over an alias that names a described object (the door stands down and the statement joins `"crm_account"`). A host with no reference resolves the alias, and both doors refuse there. Control: the referenced object's text column is served. ## Ablations The tests import the subject by relative path (`../analytics-service.js`, `../plugin.js`), so each run reads `src` and there is no `dist` leg. Every mutation went through `scripts/ablation-replace.mjs` in WRAP mode, with an outer `trap` restore on `EXIT INT TERM` against the absolute path. Predictions were written before each run. They ran from committed `08ea135b`, and `git diff 08ea135 b6e6418 -- packages/services/service-analytics` is empty. The PostgreSQL cell was live. | ablation | mutation | predicted | observed | |:--|:--|:--|:--| | A1 | `columnOf` gets back the joins-only resolution (the removed code, byte for byte) | 9 red: the undeclared-join, ad-hoc and dry-run tests on each cell, the two reference-tier face tests and the alias-tier test | **9 failed / 19 passed** | | A2 | the call site passes `undefined` in place of `this.hopReference` | 9 red: the same six live tests, the two reference-tier face tests, and "the reference, not the alias" (the alias tier stays green) | **9 failed / 19 passed** | The prediction's total (30) was an arithmetic slip: 28 tests ran. Each mutation landed: anchor 1 → 0, and the blob changed (A1 `04bf4095e712` → `db237bdc284e`, A2 `16d63d721b6e` → `5d257e0a59d3`). Each was restored and proven: the blob equals the HEAD blob, `git diff HEAD` is empty, and porcelain shows 0. An earlier pair of runs at `50d5511f`, before the ad-hoc pin existed, gave 7 failed / 19 passed for each, as predicted then. ## Fixture triage The full `service-analytics` suite turned up exactly one fixture that pinned the removed stand-down: `dimension-structured-json-door.test.ts`, "a dotted path the cube declares no join for is a synthetic traversal … not judged". It pinned the branch this PR deletes, so it was replaced, not respelled. Its stand-down now has an honest reason, a host that describes nothing on the object the hop reaches (`describes: null`), and the new block above pins the judged tiers. Consumer radius: the analytics fixtures with dotted members in `packages/rest` (8 files), `packages/runtime` (4) and `packages/driver-memory` (3) were run against this branch's build. They gave 107 passed / 3 skipped, 24 / 10 and 239 / 0, with no failures. ## Other readers of `cube.joins` in `service-analytics` (dispatch Zone 2, item 2) No other reader resolves a dotted path through `cube.joins` alone. Every path-splitting reader goes through `resolvePathHops` / `columnObjectOf`. Six sites enumerate `cube.joins` without splitting a path. They are listed for the seat and not touched here. - `strategies/native-sql-strategy.ts` `qualifyAndRegisterJoin`, `canJoin`: it qualifies a bare base column only when the cube declares a join. **Measured** at `b6e64185` (this file is byte-identical to `origin/main`). Take a configured cube with no declared join and the dimensions `note` + `owner.email`, where the lookup's target also declares `note`. The native face answers **500 `DATABASE_ERROR`** on SQLite ("ambiguous column name: note") and PostgreSQL (42702). The ObjectQL face answers 200. This is reported to the seat as a finding and is not handled in this PR. - `strategies/native-sql-strategy.ts` `canHandle`, the federated-object decline: it asks `isExternalObject` of the declared join targets only, not of an object reached through a path with no declared join. Reach not measured. - `native-sql-strategy.ts` (three sites) and `analytics-service.ts` `cubeObjects`: these read the declared joins as a fallback for a context built without `readScopedObjects`, or beside `namedQueryFields`, which adds the path-reached objects. No defect was found. ## Docs `git grep -nE "declares no join|declared join|structured-JSON|structured JSON|count_distinct"` over `content/docs/**`, excluding `releases/` and `references/`, gave 32 hits. None says the door stands down on a path without a declared join, and none speaks about a `count_distinct` over a related field's JSON-stored column. Positive control: the pattern hits `content/docs/deployment/validating-metadata.mdx:228`, the dataset-dimension door sentence. That page speaks about datasets, whose dotted fields must traverse a declared `include` (a declared join), and it stays true. No page edited. ## Deviation from the declared file surface The claim names `structured-json-dimension-door.ts` (`columnOf`) and its tests. Routing `columnOf` through the resolver's reference tier, which is the triage direction, needs the host's `HopReference`, and only the door's one caller holds it. So `analytics-service.ts` changes by one argument (`this.hopReference`) and three docblock lines in `assertDimensionsGroupScalarColumns`, the door's caller. Nothing else in that file moved. A2 above pins that line. ## Verification at `b6e64185` The branch head `fafbf053` carries a tree byte-identical to `b6e64185` (`git diff b6e6418 fafbf05` is empty), so every reading below holds for it. `b6e64185` has `origin/main` (`ef96c9ed`) merged in. Install and a full build were refreshed after the merge, and `pnpm --filter @objectstack/spec check:generated` reports all 15 artifacts up to date. - `pnpm --filter @objectstack/service-analytics test`: 164 files, 3758 passed / 56 skipped (the live-PostgreSQL cells), 0 failed. `typecheck` (`tsc --noEmit`): exit 0. - PostgreSQL 16.14 live (`OS_TEST_POSTGRES_URL`): `json-stored-door-undeclared-join`, `json-stored-door-live-drivers`, `cube-measure-relationship-path-type`, `dimension-structured-json-door` and `multi-value-json-stored-door` gave 72 passed / 0 skipped. `runtime` `analytics-json-dimension-door` and `analytics-cube-measure-field-type-door` gave 20 passed / 0 skipped (run at `9a32e950`, whose analytics source is the same). - Gates: `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands` derives 62 commands, and all 62 exit 0 at `b6e64185`. `--ran` reconciliation: "62 derived, 62 run, 0 NOT-MEASURED, 0 UNRUN", with every exit code recorded. `check:adr-0087-registration` was red once, on the changeset's "FROM → TO" label, which the gate reads as a rewrite prescription. The section describes behaviour, not a rewrite, so it was relabelled "Before and after". The gate is now green with `not-required (no-migration-prescription)`, the disposition the door's earlier entries carry. - Lint, a declared narrowing: `eslint --no-inline-config --format json` over the 4 touched `.ts` files at `b6e64185` reports 4 files, 0 errors and 0 warnings. ① All 4 are inside the population `eslint.config.mjs` lints (`packages/**/*.{ts,tsx,mts,cts}`; none ignored), and the changeset `.md` is in no `files` glob. ② The count of 4 is read from the JSON output. ③ The config enables no type-aware linting (`--print-config` gives `parserOptions` `{ecmaVersion:'latest', sourceType:'module'}`, with no `project`), so this diff cannot move a verdict on an untouched file. The repo-wide `pnpm lint` is left to CI. ## Acceptance notes - **Carrier note for release compilation (not filed).** The two pending release notes for this door, `20807-analytics-json-dimension-refused.md` and `20912-analytics-multi-value-distinct-refused.md`, list a dotted path the cube declares no join for as Unchanged. This PR makes that clause untrue, and its own changeset states the reversal. When the CHANGELOG is assembled, the release compiler should drop that clause from both entries. They cannot be corrected from this PR: editing another PR's pending changeset is a foreign-changeset edit that stays refused until a person confirms it (#17712). - The ObjectQL face's own refusals of these members (the engine's `groupBy[1]`, the cross-object measure refusal) are now unreachable for them, because the door answers first. Neither refusal changes. - No route-level pin was added. The dispatcher relays this door's envelope generically, and `packages/runtime/src/analytics-json-dimension-door.test.ts` already pins that relay for this door. The route-level readings above were taken through the real route. --- _Generated by [Claude Code](https://claude.ai/code/session_01DiCSbmJrkzNhuEAier4VoJ)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4b59a38 commit 4727fcb

5 files changed

Lines changed: 491 additions & 23 deletions

File tree

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
"@objectstack/service-analytics": minor
3+
---
4+
5+
fix(service-analytics)!: a grouped dimension or a `count_distinct` measure over a JSON-stored column reached through a relationship path the cube declares no join for is refused with `INVALID_FIELD` / 400 at the analytics door, as the same member over a declared join already was
6+
7+
Clause-②: no (narrowing)
8+
9+
<!-- adr-0087: not-required (no-migration-prescription) a refusal of a grouping or distinct-count TARGET at the analytics door, extended from a relationship path the cube declares a join for to one it does not: the door now locates the column on the object the one hop resolver names (the declared join, else the relationship field's declared reference), the object both strategies already join and read. No authorable key, spelling, export or stored shape moves (`assertNoStructuredJsonDimension` is internal to the package; `CubeSchema`, `DatasetSchema` and the analytics query body keep parsing every member), and no stored row is read or rewritten. The grouping and the distinct count had no shared meaning to preserve (one group or one distinct value per serialized document on SQLite, a 500 on PostgreSQL), and which scalar part a caller meant is not something a ledger entry can rewrite. The other categories are closed on facts: the package publishes (not `unpublished`); no ADR-0087 id covers an analytics grouping or distinct-count target, and this diff adds none (not `registered` / `already-registered`); and the change is runtime behaviour, not a declaration (not `runtime-interface-only` / `type-surface-only`). -->
10+
11+
**BREAKING**: this narrows what `POST /api/v1/analytics/query` and its dry run `POST /api/v1/analytics/sql` accept, on both strategies and every driver. It ships as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes.
12+
13+
The column of a dotted path is now located by the one hop resolver both strategies join and read it through: the cube's declared join at that path, else the relationship field's declared `reference`, else the relationship's own name for a host that cannot answer. Before, the door read the cube's declared joins alone and stood down on a path the cube declares no join for. This reverses one clause of the earlier entries for this door in the same release, which listed such a path as unchanged.
14+
15+
**Before and after**, measured on a configured cube over an object whose lookup the cube declares no join for — `owner`, declaring `reference` a person object — and a member over that lookup. An ad-hoc query's inferred cube declares no join at all, and a dotted dimension on it (`owner.prefs`) now gets the same refusal:
16+
17+
- A `dimensions` entry, or a `timeDimensions` entry with a `granularity`, over a structured-JSON field (`owner.prefs`, `json`) or a multi-value field (`owner.labels`, `tags`; or a field declared `multiple: true`). Before, on the native-SQL strategy: `200` with one group per serialized value on SQLite and `500 DATABASE_ERROR` on PostgreSQL; the ObjectQL strategy answered `400 INVALID_FIELD` from the engine under its own position (`groupBy[1]`), a name the request never wrote. Now: `400 INVALID_FIELD` from this door on both strategies, before either reads anything.
18+
- A `count_distinct` measure over the same columns. Before, on the native-SQL strategy: `200` with a count of serialized values on SQLite and `500` on PostgreSQL; the ObjectQL strategy refused it as a cross-object measure. Now: the same `400 INVALID_FIELD` from this door.
19+
20+
**What an author sees now.** The refusal the same member over a declared join already got: `400 INVALID_FIELD`, naming the member as the request wrote it, the cube, the path, the object the lookup declares as its target and the column's declared type, saying the query was not run, and naming the route. The thrown error carries `member`, `param` (`dimensions`, `timeDimensions` or `measures`), `cube`, `field` (the path, `owner.prefs`) and `object` (the target object).
21+
22+
**What to write instead.** Group by, or count distinct, a related field that stores one scalar value. For a multi-value field, run a record query on the target object filtered by one member with `$contains`, one query per member.
23+
24+
**Who is affected.** A dashboard, report or caller that grouped or counted distinct such a related column through a lookup the cube declares no join for, on SQLite, and read the serialized values as real groups or a real count. On PostgreSQL the same queries were already a 500. No example app and no shipped cube or dataset authors such a member.
25+
26+
**Unchanged.** Every member over a declared join; a scalar related column (`owner.email`), which is served on both strategies; a related column whose object the host's field metadata does not describe; an expression `sql`; a host that wires no `sourceFieldMeta`; and a dataset dimension over an `include`d relationship, whose join the dataset compiler declares.

‎packages/services/service-analytics/src/__tests__/dimension-structured-json-door.test.ts‎

Lines changed: 66 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,10 @@
2020
* column, a cube-qualified spelling, a dataset dimension, an ad-hoc
2121
* (inferred) cube, and a bucketed time dimension;
2222
* - no statement and no aggregate reaches the host bridges on a refusal;
23+
* - [#21232] a relationship path the cube declares no join for is judged on
24+
* the object the one hop resolver (`hop-object.ts`) names — the field's
25+
* declared reference, else the alias — and stands down only where the host
26+
* describes nothing there;
2327
* - GUARD: the judged types are exactly `@objectstack/spec/data`'s
2428
* `STRUCTURED_JSON_TYPES` and `isMultiValueField`, over every `FieldType` —
2529
* the predicates the engine's door reads, never a second list.
@@ -99,7 +103,16 @@ interface Refusal extends Error {
99103
cube?: string;
100104
}
101105

102-
function makeService(face: Face, opts: { sourceFieldMeta?: boolean } = {}) {
106+
/**
107+
* `describes`: the object whose fields the host describes as {@link JOINED_FIELDS}
108+
* (`null`: none). `reference`: the target a relationship resolver declares for
109+
* `ledger.account` (absent: no resolver is wired).
110+
*/
111+
function makeService(
112+
face: Face,
113+
opts: { sourceFieldMeta?: boolean; describes?: string | null; reference?: string } = {},
114+
) {
115+
const described = opts.describes === undefined ? JOINED : opts.describes;
103116
const calls = { raw: [] as string[], aggregate: [] as unknown[] };
104117
const service = new AnalyticsService({
105118
logger: silentLogger,
@@ -117,7 +130,10 @@ function makeService(face: Face, opts: { sourceFieldMeta?: boolean } = {}) {
117130
getObjectFieldNames: (n: string) => (n === OBJECT ? Object.keys(FIELDS) : undefined),
118131
...(opts.sourceFieldMeta === false
119132
? {}
120-
: { sourceFieldMeta: (o: string, f: string) => (o === OBJECT ? FIELDS[f] : o === JOINED ? JOINED_FIELDS[f] : undefined) }),
133+
: { sourceFieldMeta: (o: string, f: string) => (o === OBJECT ? FIELDS[f] : o === described ? JOINED_FIELDS[f] : undefined) }),
134+
...(opts.reference === undefined
135+
? {}
136+
: { relationshipResolver: (o: string, rel: string) => (o === OBJECT && rel === 'account' ? opts.reference : undefined) }),
121137
});
122138
return { service, calls };
123139
}
@@ -218,6 +234,52 @@ describe('a dimension on a structured-JSON field is refused at the analytics doo
218234
});
219235
});
220236

237+
describe('[#21232] a dotted path the cube declares no join for is judged on the object the one hop resolver names', () => {
238+
// `ledger_cube` declares no join; `account.hq` is an undeclared member, so
239+
// its column is the path itself. The live-driver pins over the plugin's own
240+
// composition are `json-stored-door-undeclared-join.test.ts`.
241+
const REFERENCED = 'crm_account';
242+
243+
for (const face of ['native', 'objectql'] as const) {
244+
it(`${face}: the relationship field's declared reference names the object — INVALID_FIELD / 400, nothing read`, async () => {
245+
const { service, calls } = makeService(face, { describes: REFERENCED, reference: REFERENCED });
246+
const err = await rejection(service.query({ cube: 'ledger_cube', measures: ['count'], dimensions: ['account.hq'] }));
247+
expect(envelopeOf(err)).toEqual({
248+
code: 'INVALID_FIELD', status: 400, member: 'account.hq', param: 'dimensions', field: 'account.hq', object: REFERENCED,
249+
});
250+
expect(calls.raw, 'no statement reached the raw-SQL bridge').toEqual([]);
251+
expect(calls.aggregate, 'no aggregate reached the engine bridge').toEqual([]);
252+
});
253+
}
254+
255+
it('the reference, not the alias: an alias that names a described object is not what the door reads', async () => {
256+
// The host describes `account` (the alias) with `hq` json, but the lookup
257+
// declares `crm_account`, which it does not describe: the strategy joins
258+
// `crm_account`, so the door has nothing to answer and stands down.
259+
const { service, calls } = makeService('native', { describes: JOINED, reference: REFERENCED });
260+
await service.query({ cube: 'ledger_cube', measures: ['count'], dimensions: ['account.hq'] });
261+
expect(calls.raw).toHaveLength(1);
262+
expect(calls.raw[0]).toContain(`"${REFERENCED}"`);
263+
});
264+
265+
it('a host that names no reference: the hop reads its alias, the table the strategy joins, and is judged there', async () => {
266+
const { service, calls } = makeService('native');
267+
const err = await rejection(service.query({ cube: 'ledger_cube', measures: ['count'], dimensions: ['account.hq'] }));
268+
expect(envelopeOf(err)).toEqual({
269+
code: 'INVALID_FIELD', status: 400, member: 'account.hq', param: 'dimensions', field: 'account.hq', object: JOINED,
270+
});
271+
const dry = await rejection(service.generateSql({ cube: 'ledger_cube', measures: ['count'], dimensions: ['account.hq'] }));
272+
expect(envelopeOf(dry)).toEqual(envelopeOf(err));
273+
expect(calls.raw).toEqual([]);
274+
});
275+
276+
it('CONTROL the referenced object\'s text column is served', async () => {
277+
const { service, calls } = makeService('native', { describes: REFERENCED, reference: REFERENCED });
278+
await service.query({ cube: 'ledger_cube', measures: ['count'], dimensions: ['account.name'] });
279+
expect(calls.raw).toHaveLength(1);
280+
});
281+
});
282+
221283
describe('what the door does not judge', () => {
222284
it('CONTROL a text dimension is served — the native face runs its one statement', async () => {
223285
const { service, calls } = makeService('native');
@@ -233,8 +295,8 @@ describe('what the door does not judge', () => {
233295
expect(err.message).toContain("which object 'ledger' does not have");
234296
});
235297

236-
it('a dotted path the cube declares no join for is a synthetic traversal: its object is not a declaration, so it is not judged', async () => {
237-
const { service, calls } = makeService('native');
298+
it('a dotted path whose column the host does not describe cannot be answered, so the door stands down', async () => {
299+
const { service, calls } = makeService('native', { describes: null });
238300
await service.generateSql({ cube: 'ledger_cube', measures: ['count'], dimensions: ['account.hq'] });
239301
await service.query({ cube: 'ledger_cube', measures: ['count'], dimensions: ['account.hq'] });
240302
expect(calls.raw).toHaveLength(1);

0 commit comments

Comments
 (0)