diff --git a/.changeset/20912-analytics-multi-value-distinct-refused.md b/.changeset/20912-analytics-multi-value-distinct-refused.md new file mode 100644 index 00000000000..387497e9e85 --- /dev/null +++ b/.changeset/20912-analytics-multi-value-distinct-refused.md @@ -0,0 +1,27 @@ +--- +"@objectstack/service-analytics": minor +--- + +fix(service-analytics)!: a grouped dimension on a multi-value field and a `count_distinct` measure over a JSON-stored field are refused with `INVALID_FIELD` / 400 at the analytics door, before any SQL is built; a dataset `count_distinct` measure over a field declared `multiple: true` is refused at compile time + +Clause-②: no (narrowing) + + + +**BREAKING**: this narrows what the analytics query doors accept, in two positions, and what the dataset compiler accepts, in one. It holds on `POST /api/v1/analytics/query`, on its dry run `POST /api/v1/analytics/sql` and on `POST /api/v1/analytics/dataset/query`, on every driver and on both strategies. It ships as `minor` under the launch-window convention for accept-set narrowings. + +- A cube or dataset dimension whose column is a **multi-value** field, when it groups the result (a `dimensions` entry, or a `timeDimensions` entry with a `granularity`): an inherently multi option type (`multiselect`, `checkboxes`, `tags`), or a `select`, `radio`, `lookup`, `user`, `file` or `image` field declared `multiple: true`. The same types without the flag are served. +- A `measures` entry that resolves to a `count_distinct` measure whose column is **JSON-stored**: a structured-JSON type (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`), an inherently multi option type, or a multi-capable field declared `multiple: true`. An authored cube measure, a suffix-inferred one (`tags_count_distinct`) and a compiled dataset's are judged alike; a `count` measure is not judged. +- A dataset measure that pairs `count_distinct` with a base-object field declared `multiple: true` is refused `400 DATASET_INVALID` when the dataset compiles, at registration and on the request door, beside the type row that already refused `tags`. + +The column is judged where it is declared: on the cube's object, or, for a dotted path, on the object the cube's declared join names. + +**What an author sees now.** `400 INVALID_FIELD`, naming the member as the request wrote it, the column, the object, the declaration (`select with multiple: true`), saying the query was not run, and naming the route. For a multi-value field the route is a record query on the declaring object filtered by one member with `$contains`, one query per member. For a structured-JSON field it is to store the scalar part in a field of its own. The thrown error carries `member`, `param` (`dimensions`, `timeDimensions` or `measures`), `cube`, `field` and `object`. The dataset compile refusal names the measure, the field and its declaration with `multiple: true`. + +**Why a refusal.** The engine's aggregate door already refuses both shapes, and the native-SQL strategy compiled its own statement and never reached it. Measured through this service as `AnalyticsServicePlugin` composes it over a real engine: a dimension on a `tags`, `multiselect` or `multiple: true` select answered 200 with one group per serialized array on SQLite and 500 `DATABASE_ERROR` on PostgreSQL 16; an inferred `count_distinct` over a `json`, `tags` or `multiple: true` select field answered 2, 3 and 2 on SQLite and 500 on PostgreSQL; a dataset `count_distinct` over the `multiple: true` select registered, then answered 2 on SQLite and 500 on PostgreSQL. The engine-aggregate strategy answered 400 for every one of these, under the engine's position (`groupBy[0]`, `aggregations[0].field`) rather than the member the caller wrote. The predicates are `@objectstack/spec/data`'s, the ones the engine's doors read: `isMultiValueField`, and the aggregate × field-type table's `count_distinct` row. + +**Your fix (a host calling `compileDataset` directly).** `DatasetCompileOptions` no longer has `declaredFieldType`. Pass `declaredValueShape` instead: the same read of the field's metadata, answering `{ type, multiple }` (the type, and `multiple === true`) rather than the type alone, or `undefined` when nothing answers. A host that passes neither compiles exactly as before, with no aggregate × field-type refusal at all. `AnalyticsService` and `AnalyticsServicePlugin` wire it themselves from `sourceFieldMeta`. + +**Who is affected.** A dashboard, report or caller that grouped by a multi-value field, or counted a JSON-stored field distinct, through the native-SQL strategy on SQLite and read the serialized arrays or the text-compared count as real answers. On PostgreSQL the same queries were already a 500. A dataset that pairs `count_distinct` with a `multiple: true` field no longer registers. + +**Unchanged.** A dimension or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; a member naming a column the object does not have, which keeps its existing `INVALID_FIELD` answer first; a dotted path the cube declares no join for; a measure whose `sql` is an expression; and a host that wires no `sourceFieldMeta`, where the declaration cannot be read. diff --git a/packages/rest/src/analytics-dataset-multi-value-door.test.ts b/packages/rest/src/analytics-dataset-multi-value-door.test.ts new file mode 100644 index 00000000000..0ebe3edcfd7 --- /dev/null +++ b/packages/rest/src/analytics-dataset-multi-value-door.test.ts @@ -0,0 +1,242 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * A dataset dimension on a MULTI-VALUE field, and a dataset `count_distinct` + * measure over a `select` declared `multiple: true`, are refused at the + * dataset door — `POST /api/v1/analytics/dataset/query` answers 400 before + * any SQL is built — over a real `SqlDriver`; the single-value controls are + * served unchanged. + * + * The cube face (`POST /api/v1/analytics/query`, an inferred + * `FIELD_count_distinct` included) is served by `@objectstack/runtime`'s + * dispatcher, not by this package; its rows are pinned at the service, on the + * same two dialects, in + * `packages/services/service-analytics/src/__tests__/json-stored-door-live-drivers.test.ts`. + * + * ## Measured without the refusal, through this door + * + * Four rows: `status` a, a, b, b; `tags_f` [a,b], [a,b], [b], [ab]; + * `multi_sel` (a `select` with `multiple: true`) [a,b], [a,b], [b], [b]: + * + * | the inline dataset's selection | SQLite | PostgreSQL 16 | + * |:--|:--|:--| + * | dimension `status_dim` (single-value select, the control) | 200, `A` 2 · `B` 2 (the option labels) | same | + * | dimension `tag_dim` (`tags`) / `picked_dim` (`multi_sel`) | 200, one group per serialized array | 500 | + * | measure `count_distinct` over `multi_sel` | 200, 2 | 500 | + * | measure `count_distinct` over `status` (the control) | 200, 2 | same | + * + * ## The composition, and the dialect axis of THIS file + * + * The analytics service is the one `AnalyticsServicePlugin` composes over a + * real `ObjectQL` engine — both auto-bridges live, so `NativeSQLStrategy` + * answers on a SQL driver. The SQLite cell always runs. The PostgreSQL cell + * runs where `OS_TEST_POSTGRES_URL` is set and is a named skip otherwise; no + * CI step provisions that variable for this package, so the live cell is + * red-capable and un-run in CI. The live cell owns its table, dropped before + * and after. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { AnalyticsServicePlugin, type AnalyticsService } from '@objectstack/service-analytics'; +import { RestServer } from './rest-server'; + +const OBJECT = 'rest_dataset_multi_value_ledger'; +const OPTIONS = [{ label: 'A', value: 'a' }, { label: 'B', value: 'b' }]; + +const LEDGER = { + name: OBJECT, + label: 'Dataset multi-value door ledger', + fields: { + status: { name: 'status', type: 'select' as const, options: OPTIONS }, + tags_f: { name: 'tags_f', type: 'tags' as const }, + multi_sel: { name: 'multi_sel', type: 'select' as const, multiple: true, options: OPTIONS }, + }, +}; + +const ROWS = [ + { id: 'r1', status: 'a', tags_f: ['a', 'b'], multi_sel: ['a', 'b'] }, + { id: 'r2', status: 'a', tags_f: ['a', 'b'], multi_sel: ['a', 'b'] }, + { id: 'r3', status: 'b', tags_f: ['b'], multi_sel: ['b'] }, + { id: 'r4', status: 'b', tags_f: ['ab'], multi_sel: ['b'] }, +]; + +/** The inline dataset the request carries — as a Studio preview or a widget posts it. */ +const dataset = (measures: unknown[]) => ({ + name: 'multi_value_inline', + label: 'Multi-value inline', + object: OBJECT, + dimensions: [ + { name: 'status_dim', field: 'status', type: 'string' }, + { name: 'tag_dim', field: 'tags_f', type: 'string' }, + { name: 'picked_dim', field: 'multi_sel', type: 'string' }, + ], + measures, +}); +const ROW_COUNT = [{ name: 'row_count', aggregate: 'count' }]; + +interface Cell { + id: 'sqlite' | 'pg'; + label: string; + env: string | null; + config: () => Record | null; +} + +const CELLS: readonly Cell[] = [ + { id: 'sqlite', label: 'sqlite', env: null, config: () => ({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }) }, + { + id: 'pg', + label: 'live postgres', + env: 'OS_TEST_POSTGRES_URL', + config: () => (process.env.OS_TEST_POSTGRES_URL ? { client: 'pg', connection: process.env.OS_TEST_POSTGRES_URL } : null), + }, +]; + +const quiet = { debug() {}, info() {}, warn() {}, error() {}, child() { return quiet; } }; + +function createMockServer() { + const noop = () => {}; + return { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} }; +} + +function mockProtocol() { + return { + getDiscovery: async () => ({ version: 'v0', routes: { data: '', metadata: '' } }), + getMetaTypes: async () => [], + getMetaItems: async () => [], + }; +} + +function makeRes() { + const res: any = { + statusCode: 200, + body: undefined as any, + header: () => res, + status: (code: number) => { res.statusCode = code; return res; }, + json: (body: unknown) => { res.body = body; return res; }, + end: () => res, + }; + return res; +} + +for (const cell of CELLS) { + const config = cell.config(); + describe.skipIf(!config)( + `a multi-value dataset dimension or distinct count at POST /api/v1/analytics/dataset/query — ${cell.label}${config ? '' : ` (skipped: set ${cell.env} to run this cell)`}`, + () => { + let driver: any; + let engine: ObjectQL; + /** Raw-SQL statements and engine aggregates that read THIS object. */ + const reads = { rawSql: 0, aggregate: 0 }; + let query: (measures: unknown[], selection: Record) => Promise<{ status: number; body: any }>; + + const dropTable = async () => { + if (cell.id === 'sqlite') return; + await driver?.execute(`drop table if exists ${OBJECT}`).catch(() => {}); + }; + + beforeAll(async () => { + driver = new SqlDriver(config as any); + await dropTable(); + engine = new ObjectQL({ logger: quiet } as any); + engine.registerDriver(driver, true); + await engine.init(); + engine.registry.registerObject(LEDGER as any); + await engine.syncSchemas(); + for (const row of ROWS) await engine.insert(OBJECT, { ...row } as any); + + const realExecute = (engine as any).execute.bind(engine); + (engine as any).execute = (sql: unknown, opts?: { object?: string }) => { + if (opts?.object === OBJECT) reads.rawSql += 1; + return realExecute(sql, opts); + }; + const realAggregate = engine.aggregate.bind(engine); + (engine as any).aggregate = (object: string, ...rest: unknown[]) => { + if (object === OBJECT) reads.aggregate += 1; + return (realAggregate as any)(object, ...rest); + }; + + // The plugin's own composition over the real engine: both auto-bridges. + const registered: Record = {}; + await new AnalyticsServicePlugin().init({ + getService: (name: string) => (name === 'data' ? engine : registered[name]), + registerService: (name: string, svc: unknown) => { registered[name] = svc; }, + replaceService: (name: string, svc: unknown) => { registered[name] = svc; }, + hook: () => {}, + logger: quiet, + } as never); + const service = registered.analytics as AnalyticsService; + + const rest = new RestServer( + createMockServer() as any, mockProtocol() as any, { api: { requireAuth: false } } as any, + undefined, undefined, undefined, undefined, undefined, undefined, undefined, + undefined, undefined, undefined, undefined, + async () => service, + ); + (rest as any).resolveExecCtx = async () => ({ userId: 'test-user' }); + rest.registerRoutes(); + const route = rest.getRoutes().find((r: any) => r.method === 'POST' && r.path === '/api/v1/analytics/dataset/query'); + expect(route).toBeDefined(); + query = async (measures, selection) => { + const res = makeRes(); + // What the wire carries: JSON. + const body = JSON.parse(JSON.stringify({ dataset: dataset(measures), selection })); + await route!.handler({ method: 'POST', params: {}, headers: {}, body, query: {} } as any, res); + return { status: res.statusCode, body: res.body }; + }; + }); + + afterAll(async () => { + await dropTable(); + try { await engine?.destroy(); } catch { /* noop */ } + }); + + it('a dataset dimension on a multi-value field answers 400 INVALID_FIELD naming the dataset dimension — no statement reaches the engine', async () => { + const before = { ...reads }; + for (const [member, column, declared] of [ + ['tag_dim', 'tags_f', 'tags'], + ['picked_dim', 'multi_sel', 'select with multiple: true'], + ] as const) { + const res = await query(ROW_COUNT, { measures: ['row_count'], dimensions: [member] }); + expect(res.status, JSON.stringify(res.body)).toBe(400); + expect(res.body.code, member).toBe('INVALID_FIELD'); + expect(String(res.body.message), member).toContain( + `Dimension '${member}' on cube 'multi_value_inline' groups by field '${column}', which object '${OBJECT}' declares as ${declared} — a multi-value field`, + ); + } + expect(reads, 'no raw SQL and no engine aggregate for the object').toEqual(before); + }); + + it('CONTROL a single-value select dimension is served unchanged: one group per option, counted under its label', async () => { + const res = await query(ROW_COUNT, { measures: ['row_count'], dimensions: ['status_dim'] }); + expect(res.status, JSON.stringify(res.body)).toBe(200); + const groups = (res.body.rows as Array<{ status_dim: string; row_count: number | string }>) + .map((r) => [r.status_dim, Number(r.row_count)] as const) + .sort(([a], [b]) => a.localeCompare(b)); + expect(groups).toEqual([['A', 2], ['B', 2]]); + }); + + it('a dataset count_distinct over a select with multiple: true answers 400 DATASET_INVALID — no statement reaches the engine', async () => { + const before = { ...reads }; + const res = await query( + [{ name: 'picked_kinds', aggregate: 'count_distinct', field: 'multi_sel' }], + { measures: ['picked_kinds'] }, + ); + expect(res.status, JSON.stringify(res.body)).toBe(400); + expect(res.body.code).toBe('DATASET_INVALID'); + expect(String(res.body.message)).toContain('declares as `select` with `multiple: true`'); + expect(reads, 'no raw SQL and no engine aggregate for the object').toEqual(before); + }); + + it('CONTROL a dataset count_distinct over a single-value select is served', async () => { + const res = await query( + [{ name: 'status_kinds', aggregate: 'count_distinct', field: 'status' }], + { measures: ['status_kinds'] }, + ); + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(Number(res.body.rows[0].status_kinds)).toBe(2); + }); + }, + ); +} diff --git a/packages/services/service-analytics/src/__tests__/dimension-structured-json-door.test.ts b/packages/services/service-analytics/src/__tests__/dimension-structured-json-door.test.ts index a92940cee0a..0ead1cec4be 100644 --- a/packages/services/service-analytics/src/__tests__/dimension-structured-json-door.test.ts +++ b/packages/services/service-analytics/src/__tests__/dimension-structured-json-door.test.ts @@ -21,13 +21,13 @@ * (inferred) cube, and a bucketed time dimension; * - no statement and no aggregate reaches the host bridges on a refusal; * - GUARD: the judged types are exactly `@objectstack/spec/data`'s - * `STRUCTURED_JSON_TYPES`, over every `FieldType` — the predicate the - * engine's door reads, never a second list. + * `STRUCTURED_JSON_TYPES` and `isMultiValueField`, over every `FieldType` — + * the predicates the engine's door reads, never a second list. */ import { describe, it, expect, vi } from 'vitest'; import type { Cube } from '@objectstack/spec/data'; -import { FieldType, STRUCTURED_JSON_TYPES } from '@objectstack/spec/data'; +import { FieldType, STRUCTURED_JSON_TYPES, isMultiValueField } from '@objectstack/spec/data'; import type { Dataset } from '@objectstack/spec/ui'; import type { AnalyticsQuery } from '@objectstack/spec/contracts'; import { AnalyticsService } from '../analytics-service.js'; @@ -246,7 +246,11 @@ describe('what the door does not judge', () => { expect(calls.raw).toHaveLength(1); }); - it('GUARD the judged types are exactly the spec\'s STRUCTURED_JSON_TYPES, over every FieldType', async () => { + it('GUARD the judged types are exactly the spec\'s STRUCTURED_JSON_TYPES and isMultiValueField, over every FieldType', async () => { + // The multi-value class joined the door with the engine's: an inherently + // multi option type (`multiselect`, `checkboxes`, `tags`) is judged with or + // without the flag. The flagged half is pinned in + // `multi-value-json-stored-door.test.ts`. const { service } = makeService('native'); for (const type of FieldType.options) { const query: AnalyticsQuery = { cube: OBJECT, measures: ['count'], dimensions: [`f_${type}`] }; @@ -255,7 +259,9 @@ describe('what the door does not judge', () => { (e: Refusal) => ({ code: e.code, status: e.status, member: e.member }), ); expect(verdict, type).toEqual( - STRUCTURED_JSON_TYPES.has(type) ? { code: 'INVALID_FIELD', status: 400, member: `f_${type}` } : null, + STRUCTURED_JSON_TYPES.has(type) || isMultiValueField({ type }) + ? { code: 'INVALID_FIELD', status: 400, member: `f_${type}` } + : null, ); } }); diff --git a/packages/services/service-analytics/src/__tests__/json-stored-door-live-drivers.test.ts b/packages/services/service-analytics/src/__tests__/json-stored-door-live-drivers.test.ts new file mode 100644 index 00000000000..4b598b6e3ca --- /dev/null +++ b/packages/services/service-analytics/src/__tests__/json-stored-door-live-drivers.test.ts @@ -0,0 +1,243 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The card's three rows, over a real `SqlDriver`, on the strategy that + * bypassed the engine's aggregate door — each refused before any statement + * reaches the database, each beside a scalar control that is still served. + * + * ## Measured without the refusal, on this composition + * + * `AnalyticsServicePlugin` composed over a real `ObjectQL` engine (both + * auto-bridges live, so `NativeSQLStrategy` answers on a SQL driver), four + * rows: `status` a, a, b, b; `tags_f` [a,b], [a,b], [b], [ab]; `ms` and + * `multi_sel` (a `select` with `multiple: true`) [a,b], [a,b], [b], [b]; + * `meta` {a:1}, {a:1}, {b:1}, {b:1}: + * + * | query | SQLite | PostgreSQL 16 | + * |:--|:--|:--| + * | row 1 — a cube or dataset dimension on `tags_f` / `ms` / `multi_sel` | 200, one group per serialized array | 500 `DATABASE_ERROR` | + * | row 2 — an inferred `FIELD_count_distinct` over `meta` / `tags_f` / `multi_sel` | 200, 2 / 3 / 2 | 500 | + * | row 3 — a dataset `count_distinct` measure over `multi_sel` | registers, then 200, 2 | registers, then 500 | + * | controls — the `status` dimension, `status_count_distinct`, a dataset `count_distinct` over `status` | 200 | 200 | + * + * After the change every row answers 400 on both dialects — `INVALID_FIELD` + * for rows 1 and 2 (the analytics door, naming the member), `DATASET_INVALID` + * for row 3 (the dataset compiler, at registration) — and every control is + * unchanged. + * + * ## The dialect axis + * + * The SQLite cell always runs. The PostgreSQL cell runs where + * `OS_TEST_POSTGRES_URL` is set and is a named skip otherwise; no CI step + * provisions that variable for this package, so the live cell is red-capable + * and un-run in CI. The live cell owns its table, dropped before and after. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { DatasetSchema } from '@objectstack/spec/ui'; +import { AnalyticsServicePlugin } from '../plugin.js'; +import type { AnalyticsService } from '../analytics-service.js'; + +const OBJECT = 'analytics_json_stored_door_ledger'; +const OPTIONS = [{ label: 'A', value: 'a' }, { label: 'B', value: 'b' }]; + +const LEDGER = { + name: OBJECT, + label: 'JSON-stored door ledger', + fields: { + title: { name: 'title', type: 'text' as const }, + status: { name: 'status', type: 'select' as const, options: OPTIONS }, + tags_f: { name: 'tags_f', type: 'tags' as const }, + ms: { name: 'ms', type: 'multiselect' as const, options: OPTIONS }, + multi_sel: { name: 'multi_sel', type: 'select' as const, multiple: true, options: OPTIONS }, + meta: { name: 'meta', type: 'json' as const }, + }, +}; + +const ROWS = [ + { id: 'r1', title: 'x', status: 'a', tags_f: ['a', 'b'], ms: ['a', 'b'], multi_sel: ['a', 'b'], meta: { a: 1 } }, + { id: 'r2', title: 'x', status: 'a', tags_f: ['a', 'b'], ms: ['a', 'b'], multi_sel: ['a', 'b'], meta: { a: 1 } }, + { id: 'r3', title: 'y', status: 'b', tags_f: ['b'], ms: ['b'], multi_sel: ['b'], meta: { b: 1 } }, + { id: 'r4', title: 'y', status: 'b', tags_f: ['ab'], ms: ['b'], multi_sel: ['b'], meta: { b: 1 } }, +]; + +const CUBE = { + name: 'json_stored_door_cube', + title: 'JSON-stored door', + sql: OBJECT, + public: true, + measures: { count: { label: 'Rows', type: 'count' as const, sql: '*' } }, + dimensions: Object.fromEntries( + ['status', 'tags_f', 'ms', 'multi_sel'].map((f) => [f, { label: f, type: 'string' as const, sql: f }]), + ), +}; + +const dataset = (name: string, measures: unknown[], dimensions: unknown[] = []) => + DatasetSchema.parse({ name, label: name, object: OBJECT, dimensions, measures }); + +interface Cell { + id: 'sqlite' | 'pg'; + label: string; + env: string | null; + config: () => Record | null; +} + +const CELLS: readonly Cell[] = [ + { id: 'sqlite', label: 'sqlite', env: null, config: () => ({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }) }, + { + id: 'pg', + label: 'live postgres', + env: 'OS_TEST_POSTGRES_URL', + config: () => (process.env.OS_TEST_POSTGRES_URL ? { client: 'pg', connection: process.env.OS_TEST_POSTGRES_URL } : null), + }, +]; + +const quiet: any = { debug() {}, info() {}, warn() {}, error() {}, child() { return quiet; } }; + +interface Refusal extends Error { + code?: string; + status?: number; + member?: string; + param?: string; + field?: string; +} + +/** The error a call rejected with — and a loud failure if it resolved. */ +async function rejection(call: () => unknown): Promise { + let resolved: unknown; + try { + resolved = await call(); + } catch (e) { + return e as Refusal; + } + throw new Error(`expected a refusal, got ${JSON.stringify(resolved)}`); +} + +for (const cell of CELLS) { + const config = cell.config(); + describe.skipIf(!config)( + `the JSON-stored analytics door over a real SqlDriver — ${cell.label}${config ? '' : ` (skipped: set ${cell.env} to run this cell)`}`, + () => { + let driver: any; + let engine: ObjectQL; + let service: AnalyticsService; + /** Raw-SQL statements and engine aggregates that read THIS object. */ + const reads = { rawSql: 0, aggregate: 0 }; + + const dropTable = async () => { + if (cell.id === 'sqlite') return; + await driver?.execute(`drop table if exists ${OBJECT}`).catch(() => {}); + }; + + beforeAll(async () => { + driver = new SqlDriver(config as any); + await dropTable(); + engine = new ObjectQL({ logger: quiet } as any); + engine.registerDriver(driver, true); + await engine.init(); + engine.registry.registerObject(LEDGER as any); + await engine.syncSchemas(); + for (const row of ROWS) await engine.insert(OBJECT, { ...row } as any); + + const realExecute = (engine as any).execute.bind(engine); + (engine as any).execute = (sql: unknown, opts?: { object?: string }) => { + if (opts?.object === OBJECT) reads.rawSql += 1; + return realExecute(sql, opts); + }; + const realAggregate = engine.aggregate.bind(engine); + (engine as any).aggregate = (object: string, ...rest: unknown[]) => { + if (object === OBJECT) reads.aggregate += 1; + return (realAggregate as any)(object, ...rest); + }; + + // The plugin's own composition over the real engine: both auto-bridges. + const registered: Record = {}; + await new AnalyticsServicePlugin({ cubes: [CUBE as any] }).init({ + getService: (name: string) => (name === 'data' ? engine : registered[name]), + registerService: (name: string, svc: unknown) => { registered[name] = svc; }, + replaceService: (name: string, svc: unknown) => { registered[name] = svc; }, + hook: () => {}, + logger: quiet, + } as never); + service = registered.analytics as AnalyticsService; + // The scalar-and-multi-value dataset (row 1) and the scalar distinct count (row 3's control). + service.registerDataset(dataset('json_stored_door_dim', [{ name: 'n_rows', aggregate: 'count' }], [ + { name: 'picked', field: 'multi_sel', type: 'string' }, + { name: 'state', field: 'status', type: 'string' }, + ])); + service.registerDataset( + dataset('json_stored_door_distinct_ok', [{ name: 'n_distinct', aggregate: 'count_distinct', field: 'status' }]), + ); + }); + + afterAll(async () => { + await dropTable(); + try { await engine?.destroy(); } catch { /* noop */ } + }); + + it('row 1 — a cube and a dataset dimension on a multi-value field answer 400 INVALID_FIELD naming the member, before any statement', async () => { + const before = { ...reads }; + for (const member of ['tags_f', 'ms', 'multi_sel']) { + const err = await rejection(() => service.query({ cube: CUBE.name, measures: ['count'], dimensions: [member] })); + expect({ code: err.code, status: err.status, member: err.member, param: err.param }, member).toEqual({ + code: 'INVALID_FIELD', status: 400, member, param: 'dimensions', + }); + expect(err.message, member).toContain('a multi-value field, which analytics does not group by. The query was NOT run.'); + } + const err = await rejection(() => service.query({ cube: 'json_stored_door_dim', measures: ['n_rows'], dimensions: ['picked'] })); + expect({ code: err.code, status: err.status, member: err.member, field: err.field }).toEqual({ + code: 'INVALID_FIELD', status: 400, member: 'picked', field: 'multi_sel', + }); + expect(reads, 'no raw SQL and no engine aggregate for the object').toEqual(before); + }); + + it('row 1 CONTROL — a single-value select dimension is served: one group per value, counted', async () => { + const groups = async (cube: string, measure: string, member: string) => { + const result = await service.query({ cube, measures: [measure], dimensions: [member] }); + return (result.rows as Array>) + .map((r) => [String(r[member]), Number(r[measure])] as const) + .sort(([a], [b]) => a.localeCompare(b)); + }; + expect(await groups(CUBE.name, 'count', 'status')).toEqual([['a', 2], ['b', 2]]); + expect(await groups('json_stored_door_dim', 'n_rows', 'state')).toEqual([['a', 2], ['b', 2]]); + }); + + it('row 2 — an inferred FIELD_count_distinct over a JSON-stored field answers 400 INVALID_FIELD naming the measure, before any statement', async () => { + const before = { ...reads }; + for (const field of ['meta', 'tags_f', 'multi_sel']) { + const member = `${field}_count_distinct`; + for (const cube of [OBJECT, CUBE.name]) { + const err = await rejection(() => service.query({ cube, measures: [member] })); + expect({ code: err.code, status: err.status, member: err.member, param: err.param, field: err.field }, `${cube} ${member}`).toEqual({ + code: 'INVALID_FIELD', status: 400, member, param: 'measures', field, + }); + expect(err.message, `${cube} ${member}`).toContain('which analytics does not count distinct. The query was NOT run.'); + } + } + expect(reads, 'no raw SQL and no engine aggregate for the object').toEqual(before); + }); + + it('row 2 CONTROL — a count_distinct over a single-value select is served', async () => { + for (const cube of [OBJECT, CUBE.name]) { + const result = await service.query({ cube, measures: ['status_count_distinct'] }); + expect(Number((result.rows as Array>)[0].status_count_distinct), cube).toBe(2); + } + }); + + it('row 3 — a dataset count_distinct measure over a multiple: true select is refused DATASET_INVALID at registration; the scalar control registers and answers', async () => { + const before = { ...reads }; + const err = await rejection(() => service.registerDataset( + dataset('json_stored_door_distinct', [{ name: 'n_distinct', aggregate: 'count_distinct', field: 'multi_sel' }]), + )); + expect({ code: err.code, status: err.status }).toEqual({ code: 'DATASET_INVALID', status: 400 }); + expect(err.message).toContain('declares as `select` with `multiple: true`'); + expect(reads, 'no raw SQL and no engine aggregate for the object').toEqual(before); + + const result = await service.query({ cube: 'json_stored_door_distinct_ok', measures: ['n_distinct'] }); + expect(Number((result.rows as Array>)[0].n_distinct)).toBe(2); + }); + }, + ); +} diff --git a/packages/services/service-analytics/src/__tests__/multi-value-json-stored-door.test.ts b/packages/services/service-analytics/src/__tests__/multi-value-json-stored-door.test.ts new file mode 100644 index 00000000000..9edaa654fee --- /dev/null +++ b/packages/services/service-analytics/src/__tests__/multi-value-json-stored-door.test.ts @@ -0,0 +1,395 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The analytics door refuses what the engine's aggregate door refuses: a + * GROUPED dimension on a MULTI-VALUE field, and a `count_distinct` measure + * over a JSON-STORED field — `INVALID_FIELD` / 400, naming the member the + * caller wrote, before either strategy builds anything. A dataset measure's + * `count_distinct` over a multi-capable field declared `multiple: true` is + * refused at compile time beside the type row (`DATASET_INVALID` / 400). + * + * What these pins hold (the live-driver pins on SQLite and PostgreSQL are + * `json-stored-door-live-drivers.test.ts`; the HTTP pins for the dataset route + * are `packages/rest/src/analytics-dataset-multi-value-door.test.ts`): + * + * - one answer on BOTH strategy faces — `NativeSQLStrategy`, which compiled + * the `GROUP BY` / `COUNT(DISTINCT …)` itself and never reached the engine's + * door, and `ObjectQLStrategy`, which reached it and was refused there under + * the engine's position (`groupBy[0]`, `aggregations[0].field`); + * - the DECLARATION, not the type alone: a `select` (or `lookup`, `user`, + * `file`, `image`, `radio`) with `multiple: true` is judged like `tags`, and + * the same type without the flag is served; + * - no statement and no aggregate reaches the host bridges on a refusal; + * - GUARD: the judged population is exactly what the spec's predicates + * answer, over every `FieldType` with and without `multiple: true` — the + * predicates the engine's doors read, never a second list. + */ + +import { describe, it, expect, vi } from 'vitest'; +import type { Cube } from '@objectstack/spec/data'; +import { + FieldType, + STRUCTURED_JSON_TYPES, + isAggregateCompatibleWithFieldType, + isMultiValueField, +} from '@objectstack/spec/data'; +import { DatasetSchema, type Dataset } from '@objectstack/spec/ui'; +import { AnalyticsService } from '../analytics-service.js'; +import { compileDataset } from '../dataset-compiler.js'; + +const silentLogger = { + info: vi.fn(), + debug: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + child: vi.fn().mockReturnThis(), +} as any; + +const OBJECT = 'mv_ledger'; +const JOINED = 'mv_account'; + +type Declared = { type: string; multiple?: boolean }; + +/** The base object's fields: the scalar controls, every multi-value declaration, and a json field. */ +const FIELDS: Record = { + title: { type: 'text' }, + status: { type: 'select' }, + owner: { type: 'lookup' }, + account: { type: 'lookup' }, + tags_f: { type: 'tags' }, + ms: { type: 'multiselect' }, + boxes: { type: 'checkboxes' }, + multi_sel: { type: 'select', multiple: true }, + multi_radio: { type: 'radio', multiple: true }, + multi_lookup: { type: 'lookup', multiple: true }, + multi_user: { type: 'user', multiple: true }, + multi_file: { type: 'file', multiple: true }, + multi_image: { type: 'image', multiple: true }, + meta: { type: 'json' }, + // GUARD population: one field per `FieldType`, bare and flagged. + ...Object.fromEntries(FieldType.options.map((type) => [`f_${type}`, { type }])), + ...Object.fromEntries(FieldType.options.map((type) => [`m_${type}`, { type, multiple: true }])), +}; + +/** The joined object: a text column and a multi-value one. */ +const JOINED_FIELDS: Record = { + name: { type: 'text' }, + labels: { type: 'tags' }, +}; + +/** Every declaration this card judges, as `[field, how it is declared]`. */ +const MULTI_VALUE: ReadonlyArray = [ + ['tags_f', 'tags'], + ['ms', 'multiselect'], + ['boxes', 'checkboxes'], + ['multi_sel', 'select with multiple: true'], + ['multi_radio', 'radio with multiple: true'], + ['multi_lookup', 'lookup with multiple: true'], + ['multi_user', 'user with multiple: true'], + ['multi_file', 'file with multiple: true'], + ['multi_image', 'image with multiple: true'], +]; + +/** An authored cube: one dimension per field, a key over another column, and three authored distinct counts. */ +const LEDGER_CUBE: Cube = { + name: 'mv_cube', + title: 'MV ledger', + sql: OBJECT, + public: true, + measures: { + count: { label: 'Rows', type: 'count', sql: '*' }, + distinct_labels: { label: 'Distinct labels', type: 'count_distinct', sql: 'multi_sel' }, + distinct_status: { label: 'Distinct status', type: 'count_distinct', sql: 'status' }, + // `count` over a multi-value column compares nothing: not judged. + rows_with_tags: { label: 'Rows with tags', type: 'count', sql: 'tags_f' }, + }, + dimensions: { + picked: { label: 'Picked', type: 'string', sql: 'multi_sel' }, + ...Object.fromEntries(Object.keys(FIELDS).map((f) => [f, { label: f, type: 'string' as const, sql: f }])), + }, +} as Cube; + +const LEDGER_DATASET = { + name: 'mv_ds', + label: 'MV dataset', + object: OBJECT, + include: ['account'], + dimensions: [ + { name: 'status_dim', field: 'status', type: 'string' }, + { name: 'picked_dim', field: 'multi_sel', type: 'string' }, + { name: 'tag_dim', field: 'tags_f', type: 'string' }, + { name: 'acct_name', field: 'account.name', type: 'string' }, + { name: 'acct_labels', field: 'account.labels', type: 'string' }, + ], + measures: [ + { name: 'row_count', aggregate: 'count' }, + { name: 'acct_label_kinds', aggregate: 'count_distinct', field: 'account.labels' }, + ], +} as unknown as Dataset; + +type Face = 'native' | 'objectql'; + +interface Refusal extends Error { + code?: string; + status?: number; + member?: string; + param?: string; + field?: string; + object?: string; + cube?: string; +} + +function makeService(face: Face) { + const calls = { raw: [] as string[], aggregate: [] as unknown[] }; + const service = new AnalyticsService({ + logger: silentLogger, + cubes: [LEDGER_CUBE], + queryCapabilities: () => ({ nativeSql: face === 'native', objectqlAggregate: face === 'objectql', inMemory: false }), + executeRawSql: async (_object: string, sql: string) => { + calls.raw.push(sql); + return [{ status: 'a', count: 2 }]; + }, + executeAggregate: async (_object: string, options: unknown) => { + calls.aggregate.push(options); + return [{ status: 'a', count: 2 }]; + }, + isRegisteredObject: (n: string) => n === OBJECT, + getObjectFieldNames: (n: string) => (n === OBJECT ? Object.keys(FIELDS) : undefined), + sourceFieldMeta: (o: string, f: string) => (o === OBJECT ? FIELDS[f] : o === JOINED ? JOINED_FIELDS[f] : undefined), + relationshipResolver: (o: string, rel: string) => (o === OBJECT && rel === 'account' ? JOINED : undefined), + }); + return { service, calls }; +} + +/** The error a call rejected with — and a loud failure if it resolved. */ +async function rejection(call: Promise): Promise { + let resolved: unknown; + try { + resolved = await call; + } catch (e) { + return e as Refusal; + } + throw new Error(`expected a refusal, got ${JSON.stringify(resolved)}`); +} + +/** The ADR-0112 envelope plus the member the caller wrote. */ +function envelopeOf(err: Refusal) { + return { code: err.code, status: err.status, member: err.member, param: err.param, field: err.field, object: err.object }; +} + +describe('row 1 — a grouped dimension on a multi-value field is refused at the analytics door, on both faces', () => { + for (const face of ['native', 'objectql'] as const) { + it(`${face}: every multi-value declaration answers INVALID_FIELD / 400 naming the member — no statement, no aggregate`, async () => { + const { service, calls } = makeService(face); + for (const [field, declared] of MULTI_VALUE) { + const err = await rejection(service.query({ cube: 'mv_cube', measures: ['count'], dimensions: [field] })); + expect(envelopeOf(err), field).toEqual({ + code: 'INVALID_FIELD', status: 400, member: field, param: 'dimensions', field, object: OBJECT, + }); + expect(err.message, field).toContain( + `Dimension '${field}' on cube 'mv_cube' groups by field '${field}', which object '${OBJECT}' declares as ${declared} ` + + '— a multi-value field, which analytics does not group by. The query was NOT run.', + ); + } + expect(calls.raw, 'no statement reached the raw-SQL bridge').toEqual([]); + expect(calls.aggregate, 'no aggregate reached the engine bridge').toEqual([]); + }); + + it(`${face}: CONTROL the same types without multiple: true, and the text column, are served`, async () => { + const { service, calls } = makeService(face); + for (const field of ['status', 'owner', 'title']) { + await service.query({ cube: 'mv_cube', measures: ['count'], dimensions: [field] }); + } + expect(calls.raw.length + calls.aggregate.length).toBe(3); + }); + } + + it('names the member as the caller spelled it: a cube key over another column, the cube-qualified spelling, an ad-hoc cube', async () => { + const { service, calls } = makeService('native'); + const keyed = await rejection(service.query({ cube: 'mv_cube', measures: ['count'], dimensions: ['title', 'picked'] })); + expect(envelopeOf(keyed)).toEqual({ code: 'INVALID_FIELD', status: 400, member: 'picked', param: 'dimensions', field: 'multi_sel', object: OBJECT }); + const qualified = await rejection(service.query({ cube: 'mv_cube', measures: ['count'], dimensions: ['mv_cube.picked'] })); + expect(envelopeOf(qualified)).toMatchObject({ code: 'INVALID_FIELD', status: 400, member: 'mv_cube.picked', field: 'multi_sel' }); + const adhoc = await rejection(service.query({ cube: OBJECT, measures: ['count'], dimensions: ['multi_sel'] })); + expect(envelopeOf(adhoc)).toEqual({ code: 'INVALID_FIELD', status: 400, member: 'multi_sel', param: 'dimensions', field: 'multi_sel', object: OBJECT }); + expect(calls.raw).toEqual([]); + }); + + it('a bucketed time dimension on a multi-value field is refused under timeDimensions', async () => { + const { service, calls } = makeService('native'); + const err = await rejection(service.query({ + cube: OBJECT, + measures: ['count'], + timeDimensions: [{ dimension: 'tags_f', granularity: 'month' }], + })); + expect(envelopeOf(err)).toEqual({ + code: 'INVALID_FIELD', status: 400, member: 'tags_f', param: 'timeDimensions', field: 'tags_f', object: OBJECT, + }); + expect(err.message).toContain(`Time dimension 'tags_f' on cube '${OBJECT}' buckets field 'tags_f'`); + expect(calls.raw).toEqual([]); + }); + + it('a dataset dimension is refused naming the dataset dimension — and over an included relationship, on the JOINED object', async () => { + const { service, calls } = makeService('native'); + const picked = await rejection(service.queryDataset(LEDGER_DATASET, { measures: ['row_count'], dimensions: ['picked_dim'] })); + expect(envelopeOf(picked)).toEqual({ code: 'INVALID_FIELD', status: 400, member: 'picked_dim', param: 'dimensions', field: 'multi_sel', object: OBJECT }); + const joined = await rejection(service.queryDataset(LEDGER_DATASET, { measures: ['row_count'], dimensions: ['acct_labels'] })); + expect(envelopeOf(joined)).toEqual({ code: 'INVALID_FIELD', status: 400, member: 'acct_labels', param: 'dimensions', field: 'account.labels', object: JOINED }); + expect(joined.message).toContain(`whose column 'labels' the joined object '${JOINED}' declares as tags`); + expect(joined.message).toContain(`a record query on '${JOINED}' with where { "labels": { "$contains": VALUE } }`); + expect(calls.raw).toEqual([]); + // CONTROL the scalar dataset dimensions are served. + await service.queryDataset(LEDGER_DATASET, { measures: ['row_count'], dimensions: ['status_dim'] }); + await service.queryDataset(LEDGER_DATASET, { measures: ['row_count'], dimensions: ['acct_name'] }); + expect(calls.raw).toHaveLength(2); + }); + + it('the dry-run door refuses before building the statement', async () => { + const { service } = makeService('native'); + const err = await rejection(service.generateSql({ cube: 'mv_cube', measures: ['count'], dimensions: ['multi_sel'] })); + expect(envelopeOf(err)).toMatchObject({ code: 'INVALID_FIELD', status: 400, member: 'multi_sel' }); + }); +}); + +describe('row 2 — a count_distinct measure over a JSON-stored field is refused at the analytics door, on both faces', () => { + for (const face of ['native', 'objectql'] as const) { + it(`${face}: an inferred FIELD_count_distinct over json, a multi-option type and a multiple: true select — ad-hoc and authored cube alike`, async () => { + const { service, calls } = makeService(face); + for (const cube of [OBJECT, 'mv_cube']) { + for (const [field, declared, kind] of [ + ['meta', 'json', 'a structured-JSON value'], + ['tags_f', 'tags', 'a multi-value field'], + ['multi_sel', 'select with multiple: true', 'a multi-value field'], + ] as const) { + const member = `${field}_count_distinct`; + const err = await rejection(service.query({ cube, measures: [member] })); + expect(envelopeOf(err), `${cube} ${member}`).toEqual({ + code: 'INVALID_FIELD', status: 400, member, param: 'measures', field, object: OBJECT, + }); + expect(err.message, `${cube} ${member}`).toContain( + `Measure '${member}' on cube '${cube}' counts distinct field '${field}', which object '${OBJECT}' declares as ${declared} ` + + `— ${kind}, which analytics does not count distinct. The query was NOT run.`, + ); + } + } + expect(calls.raw, 'no statement reached the raw-SQL bridge').toEqual([]); + expect(calls.aggregate, 'no aggregate reached the engine bridge').toEqual([]); + }); + + it(`${face}: an AUTHORED count_distinct measure over a multiple: true select is refused naming the measure`, async () => { + const { service, calls } = makeService(face); + const err = await rejection(service.query({ cube: 'mv_cube', measures: ['count', 'distinct_labels'] })); + expect(envelopeOf(err)).toEqual({ + code: 'INVALID_FIELD', status: 400, member: 'distinct_labels', param: 'measures', field: 'multi_sel', object: OBJECT, + }); + expect(err.message).toContain('Count the records that hold one member instead'); + expect(calls.raw.length + calls.aggregate.length).toBe(0); + }); + + it(`${face}: CONTROL a scalar count_distinct and a count over a multi-value column are served`, async () => { + const { service, calls } = makeService(face); + await service.query({ cube: OBJECT, measures: ['status_count_distinct'] }); + await service.query({ cube: 'mv_cube', measures: ['distinct_status', 'rows_with_tags'] }); + expect(calls.raw.length + calls.aggregate.length).toBe(2); + }); + } + + it('a dataset count_distinct over an included relationship\'s multi-value field is refused on the JOINED object at query time', async () => { + const { service, calls } = makeService('native'); + const err = await rejection(service.queryDataset(LEDGER_DATASET, { measures: ['acct_label_kinds'], dimensions: ['status_dim'] })); + expect(envelopeOf(err)).toEqual({ + code: 'INVALID_FIELD', status: 400, member: 'acct_label_kinds', param: 'measures', field: 'account.labels', object: JOINED, + }); + expect(calls.raw).toEqual([]); + }); + + it('the structured-JSON route is the scalar one, the multi-value route is the membership filter', async () => { + const { service } = makeService('native'); + const json = await rejection(service.query({ cube: OBJECT, measures: ['meta_count_distinct'] })); + expect(json.message).toContain('Count distinct values of a field that stores one scalar value'); + expect(json.message).not.toContain('$contains'); + const multi = await rejection(service.query({ cube: OBJECT, measures: ['multi_sel_count_distinct'] })); + expect(multi.message).toContain( + `Count the records that hold one member instead: count with where { "multi_sel": { "$contains": VALUE } } in a record query on '${OBJECT}'`, + ); + }); +}); + +describe('row 3 — a dataset count_distinct measure over a multiple: true field is refused at compile time, by the declaration', () => { + const dataset = (field: string) => DatasetSchema.parse({ + name: 'mv_distinct', label: 'MV distinct', object: OBJECT, + dimensions: [], measures: [{ name: 'n_distinct', aggregate: 'count_distinct', field }], + }); + const shapeOf = (_o: string, f: string) => { + const d = FIELDS[f]; + return d ? { type: d.type, multiple: d.multiple === true } : undefined; + }; + + it('compileDataset reads the declaration: every multiple: true declaration answers DATASET_INVALID / 400 and names the flag', () => { + for (const [field, declared] of MULTI_VALUE) { + let err: Refusal | undefined; + try { + compileDataset(dataset(field), undefined, { declaredValueShape: shapeOf }); + } catch (e) { + err = e as Refusal; + } + expect(err, field).toBeDefined(); + expect({ code: err!.code, status: err!.status }, field).toEqual({ code: 'DATASET_INVALID', status: 400 }); + expect(err!.message, field).toContain(`which object "${OBJECT}" declares as \`${declared.replace(' with multiple: true', '')}\``); + if (declared.endsWith('with multiple: true')) expect(err!.message, field).toContain('with `multiple: true`'); + expect(err!.message, field).toContain('COMPARES the stored values for equality'); + } + }); + + it('CONTROL the same types without the flag compile', () => { + for (const field of ['status', 'owner']) { + expect(compileDataset(dataset(field), undefined, { declaredValueShape: shapeOf }).cube.measures.n_distinct?.type, field) + .toBe('count_distinct'); + } + }); + + it('through the service: registration and the request door both refuse, before any statement', async () => { + const { service, calls } = makeService('native'); + const registered = await rejection(Promise.resolve().then(() => service.registerDataset(dataset('multi_sel')))); + expect({ code: registered.code, status: registered.status }).toEqual({ code: 'DATASET_INVALID', status: 400 }); + const requested = await rejection(service.queryDataset(dataset('multi_sel'), { measures: ['n_distinct'] })); + expect({ code: requested.code, status: requested.status }).toEqual({ code: 'DATASET_INVALID', status: 400 }); + expect(calls.raw).toEqual([]); + // CONTROL a single-value select is served. + await service.queryDataset(dataset('status'), { measures: ['n_distinct'] }); + expect(calls.raw).toHaveLength(1); + }); +}); + +describe('GUARD the judged population is the spec predicates\', over every FieldType, bare and with multiple: true', () => { + it('a grouped dimension is refused exactly when STRUCTURED_JSON_TYPES or isMultiValueField says so', async () => { + const { service } = makeService('native'); + for (const type of FieldType.options) { + for (const multiple of [false, true]) { + const field = `${multiple ? 'm' : 'f'}_${type}`; + const verdict = await service.generateSql({ cube: OBJECT, measures: ['count'], dimensions: [field] }).then( + () => null, + (e: Refusal) => ({ code: e.code, status: e.status, member: e.member }), + ); + const judged = STRUCTURED_JSON_TYPES.has(type) || isMultiValueField({ type, multiple }); + expect(verdict, field).toEqual(judged ? { code: 'INVALID_FIELD', status: 400, member: field } : null); + } + } + }); + + it('a count_distinct measure is refused exactly when the table\'s row refuses the type or isMultiValueField says so', async () => { + const { service } = makeService('native'); + for (const type of FieldType.options) { + for (const multiple of [false, true]) { + const field = `${multiple ? 'm' : 'f'}_${type}`; + const member = `${field}_count_distinct`; + const verdict = await service.generateSql({ cube: OBJECT, measures: [member] }).then( + () => null, + (e: Refusal) => ({ code: e.code, status: e.status, member: e.member }), + ); + const judged = !isAggregateCompatibleWithFieldType('count_distinct', type) || isMultiValueField({ type, multiple }); + expect(verdict, field).toEqual(judged ? { code: 'INVALID_FIELD', status: 400, member } : null); + } + } + }); +}); diff --git a/packages/services/service-analytics/src/analytics-service.ts b/packages/services/service-analytics/src/analytics-service.ts index 524f09364e1..c4965ee89bc 100644 --- a/packages/services/service-analytics/src/analytics-service.ts +++ b/packages/services/service-analytics/src/analytics-service.ts @@ -1884,8 +1884,11 @@ export class AnalyticsService implements IAnalyticsService { // the result-column enrichment already reads. Same source, one call // shape, so a host that wired `sourceFieldMeta` gets the compile-time // refusal with no second hook to remember. - declaredFieldType: (object: string, field: string) => - this.sourceFieldMeta?.(object, field)?.type, + declaredValueShape: (object: string, field: string) => { + const meta = this.sourceFieldMeta?.(object, field); + if (typeof meta?.type !== 'string' || meta.type === '') return undefined; + return { type: meta.type, multiple: meta.multiple === true }; + }, }); } @@ -2787,7 +2790,11 @@ export class AnalyticsService implements IAnalyticsService { if (!entry) return member; return typeof entry.sql === 'string' ? entry.sql : ''; }, - (o, field) => fieldMeta(o, field)?.type, + (o, field) => { + const meta = fieldMeta(o, field); + if (typeof meta?.type !== 'string' || meta.type === '') return undefined; + return { type: meta.type, multiple: meta.multiple === true }; + }, ); } diff --git a/packages/services/service-analytics/src/dataset-compiler.ts b/packages/services/service-analytics/src/dataset-compiler.ts index e0c76a3bd14..67ecd94ac48 100644 --- a/packages/services/service-analytics/src/dataset-compiler.ts +++ b/packages/services/service-analytics/src/dataset-compiler.ts @@ -1,10 +1,12 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. import type { Cube, Metric, Dimension as CubeDimension, CubeJoin } from '@objectstack/spec/data'; +import type { ValueShapeFieldDef } from '@objectstack/spec/data'; import { AGGREGATE_FIELD_TYPE_COMPATIBILITY, AggregationFunction, isAggregateCompatibleWithFieldType, + isMultiValueField, } from '@objectstack/spec/data'; import type { Dataset, DatasetMeasure, DatasetDimension } from '@objectstack/spec/ui'; import { resolveI18nLabel } from '@objectstack/spec/ui'; @@ -159,10 +161,11 @@ export interface DatasetCompileOptions { */ isExternalObject?: (objectName: string) => boolean; /** - * [#16737 / #16099] The DECLARED `FieldType` of `field` on `objectName`, or - * `undefined` when nothing authoritative can answer. Supplied by the host - * from the same `AnalyticsServiceConfig.sourceFieldMeta` the result-column - * enrichment reads. + * [#16737 / #16099] The DECLARATION of `field` on `objectName` — its + * `FieldType` and its `multiple` flag — or `undefined` when nothing + * authoritative can answer. Supplied by the host from the same + * `AnalyticsServiceConfig.sourceFieldMeta` the result-column enrichment + * reads, in the shape the strategies' `declaredValueShape` hook carries. * * It is what makes {@link assertAggregateFieldTypeCompatible} decidable at * COMPILE time — the compile leg of the director ruling (decision batch #59, @@ -170,8 +173,14 @@ export interface DatasetCompileOptions { * `AGGREGATE_FIELD_TYPE_COMPATIBILITY` in `@objectstack/spec`. Absent hook, * unknown object, unknown field → the pair is not judged, matching every * other probe on this interface. + * + * [#20912] It carries the declaration, not the type alone, because the + * table is per TYPE and cannot see `multiple: true`: a `select` flagged + * `multiple` is a list stored as JSON, and `count_distinct` over it splits + * across backends exactly as over `tags`. This option was + * `declaredFieldType` (the type alone) until then. */ - declaredFieldType?: (objectName: string, field: string) => string | undefined; + declaredValueShape?: (objectName: string, field: string) => ValueShapeFieldDef | undefined; } /** Map a dataset measure's aggregate to the Cube metric `type`. */ @@ -217,15 +226,17 @@ function aggregateToMetricType(m: DatasetMeasure): Metric['type'] { * does not exist on PostgreSQL). * - [#20808] `count_distinct` COMPARES the stored values for equality, and * reaches this sentence only over a JSON-stored field (the table's - * `count_distinct` row refuses the structured-JSON and multi-option types): - * the in-memory driver counted equal documents apart, SQLite compared the - * serialized text, and PostgreSQL has no equality operator for `json`. + * `count_distinct` row refuses the structured-JSON and multi-option types; + * [#20912] `isMultiValueField` refuses a multi-capable type declared + * `multiple: true`, and `column` then names the flag): the in-memory driver + * counted equal documents apart, SQLite compared the serialized text, and + * PostgreSQL has no equality operator for `json`. * * `count` accepts every type and never reaches this sentence. */ -const DIVERGENCE_BY_AGGREGATE = (aggregate: string, fieldType: string): string => +const DIVERGENCE_BY_AGGREGATE = (aggregate: string, fieldType: string, column = `\`${fieldType}\` column`): string => aggregate === 'count_distinct' - ? `"${aggregate}" COMPARES the stored values for equality, so over a \`${fieldType}\` column ` + ? `"${aggregate}" COMPARES the stored values for equality, so over a ${column} ` + 'the answer is decided by how each backend compares a JSON-stored value rather than by ' + 'the data — one counts every row apart, one compares the serialized text, another has no ' + 'equality for the type and fails at query time — and one dataset would mean two things ' @@ -362,7 +373,7 @@ const REMEDY_BY_SOURCE_CLASS = (fieldType: string, aggregate: string): string => * * ## Tiering — "cannot answer, do not block", the same as every sibling probe * - * - No `declaredFieldType` hook (no data engine wired) → not judged. + * - No `declaredValueShape` hook (no data engine wired) → not judged. * - A field the hook cannot resolve → not judged. * - A RELATIONSHIP-PATH field (`account.closed_at`) → not judged. The hook * resolves a column on the BASE object, so it would answer about a different @@ -379,32 +390,45 @@ function assertAggregateFieldTypeCompatible( datasetName: string, objectName: string, measure: DatasetMeasure, - declaredFieldType?: (objectName: string, field: string) => string | undefined, + declaredValueShape?: (objectName: string, field: string) => ValueShapeFieldDef | undefined, ): void { - if (!declaredFieldType) return; + if (!declaredValueShape) return; const aggregate = measure.aggregate; const field = measure.field; if (!aggregate || !field) return; // A dotted reference resolves on a JOINED object; this hook answers for the // base one. Not judged rather than judged wrongly. if (field.includes('.')) return; - const fieldType = declaredFieldType(objectName, field); - if (!fieldType) return; + const shape = declaredValueShape(objectName, field); + if (!shape?.type) return; + const fieldType = shape.type; + // [#20912] …and the DECLARATION half the per-type table cannot see, asked + // the way the engine's `count_distinct` door asks it: `isMultiValueField` + // beside the table's row. A multi-capable type flagged `multiple: true` + // (`select`, `lookup`, …) is a list stored as JSON — the very storage the + // row refuses `tags` for. Only `count_distinct` can be moved by it: the + // `sum` / `avg` / `min` / `max` rows accept no multi-capable type, and + // `count` reads no value. + const flaggedList = aggregate === 'count_distinct' + && isMultiValueField(shape) + && isAggregateCompatibleWithFieldType(aggregate, fieldType); // [#17560] Every aggregate is judged, through this one door. ⛔ There is no // scope condition here any more — the VERDICT is the spec table's and only - // the spec table's. - if (isAggregateCompatibleWithFieldType(aggregate, fieldType)) return; + // the spec table's, plus the declaration above. + if (isAggregateCompatibleWithFieldType(aggregate, fieldType) && !flaggedList) return; const accepted = AGGREGATE_FIELD_TYPE_COMPATIBILITY[aggregate]; + const declared = flaggedList ? `\`${fieldType}\` with \`multiple: true\`` : `\`${fieldType}\``; // [#5716] `DATASET_INVALID` / 400 — a verdict about the dataset DOCUMENT, // decided from metadata alone before any query runs, and fixable only by the // author who wrote the pair. throw datasetInvalidError( `[dataset-compiler] dataset "${datasetName}" measure "${measure.name}" applies aggregate ` + `"${aggregate}" to field "${field}", which object "${objectName}" declares as ` + - `\`${fieldType}\`. That pair is not accepted: ` + - `${DIVERGENCE_BY_AGGREGATE(aggregate, fieldType)}` + - `"${aggregate}" accepts: ${accepted.join(', ')}. ` + + `${declared}. That pair is not accepted: ` + + `${DIVERGENCE_BY_AGGREGATE(aggregate, fieldType, flaggedList ? `${declared} column` : undefined)}` + + `"${aggregate}" accepts: ${accepted.join(', ')}` + + `${flaggedList ? ', none of them with `multiple: true`' : ''}. ` + `${REMEDY_BY_SOURCE_CLASS(fieldType, aggregate)}`, ); } @@ -686,7 +710,7 @@ export function compileDataset( // declared type can carry. Placed after the join-declaration check so a // dotted field is refused for the reason it is actually wrong (an // undeclared relationship) before this gate stands down on it. - assertAggregateFieldTypeCompatible(dataset.name, dataset.object, m, options?.declaredFieldType); + assertAggregateFieldTypeCompatible(dataset.name, dataset.object, m, options?.declaredValueShape); // Filed under its name (`measures[m.name]` below), with no inner copy — // the same #20300 retirement as the dimension's. const metric: Metric = { diff --git a/packages/services/service-analytics/src/structured-json-dimension-door.ts b/packages/services/service-analytics/src/structured-json-dimension-door.ts index 47ce2b2acc1..1915a5a4205 100644 --- a/packages/services/service-analytics/src/structured-json-dimension-door.ts +++ b/packages/services/service-analytics/src/structured-json-dimension-door.ts @@ -4,7 +4,10 @@ * [#20807] A GROUPED dimension on a STRUCTURED-JSON field — `json`, * `composite`, `repeater`, `record`, `location`, `address`, `vector` — is * refused `INVALID_FIELD` / 400 at the analytics door, naming the member the - * caller wrote, before either strategy builds anything. + * caller wrote, before either strategy builds anything. [#20912] So is a + * grouped dimension on a MULTI-VALUE field, and a `count_distinct` measure + * over a JSON-stored field: every JSON-stored column the engine's aggregate + * door refuses is refused here, ahead of the strategy that bypasses it. * * ## What ran before this door, measured on `origin/main` `793fb839` * @@ -60,6 +63,54 @@ * whose `sql` is not a bare object name, an expression `sql`, a dotted path * the cube declares no join for (a synthetic traversal: which object it lands * on is the strategy's guess, not a declaration), and every other field type. + * The same tiers hold for the two #20912 judgements below. + * + * ## [#20912] The engine's second class, and its second door + * + * #20808 taught the engine's aggregate door two more refusals, and the native + * strategy bypassed both the same way. Measured on `origin/main` `8d329f02` + * through this service as `AnalyticsServicePlugin` composes it on a real + * engine, four rows (`tags` [a,b], [a,b], [b], [ab]; a `multiselect` and a + * `select` with `multiple: true` [a,b], [a,b], [b], [b]; a `json` document + * per row, two of them equal): + * + * | query | SQLite | PostgreSQL 16.13 | `ObjectQLStrategy` | + * |:--|:--|:--|:--| + * | a cube or dataset dimension on a multi-value field | 200, one group per serialized array | **500 `DATABASE_ERROR`** | 400, the engine's `groupBy[0]` | + * | an inferred `FIELD_count_distinct` over `json` / `tags` / the `multiple: true` select | 200, 2 / 3 / 2 | **500** | 400, the engine's `aggregations[0].field` | + * + * So the door judges both, ahead of both strategies, in the engine's words + * with the member the caller wrote: + * + * - A GROUPED member whose column is a MULTI-VALUE field — + * {@link isMultiValueField}, which reads the declaration: an inherently + * multi option type (`multiselect`, `checkboxes`, `tags`) or a + * multi-capable type flagged `multiple: true` (`select`, `radio`, + * `lookup`, `user`, `file`, `image`). The same type without the flag + * stores one scalar value and is served. + * - A `measures` entry that resolves to a `count_distinct` measure whose + * column is JSON-STORED: the compatibility table's `count_distinct` row + * refuses the type (`@objectstack/spec/data`'s + * `isAggregateCompatibleWithFieldType`), or `isMultiValueField` says the + * declaration is a list. The member resolves the way the strategies resolve + * a measure (cube-qualified, flattened), and its column the way a dimension's + * does (`columnOf`, a declared join included). An authored measure, a + * suffix-inferred one (`tags_count_distinct`) and a compiled dataset's are + * one population here; a `count` measure compares nothing and is not judged. + * + * ⛔ Every predicate is the spec's — the ones the engine's two doors read — + * and the declaration arrives as `{ type, multiple }` + * (`ValueShapeFieldDef`), never the type alone: a type-only reading would + * refuse `tags` and serve a `select` with `multiple: true`, which is the same + * JSON column. + * + * The multi-value route is a RECORD query's membership filter, named on the + * object that declares the column: `$contains` over a JSON-stored list is + * lowered as membership by the engine on every driver. ⚠️ It is not this + * service's own `where`: measured on the same base, `NativeSQLStrategy` + * compiles that operator to `LIKE` — a substring match on SQLite (`[ab]` + * counts as holding `a`) and a 500 on PostgreSQL (`json ~~ text`) — so a route + * through it would trade one wrong answer for another. * * ## The envelope * @@ -72,13 +123,26 @@ * verdict — and `object` is the object that declares it. * * @see https://github.com/objectstack-ai/objectstack/issues/20807 + * @see https://github.com/objectstack-ai/objectstack/issues/20912 */ -import type { Cube } from '@objectstack/spec/data'; -import { STRUCTURED_JSON_TYPES } from '@objectstack/spec/data'; +import type { Cube, ValueShapeFieldDef } from '@objectstack/spec/data'; +import { + FieldType, + STRUCTURED_JSON_TYPES, + isAggregateCompatibleWithFieldType, + isMultiValueField, +} from '@objectstack/spec/data'; import type { AnalyticsQuery } from '@objectstack/spec/contracts'; +import type { AnalyticsRequestKey } from './dataset-refusal.js'; import { invalidMemberError } from './dataset-refusal.js'; +/** The declared `FieldType` vocabulary — the only types the compatibility table can answer for. */ +const DECLARED_FIELD_TYPES: ReadonlySet = new Set(FieldType.options); + +/** Which JSON-stored class a column belongs to, as the engine's two doors name them. */ +type JsonStoredClass = 'structured-json' | 'multi-value'; + /** One grouped member, tagged with the request key that carried it. */ interface GroupedMember { readonly member: string; @@ -129,16 +193,93 @@ function columnOf(cube: Cube, baseObject: string, sql: string): DimensionColumn return typeof joined === 'string' && joined !== '' ? { object: joined, column, path } : null; } +/** + * The class of a column a member GROUPS by, or `null` when it stores one + * scalar value — the engine's `groupBy` door's two predicates, in its order: + * {@link STRUCTURED_JSON_TYPES}, then {@link isMultiValueField} (the + * declaration, `multiple` included). + */ +function groupedClassOf(shape: ValueShapeFieldDef): JsonStoredClass | null { + if (STRUCTURED_JSON_TYPES.has(shape.type)) return 'structured-json'; + return isMultiValueField(shape) ? 'multi-value' : null; +} + +/** + * [#20912] The class of a column a measure COUNTS DISTINCT, or `null` when + * every backend compares its values alike — the engine's `count_distinct` + * door's two predicates: the compatibility table's `count_distinct` row + * ({@link isAggregateCompatibleWithFieldType}, the TYPE half) beside + * {@link isMultiValueField} (the DECLARATION half, which the per-type row + * cannot see). A type outside `FieldType` (a driver-internal alias) is not + * judged: the table is fail-closed on vocabulary, and "cannot answer, do not + * block" is this consumer's tier. + */ +function distinctClassOf(shape: ValueShapeFieldDef): JsonStoredClass | null { + if (!DECLARED_FIELD_TYPES.has(shape.type)) return null; + const multiValue = isMultiValueField(shape); + if (isAggregateCompatibleWithFieldType('count_distinct', shape.type) && !multiValue) return null; + return multiValue ? 'multi-value' : 'structured-json'; +} + +/** The declaration as the words say it: the type, and the flag when the field carries it. */ +function declaredAs(shape: ValueShapeFieldDef): string { + return shape.multiple === true ? `${shape.type} with multiple: true` : shape.type; +} + +/** Who declares the column, as the words say it: the cube's own object, or the joined one. */ +function declarerOf(target: DimensionColumn): string { + return target.path === target.column + ? `which object '${target.object}' declares` + : `whose column '${target.column}' the joined object '${target.object}' declares`; +} + +/** + * [#20912] The cube measure a `measures` entry resolves to — + * `NativeSQLStrategy.lookupMember(…, 'measure')`'s resolution, which + * `ObjectQLStrategy`'s mirrors: the key itself, then for a dotted entry the + * key after its first segment (the `.` qualifier, or the legacy second + * segment), then the underscore-flattened key. A measure has no synthetic + * relation tier, so nothing else resolves. Own keys only. + */ +function measureEntryOf(cube: Cube, member: string): { type?: unknown; sql?: unknown } | undefined { + const bag = (cube.measures ?? {}) as Record; + const own = (key: string) => (Object.prototype.hasOwnProperty.call(bag, key) ? bag[key] : undefined); + if (own(member)) return own(member); + if (!member.includes('.')) return undefined; + return own(member.split('.').slice(1).join('.')) ?? own(member.replace(/\./g, '_')); +} + +/** The ADR-0112 envelope every refusal of this door carries, naming the member and the column. */ +function refusal( + message: string, + member: string, + param: AnalyticsRequestKey, + cube: Cube, + target: DimensionColumn, +): Error { + const err = invalidMemberError(message, { member, param, cube: cube.name }) as Error & { + field?: string; + object?: string; + }; + err.field = target.path; + err.object = target.object; + return err; +} + /** * Refuse the first grouped member of `query` whose column is a declared - * structured-JSON field — `INVALID_FIELD` / 400. See the module header. + * structured-JSON (#20807) or multi-value (#20912) field, then the first + * `count_distinct` measure whose column is a declared JSON-stored field + * (#20912) — `INVALID_FIELD` / 400. See the module header. * * @param baseObject - The object `cube.sql` names (the caller has checked it * is a bare object name). * @param sqlOf - The dimension `sql` a member resolves to, or the member itself * when the cube declares none — the strategies' own lookup. - * @param declaredFieldType - The declared `FieldType` of a column on an - * object, or `undefined` when nothing authoritative answers. + * @param declaredValueShape - The DECLARATION of a column on an object — its + * `FieldType` and its `multiple` flag — or `undefined` when nothing + * authoritative answers. The flag is half the verdict: a `select` with + * `multiple: true` is a list stored as JSON, whatever its type name says. * * The words put the verdict first, then that the query did not run, then the * route, then the reason: a door that bounds a 4xx message keeps the front of @@ -149,30 +290,67 @@ export function assertNoStructuredJsonDimension( cube: Cube, baseObject: string, sqlOf: (member: string) => string, - declaredFieldType: (object: string, field: string) => string | undefined, + declaredValueShape: (object: string, field: string) => ValueShapeFieldDef | undefined, ): void { for (const { member, param } of groupedMembers(query)) { const target = columnOf(cube, baseObject, sqlOf(member)); if (!target) continue; - const type = declaredFieldType(target.object, target.column); - if (typeof type !== 'string' || !STRUCTURED_JSON_TYPES.has(type)) continue; + const shape = declaredValueShape(target.object, target.column); + if (typeof shape?.type !== 'string') continue; + const cls = groupedClassOf(shape); + if (cls === null) continue; const kind = param === 'timeDimensions' ? 'Time dimension' : 'Dimension'; const verb = param === 'timeDimensions' ? 'buckets' : 'groups by'; - const declarer = target.path === target.column - ? `which object '${target.object}' declares` - : `whose column '${target.column}' the joined object '${target.object}' declares`; - const err = invalidMemberError( - `${kind} '${member}' on cube '${cube.name}' ${verb} field '${target.path}', ${declarer} ` - + `as ${type} — a structured-JSON value, which analytics does not group by. ` - + 'The query was NOT run. Group by a field that stores one scalar value: store the part you ' - + 'group on in a field of its own and group by that field. A JSON document is no group key ' - + 'the SQL dialects share: one grouped each serialized document apart, another refused the ' - + 'statement.', - { member, param, cube: cube.name }, - ) as Error & { field?: string; object?: string }; - err.field = target.path; - err.object = target.object; - throw err; + const head = `${kind} '${member}' on cube '${cube.name}' ${verb} field '${target.path}', ${declarerOf(target)} ` + + `as ${declaredAs(shape)} — `; + throw refusal( + cls === 'structured-json' + ? head + + 'a structured-JSON value, which analytics does not group by. ' + + 'The query was NOT run. Group by a field that stores one scalar value: store the part you ' + + 'group on in a field of its own and group by that field. A JSON document is no group key ' + + 'the SQL dialects share: one grouped each serialized document apart, another refused the ' + + 'statement.' + : head + + 'a multi-value field, which analytics does not group by. The query was NOT run. ' + + `Filter by one member instead: a record query on '${target.object}' with ` + + `where { "${target.column}": { "$contains": VALUE } } counts or lists the records that hold ` + + 'VALUE, one query per member. A list of values is no group key the SQL dialects share: one ' + + 'grouped each serialized list apart, another refused the statement.', + member, + param, + cube, + target, + ); + } + + for (const member of query.measures ?? []) { + const entry = measureEntryOf(cube, member); + if (entry?.type !== 'count_distinct' || typeof entry.sql !== 'string') continue; + const target = columnOf(cube, baseObject, entry.sql); + if (!target) continue; + const shape = declaredValueShape(target.object, target.column); + if (typeof shape?.type !== 'string') continue; + const cls = distinctClassOf(shape); + if (cls === null) continue; + + const route = cls === 'multi-value' + ? 'Count the records that hold one member instead: count with ' + + `where { "${target.column}": { "$contains": VALUE } } in a record query on '${target.object}', ` + + 'one query per member.' + : 'Count distinct values of a field that stores one scalar value: store the part you count in a ' + + 'field of its own and count_distinct that field, or count the rows with count.'; + throw refusal( + `Measure '${member}' on cube '${cube.name}' counts distinct field '${target.path}', ${declarerOf(target)} ` + + `as ${declaredAs(shape)} — ${cls === 'multi-value' ? 'a multi-value field' : 'a structured-JSON value'}, ` + + `which analytics does not count distinct. The query was NOT run. ${route} ` + + 'A JSON-stored value is no distinct key the SQL dialects share: one compared the serialized ' + + 'text, another refused the statement.', + member, + 'measures', + cube, + target, + ); } }