fix(service-analytics)!: the analytics door refuses a multi-value grouping and a JSON-stored count_distinct, judged on the declaration, as the engine does

.changeset/20912-analytics-multi-value-distinct-refused.md

@@ -0,0 +1,27 @@
+---
+"@objectstack/service-analytics": minor
+---
+
+fix(service-analytics)!: a grouped dimension on a multi-value field and a `count_distinct` measure over a JSON-stored field are refused with `INVALID_FIELD` / 400 at the analytics door, before any SQL is built; a dataset `count_distinct` measure over a field declared `multiple: true` is refused at compile time
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (runtime-interface-only packages/services/service-analytics/src/dataset-compiler.ts#DatasetCompileOptions) a published options interface of `compileDataset`, whose one probe member is renamed and now answers the field's declaration instead of its type. It is not a Zod schema, a `packages/spec` declaration or an object definition, it is not a projection of a schema, and no metadata surface references it, so `objectstack migrate meta` has nothing to rewrite; the TypeScript host that passed the old member is told by its compiler at its own call site. The other halves refuse a QUERY shape at the analytics door (a grouping or distinct-count target) and one aggregate x declaration pair at the dataset compile door: no authorable key, spelling or stored row moves (`CubeSchema`, `DatasetSchema` and the analytics query body keep parsing every member), and which scalar a caller meant to group on or count is not something a ledger entry can rewrite. The other categories are closed on facts: the package publishes (not `unpublished`); no ADR-0087 id covers an analytics grouping or distinct-count target or this interface, and this diff adds none (not `registered` / `already-registered`); and the interface was concretely typed at the merge base, not erased (not `type-surface-only`). -->
+
+**BREAKING**: this narrows what the analytics query doors accept, in two positions, and what the dataset compiler accepts, in one. It holds on `POST /api/v1/analytics/query`, on its dry run `POST /api/v1/analytics/sql` and on `POST /api/v1/analytics/dataset/query`, on every driver and on both strategies. It ships as `minor` under the launch-window convention for accept-set narrowings.
+
+- A cube or dataset dimension whose column is a **multi-value** field, when it groups the result (a `dimensions` entry, or a `timeDimensions` entry with a `granularity`): an inherently multi option type (`multiselect`, `checkboxes`, `tags`), or a `select`, `radio`, `lookup`, `user`, `file` or `image` field declared `multiple: true`. The same types without the flag are served.
+- A `measures` entry that resolves to a `count_distinct` measure whose column is **JSON-stored**: a structured-JSON type (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`), an inherently multi option type, or a multi-capable field declared `multiple: true`. An authored cube measure, a suffix-inferred one (`tags_count_distinct`) and a compiled dataset's are judged alike; a `count` measure is not judged.
+- A dataset measure that pairs `count_distinct` with a base-object field declared `multiple: true` is refused `400 DATASET_INVALID` when the dataset compiles, at registration and on the request door, beside the type row that already refused `tags`.
+
+The column is judged where it is declared: on the cube's object, or, for a dotted path, on the object the cube's declared join names.
+
+**What an author sees now.** `400 INVALID_FIELD`, naming the member as the request wrote it, the column, the object, the declaration (`select with multiple: true`), saying the query was not run, and naming the route. For a multi-value field the route is a record query on the declaring object filtered by one member with `$contains`, one query per member. For a structured-JSON field it is to store the scalar part in a field of its own. The thrown error carries `member`, `param` (`dimensions`, `timeDimensions` or `measures`), `cube`, `field` and `object`. The dataset compile refusal names the measure, the field and its declaration with `multiple: true`.
+
+**Why a refusal.** The engine's aggregate door already refuses both shapes, and the native-SQL strategy compiled its own statement and never reached it. Measured through this service as `AnalyticsServicePlugin` composes it over a real engine: a dimension on a `tags`, `multiselect` or `multiple: true` select answered 200 with one group per serialized array on SQLite and 500 `DATABASE_ERROR` on PostgreSQL 16; an inferred `count_distinct` over a `json`, `tags` or `multiple: true` select field answered 2, 3 and 2 on SQLite and 500 on PostgreSQL; a dataset `count_distinct` over the `multiple: true` select registered, then answered 2 on SQLite and 500 on PostgreSQL. The engine-aggregate strategy answered 400 for every one of these, under the engine's position (`groupBy[0]`, `aggregations[0].field`) rather than the member the caller wrote. The predicates are `@objectstack/spec/data`'s, the ones the engine's doors read: `isMultiValueField`, and the aggregate × field-type table's `count_distinct` row.
+
+**Your fix (a host calling `compileDataset` directly).** `DatasetCompileOptions` no longer has `declaredFieldType`. Pass `declaredValueShape` instead: the same read of the field's metadata, answering `{ type, multiple }` (the type, and `multiple === true`) rather than the type alone, or `undefined` when nothing answers. A host that passes neither compiles exactly as before, with no aggregate × field-type refusal at all. `AnalyticsService` and `AnalyticsServicePlugin` wire it themselves from `sourceFieldMeta`.
+
+**Who is affected.** A dashboard, report or caller that grouped by a multi-value field, or counted a JSON-stored field distinct, through the native-SQL strategy on SQLite and read the serialized arrays or the text-compared count as real answers. On PostgreSQL the same queries were already a 500. A dataset that pairs `count_distinct` with a `multiple: true` field no longer registers.
+
+**Unchanged.** A dimension or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; a member naming a column the object does not have, which keeps its existing `INVALID_FIELD` answer first; a dotted path the cube declares no join for; a measure whose `sql` is an expression; and a host that wires no `sourceFieldMeta`, where the declaration cannot be read.

packages/rest/src/analytics-dataset-multi-value-door.test.ts

@@ -0,0 +1,242 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * A dataset dimension on a MULTI-VALUE field, and a dataset `count_distinct`
+ * measure over a `select` declared `multiple: true`, are refused at the
+ * dataset door — `POST /api/v1/analytics/dataset/query` answers 400 before
+ * any SQL is built — over a real `SqlDriver`; the single-value controls are
+ * served unchanged.
+ *
+ * The cube face (`POST /api/v1/analytics/query`, an inferred
+ * `FIELD_count_distinct` included) is served by `@objectstack/runtime`'s
+ * dispatcher, not by this package; its rows are pinned at the service, on the
+ * same two dialects, in
+ * `packages/services/service-analytics/src/__tests__/json-stored-door-live-drivers.test.ts`.
+ *
+ * ## Measured without the refusal, through this door
+ *
+ * Four rows: `status` a, a, b, b; `tags_f` [a,b], [a,b], [b], [ab];
+ * `multi_sel` (a `select` with `multiple: true`) [a,b], [a,b], [b], [b]:
+ *
+ * | the inline dataset's selection | SQLite | PostgreSQL 16 |
+ * |:--|:--|:--|
+ * | dimension `status_dim` (single-value select, the control) | 200, `A` 2 · `B` 2 (the option labels) | same |
+ * | dimension `tag_dim` (`tags`) / `picked_dim` (`multi_sel`) | 200, one group per serialized array | 500 |
+ * | measure `count_distinct` over `multi_sel` | 200, 2 | 500 |
+ * | measure `count_distinct` over `status` (the control) | 200, 2 | same |
+ *
+ * ## The composition, and the dialect axis of THIS file
+ *
+ * The analytics service is the one `AnalyticsServicePlugin` composes over a
+ * real `ObjectQL` engine — both auto-bridges live, so `NativeSQLStrategy`
+ * answers on a SQL driver. The SQLite cell always runs. The PostgreSQL cell
+ * runs where `OS_TEST_POSTGRES_URL` is set and is a named skip otherwise; no
+ * CI step provisions that variable for this package, so the live cell is
+ * red-capable and un-run in CI. The live cell owns its table, dropped before
+ * and after.
+ */
+
+import { describe, it, expect, beforeAll, afterAll } from 'vitest';
+import { ObjectQL } from '@objectstack/objectql';
+import { SqlDriver } from '@objectstack/driver-sql';
+import { AnalyticsServicePlugin, type AnalyticsService } from '@objectstack/service-analytics';
+import { RestServer } from './rest-server';
+
+const OBJECT = 'rest_dataset_multi_value_ledger';
+const OPTIONS = [{ label: 'A', value: 'a' }, { label: 'B', value: 'b' }];
+
+const LEDGER = {
+  name: OBJECT,
+  label: 'Dataset multi-value door ledger',
+  fields: {
+    status: { name: 'status', type: 'select' as const, options: OPTIONS },
+    tags_f: { name: 'tags_f', type: 'tags' as const },
+    multi_sel: { name: 'multi_sel', type: 'select' as const, multiple: true, options: OPTIONS },
+  },
+};
+
+const ROWS = [
+  { id: 'r1', status: 'a', tags_f: ['a', 'b'], multi_sel: ['a', 'b'] },
+  { id: 'r2', status: 'a', tags_f: ['a', 'b'], multi_sel: ['a', 'b'] },
+  { id: 'r3', status: 'b', tags_f: ['b'], multi_sel: ['b'] },
+  { id: 'r4', status: 'b', tags_f: ['ab'], multi_sel: ['b'] },
+];
+
+/** The inline dataset the request carries — as a Studio preview or a widget posts it. */
+const dataset = (measures: unknown[]) => ({
+  name: 'multi_value_inline',
+  label: 'Multi-value inline',
+  object: OBJECT,
+  dimensions: [
+    { name: 'status_dim', field: 'status', type: 'string' },
+    { name: 'tag_dim', field: 'tags_f', type: 'string' },
+    { name: 'picked_dim', field: 'multi_sel', type: 'string' },
+  ],
+  measures,
+});
+const ROW_COUNT = [{ name: 'row_count', aggregate: 'count' }];
+
+interface Cell {
+  id: 'sqlite' | 'pg';
+  label: string;
+  env: string | null;
+  config: () => Record<string, unknown> | null;
+}
+
+const CELLS: readonly Cell[] = [
+  { id: 'sqlite', label: 'sqlite', env: null, config: () => ({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }) },
+  {
+    id: 'pg',
+    label: 'live postgres',
+    env: 'OS_TEST_POSTGRES_URL',
+    config: () => (process.env.OS_TEST_POSTGRES_URL ? { client: 'pg', connection: process.env.OS_TEST_POSTGRES_URL } : null),
+  },
+];
+
+const quiet = { debug() {}, info() {}, warn() {}, error() {}, child() { return quiet; } };
+
+function createMockServer() {
+  const noop = () => {};
+  return { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} };
+}
+
+function mockProtocol() {
+  return {
+    getDiscovery: async () => ({ version: 'v0', routes: { data: '', metadata: '' } }),
+    getMetaTypes: async () => [],
+    getMetaItems: async () => [],
+  };
+}
+
+function makeRes() {
+  const res: any = {
+    statusCode: 200,
+    body: undefined as any,
+    header: () => res,
+    status: (code: number) => { res.statusCode = code; return res; },
+    json: (body: unknown) => { res.body = body; return res; },
+    end: () => res,
+  };
+  return res;
+}
+
+for (const cell of CELLS) {
+  const config = cell.config();
+  describe.skipIf(!config)(
+    `a multi-value dataset dimension or distinct count at POST /api/v1/analytics/dataset/query — ${cell.label}${config ? '' : ` (skipped: set ${cell.env} to run this cell)`}`,
+    () => {
+      let driver: any;
+      let engine: ObjectQL;
+      /** Raw-SQL statements and engine aggregates that read THIS object. */
+      const reads = { rawSql: 0, aggregate: 0 };
+      let query: (measures: unknown[], selection: Record<string, unknown>) => Promise<{ status: number; body: any }>;
+
+      const dropTable = async () => {
+        if (cell.id === 'sqlite') return;
+        await driver?.execute(`drop table if exists ${OBJECT}`).catch(() => {});
+      };
+
+      beforeAll(async () => {
+        driver = new SqlDriver(config as any);
+        await dropTable();
+        engine = new ObjectQL({ logger: quiet } as any);
+        engine.registerDriver(driver, true);
+        await engine.init();
+        engine.registry.registerObject(LEDGER as any);
+        await engine.syncSchemas();
+        for (const row of ROWS) await engine.insert(OBJECT, { ...row } as any);
+
+        const realExecute = (engine as any).execute.bind(engine);
+        (engine as any).execute = (sql: unknown, opts?: { object?: string }) => {
+          if (opts?.object === OBJECT) reads.rawSql += 1;
+          return realExecute(sql, opts);
+        };
+        const realAggregate = engine.aggregate.bind(engine);
+        (engine as any).aggregate = (object: string, ...rest: unknown[]) => {
+          if (object === OBJECT) reads.aggregate += 1;
+          return (realAggregate as any)(object, ...rest);
+        };
+
+        // The plugin's own composition over the real engine: both auto-bridges.
+        const registered: Record<string, unknown> = {};
+        await new AnalyticsServicePlugin().init({
+          getService: (name: string) => (name === 'data' ? engine : registered[name]),
+          registerService: (name: string, svc: unknown) => { registered[name] = svc; },
+          replaceService: (name: string, svc: unknown) => { registered[name] = svc; },
+          hook: () => {},
+          logger: quiet,
+        } as never);
+        const service = registered.analytics as AnalyticsService;
+
+        const rest = new RestServer(
+          createMockServer() as any, mockProtocol() as any, { api: { requireAuth: false } } as any,
+          undefined, undefined, undefined, undefined, undefined, undefined, undefined,
+          undefined, undefined, undefined, undefined,
+          async () => service,
+        );
+        (rest as any).resolveExecCtx = async () => ({ userId: 'test-user' });
+        rest.registerRoutes();
+        const route = rest.getRoutes().find((r: any) => r.method === 'POST' && r.path === '/api/v1/analytics/dataset/query');
+        expect(route).toBeDefined();
+        query = async (measures, selection) => {
+          const res = makeRes();
+          // What the wire carries: JSON.
+          const body = JSON.parse(JSON.stringify({ dataset: dataset(measures), selection }));
+          await route!.handler({ method: 'POST', params: {}, headers: {}, body, query: {} } as any, res);
+          return { status: res.statusCode, body: res.body };
+        };
+      });
+
+      afterAll(async () => {
+        await dropTable();
+        try { await engine?.destroy(); } catch { /* noop */ }
+      });
+
+      it('a dataset dimension on a multi-value field answers 400 INVALID_FIELD naming the dataset dimension — no statement reaches the engine', async () => {
+        const before = { ...reads };
+        for (const [member, column, declared] of [
+          ['tag_dim', 'tags_f', 'tags'],
+          ['picked_dim', 'multi_sel', 'select with multiple: true'],
+        ] as const) {
+          const res = await query(ROW_COUNT, { measures: ['row_count'], dimensions: [member] });
+          expect(res.status, JSON.stringify(res.body)).toBe(400);
+          expect(res.body.code, member).toBe('INVALID_FIELD');
+          expect(String(res.body.message), member).toContain(
+            `Dimension '${member}' on cube 'multi_value_inline' groups by field '${column}', which object '${OBJECT}' declares as ${declared} — a multi-value field`,
+          );
+        }
+        expect(reads, 'no raw SQL and no engine aggregate for the object').toEqual(before);
+      });
+
+      it('CONTROL a single-value select dimension is served unchanged: one group per option, counted under its label', async () => {
+        const res = await query(ROW_COUNT, { measures: ['row_count'], dimensions: ['status_dim'] });
+        expect(res.status, JSON.stringify(res.body)).toBe(200);
+        const groups = (res.body.rows as Array<{ status_dim: string; row_count: number | string }>)
+          .map((r) => [r.status_dim, Number(r.row_count)] as const)
+          .sort(([a], [b]) => a.localeCompare(b));
+        expect(groups).toEqual([['A', 2], ['B', 2]]);
+      });
+
+      it('a dataset count_distinct over a select with multiple: true answers 400 DATASET_INVALID — no statement reaches the engine', async () => {
+        const before = { ...reads };
+        const res = await query(
+          [{ name: 'picked_kinds', aggregate: 'count_distinct', field: 'multi_sel' }],
+          { measures: ['picked_kinds'] },
+        );
+        expect(res.status, JSON.stringify(res.body)).toBe(400);
+        expect(res.body.code).toBe('DATASET_INVALID');
+        expect(String(res.body.message)).toContain('declares as `select` with `multiple: true`');
+        expect(reads, 'no raw SQL and no engine aggregate for the object').toEqual(before);
+      });
+
+      it('CONTROL a dataset count_distinct over a single-value select is served', async () => {
+        const res = await query(
+          [{ name: 'status_kinds', aggregate: 'count_distinct', field: 'status' }],
+          { measures: ['status_kinds'] },
+        );
+        expect(res.status, JSON.stringify(res.body)).toBe(200);
+        expect(Number(res.body.rows[0].status_kinds)).toBe(2);
+      });
+    },
+  );
+}

packages/services/service-analytics/src/__tests__/dimension-structured-json-door.test.ts

@@ -21,13 +21,13 @@
  *   (inferred) cube, and a bucketed time dimension;
  * - no statement and no aggregate reaches the host bridges on a refusal;
  * - GUARD: the judged types are exactly `@objectstack/spec/data`'s
- *   `STRUCTURED_JSON_TYPES`, over every `FieldType` — the predicate the
- *   engine's door reads, never a second list.
+ *   `STRUCTURED_JSON_TYPES` and `isMultiValueField`, over every `FieldType` —
+ *   the predicates the engine's door reads, never a second list.
  */
 
 import { describe, it, expect, vi } from 'vitest';
 import type { Cube } from '@objectstack/spec/data';
-import { FieldType, STRUCTURED_JSON_TYPES } from '@objectstack/spec/data';
+import { FieldType, STRUCTURED_JSON_TYPES, isMultiValueField } from '@objectstack/spec/data';
 import type { Dataset } from '@objectstack/spec/ui';
 import type { AnalyticsQuery } from '@objectstack/spec/contracts';
 import { AnalyticsService } from '../analytics-service.js';
@@ -246,7 +246,11 @@ describe('what the door does not judge', () => {
     expect(calls.raw).toHaveLength(1);
   });
 
-  it('GUARD the judged types are exactly the spec\'s STRUCTURED_JSON_TYPES, over every FieldType', async () => {
+  it('GUARD the judged types are exactly the spec\'s STRUCTURED_JSON_TYPES and isMultiValueField, over every FieldType', async () => {
+    // The multi-value class joined the door with the engine's: an inherently
+    // multi option type (`multiselect`, `checkboxes`, `tags`) is judged with or
+    // without the flag. The flagged half is pinned in
+    // `multi-value-json-stored-door.test.ts`.
     const { service } = makeService('native');
     for (const type of FieldType.options) {
       const query: AnalyticsQuery = { cube: OBJECT, measures: ['count'], dimensions: [`f_${type}`] };
@@ -255,7 +259,9 @@ describe('what the door does not judge', () => {
         (e: Refusal) => ({ code: e.code, status: e.status, member: e.member }),
       );
       expect(verdict, type).toEqual(
-        STRUCTURED_JSON_TYPES.has(type) ? { code: 'INVALID_FIELD', status: 400, member: `f_${type}` } : null,
+        STRUCTURED_JSON_TYPES.has(type) || isMultiValueField({ type })
+          ? { code: 'INVALID_FIELD', status: 400, member: `f_${type}` }
+          : null,
       );
     }
   });