fix(driver-memory): the cube face lowers the FilterArray spelling of where instead of answering every row

.changeset/20859-cube-filter-array-lowered.md

@@ -0,0 +1,34 @@
+---
+'@objectstack/driver-memory': minor
+---
+
+`MemoryAnalyticsService` lowers the `FilterArray` spelling of `where` instead of dropping it
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) Nothing authorable changes spelling or type: no spec key, export name or stored row moves. What changes is which `where` values one runtime face of this package answers: a `where` array that `isFilterAST` rejects, which `AnalyticsQuerySchema.where` (a `FilterConditionSchema`) never admitted, is now refused as the analytics `where` door and the query engine already refuse it, where it used to answer every row. Which filter the caller meant by such an array is not something a ledger entry can decide. The package publishes (not `unpublished`); no ADR-0087 id covers a runtime filter-shape refusal (not `registered` / `already-registered`); and the change is runtime behaviour, not a declaration (not `runtime-interface-only` / `type-surface-only`). -->
+
+An array `where` such as `[['stage', '=', 'won']]` used to skip every filter step of the
+analytics (cube) face: `query()` aggregated every row and `generateSql()` echoed no `WHERE`,
+while the object spelling `{ stage: 'won' }` answered its rows. The array is now lowered by
+`@objectstack/spec`'s `isFilterAST` / `parseFilterAST` — the lowering the analytics `where` door
+and the engine already apply — so both spellings answer the same rows and echo the same `WHERE`
+on both exits. `[]` still means no filter.
+
+**BREAKING**: `query()` and `generateSql()` now refuse, with `INVALID_FILTER` / 400, a `where`
+array that is not a filter — one `isFilterAST` rejects. Each such array used to answer EVERY row
+and echo no `WHERE`; the analytics `where` door and the query engine refuse the same shapes.
+Measured on a fixture where `d` is `'v1'`, `'v2'`, `null` and absent, each of these answered all
+four rows:
+
+- an infix join, `[['d', '=', 'v1'], 'or', ['d', '=', 'v2']]`;
+- an operator outside the filter-array vocabulary, `[['d', 'sounds_like', 'v1']]`;
+- a list of scalars, `[1, 2, 3]`;
+- a cube-style entry list, `[{ member: 'd', operator: 'equals', values: ['v1'] }]`.
+
+An array that does lower but carries a comparand or operator the face refuses in its object
+spelling (`[['d', 'in', 'v1']]`, `[['d', 'starts_with', 'v']]`) is now refused as that object
+spelling is; it too used to answer every row.
+
+The fix is to write the filter you mean: the prefix form `['or', condA, condB]` for an infix
+join, an operator from the filter-array vocabulary, or the `FilterCondition` object.

packages/drivers/driver-memory/src/memory-analytics-filter-array-where.test.ts

@@ -0,0 +1,162 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * The analytics (cube) face LOWERS the `FilterArray` spelling of `where`, as the
+ * analytics `where` door does, through the same `@objectstack/spec` pair —
+ * `isFilterAST` gates the shape, `parseFilterAST` lowers it — at the entry of
+ * `normalizeFilters`, before the shared comparand doors and the shared lowering.
+ * So both spellings of one filter meet the same doors and answer the same rows.
+ *
+ * Measured on `main` before this door, on the fixture below (`d` holds a value
+ * on rows 1 and 2, is null on row 3, and is absent on row 4):
+ *
+ *   | cube `where`                  | `query()` rows | `generateSql()` echo |
+ *   |-------------------------------|----------------|----------------------|
+ *   | `{d: 'v1'}`                   | 1              | `WHERE d = 'v1'`     |
+ *   | `[['d', '=', 'v1']]`          | 1, 2, 3, 4     | no `WHERE` at all    |
+ *
+ * `normalizeFilters` read only a non-array object `where`, so the array skipped
+ * the doors, the gate and the flatten, and its predicate vanished. Fewer
+ * predicates means MORE rows, and a widened aggregate looks exactly like a
+ * working one — the silent-widening class this face's gate already refuses for
+ * the object spelling. The echo is the control that caught it: an aggregate
+ * cannot tell you it read every row, but the echoed SQL can.
+ *
+ * The three arrival answers are the analytics `where` door's and the engine's:
+ * `[]` is no filter; an array `isFilterAST` accepts is lowered; any other array
+ * is REFUSED `INVALID_FILTER` / 400 on both exits — never every row.
+ */
+
+import { describe, it, expect } from 'vitest';
+import type { Cube } from '@objectstack/spec/data';
+import { InMemoryDriver } from './memory-driver.js';
+import { MemoryAnalyticsService } from './memory-analytics.js';
+
+const ids = (rows: ReadonlyArray<Record<string, unknown>>) => rows.map((r) => String(r.id)).sort();
+
+/** The echo's WHERE clause alone; `undefined` when the echo carries none. */
+const whereOf = (sql: string) => /WHERE (.*?)(?: GROUP BY|$)/.exec(sql)?.[1];
+
+const CUBE = {
+  name: 'arr',
+  title: 'Arr',
+  sql: 'arr_row',
+  measures: { count: { label: 'Count', type: 'count', sql: 'id' } },
+  dimensions: Object.fromEntries(['id', 'd'].map((n) => [n, { label: n, type: 'string', sql: n }])),
+} as unknown as Cube;
+
+/** `d`: a value on 1 and 2, null on 3, absent on 4. */
+const ROWS = [{ id: '1', d: 'v1' }, { id: '2', d: 'v2' }, { id: '3', d: null }, { id: '4' }];
+
+async function setup() {
+  const driver = new InMemoryDriver({});
+  await driver.connect();
+  for (const row of ROWS) await driver.create('arr_row', { ...(row as Record<string, unknown>) });
+  return new MemoryAnalyticsService({ driver, cubes: [CUBE] });
+}
+
+const cubeQuery = (where?: unknown) =>
+  ({ cube: 'arr', measures: ['count'], dimensions: ['id'], ...(where === undefined ? {} : { where }) }) as any;
+
+/** The cube's rows and the echo's WHERE, for one `where`. */
+async function answer(where?: unknown) {
+  const service = await setup();
+  return {
+    rows: ids((await service.query(cubeQuery(where))).rows),
+    echo: whereOf((await service.generateSql(cubeQuery(where))).sql),
+  };
+}
+
+/** Both exits refuse `where` in the ADR-0112 `INVALID_FILTER` / 400 envelope. */
+async function expectRefusedOnBothExits(where: unknown) {
+  const service = await setup();
+  for (const [exit, run] of [
+    ['query', () => service.query(cubeQuery(where))],
+    ['generateSql', () => service.generateSql(cubeQuery(where))],
+  ] as const) {
+    await expect(run(), `${exit}: ${JSON.stringify(where)} must be refused`).rejects.toMatchObject({
+      code: 'INVALID_FILTER',
+      status: 400,
+    });
+  }
+}
+
+describe('the cube face lowers the FilterArray spelling of `where`', () => {
+  it("the card's row: [['d','=','v1']] answers the object spelling's rows, and the echo carries its WHERE", async () => {
+    const array = await answer([['d', '=', 'v1']]);
+    const object = await answer({ d: 'v1' });
+    // Soft, so a red run reports BOTH halves of the card's reading — the rows
+    // and the echo — rather than stopping at the first.
+    expect.soft(array.rows).toEqual(['1']);
+    expect.soft(array.rows).toEqual(object.rows);
+    expect.soft(array.echo).toBe("d = 'v1'");
+    expect.soft(array.echo).toBe(object.echo);
+  });
+
+  // Each row is one FilterArray spelling next to the FilterCondition it lowers
+  // to. Both are fed to the face; they must answer the same rows and echo the
+  // same WHERE — the array meets the comparand doors, the shared lowering and
+  // the vocabulary gate exactly as the object spelling does.
+  const SPELLINGS: Array<[string, unknown, unknown, string[]]> = [
+    ['a bare comparison', ['d', '=', 'v1'], { d: 'v1' }, ['1']],
+    ['a flat list (implicit AND)', [['d', '!=', 'v1'], ['d', '!=', 'v2']], { $and: [{ d: { $ne: 'v1' } }, { d: { $ne: 'v2' } }] }, ['3', '4']],
+    ['an "and" group', ['and', ['id', '!=', '1'], ['d', 'in', ['v1', 'v2']]], { $and: [{ id: { $ne: '1' } }, { d: { $in: ['v1', 'v2'] } }] }, ['2']],
+    ['an "or" group', ['or', ['d', '=', 'v1'], ['d', '=', 'v2']], { $or: [{ d: 'v1' }, { d: 'v2' }] }, ['1', '2']],
+    // A negative-polarity leaf: the shared lowering puts the NULL escape
+    // around it on this spelling as on the other.
+    ['a negative-polarity leaf', [['d', '!=', 'v1']], { d: { $ne: 'v1' } }, ['2', '3', '4']],
+    // The null predicate takes its direction from the operator NAME.
+    ['is_null', [['d', 'is_null']], { d: { $null: true } }, ['3', '4']],
+    ['is_not_null', [['d', 'is_not_null']], { d: { $null: false } }, ['1', '2']],
+  ];
+  for (const [label, array, object, expected] of SPELLINGS) {
+    it(`${label}: ${JSON.stringify(array)} answers ${JSON.stringify(object)}'s rows and echo`, async () => {
+      const a = await answer(array);
+      const o = await answer(object);
+      expect(a.rows).toEqual(expected);
+      expect(a.rows).toEqual(o.rows);
+      expect(a.echo).toBeDefined();
+      expect(a.echo).toBe(o.echo);
+    });
+  }
+
+  it('CONTROL: `[]` is "no filter" — every row and no WHERE, as with no `where` at all', async () => {
+    const empty = await answer([]);
+    const none = await answer();
+    expect(empty.rows).toEqual(['1', '2', '3', '4']);
+    expect(empty.rows).toEqual(none.rows);
+    expect(empty.echo).toBeUndefined();
+    expect(none.echo).toBeUndefined();
+  });
+});
+
+describe('an array the face cannot lower is refused on both exits — never every row', () => {
+  // Not a FilterArray at all: `isFilterAST` refuses the shape before
+  // `parseFilterAST`'s lenient fallback can read it.
+  const NOT_LOWERABLE: Array<[string, unknown]> = [
+    ['the infix join', [['d', '=', 'v1'], 'or', ['d', '=', 'v2']]],
+    ['an operator outside the vocabulary', [['d', 'sounds_like', 'v1']]],
+    ['a list of scalars', [1, 2, 3]],
+    ['a cube-style entry list', [{ member: 'd', operator: 'equals', values: ['v1'] }]],
+  ];
+  for (const [label, where] of NOT_LOWERABLE) {
+    it(`${label}: ${JSON.stringify(where)}`, async () => {
+      await expectRefusedOnBothExits(where);
+    });
+  }
+
+  // Lowerable, but the lowered condition carries what the shared doors or this
+  // face's gate refuse — refused as the object spelling of the same condition
+  // is (the CONTROL half of each row).
+  const REFUSED_AS_OBJECT: Array<[string, unknown, unknown]> = [
+    ['a scalar "in" comparand', [['d', 'in', 'v1']], { d: { $in: 'v1' } }],
+    ['an undefined comparand', [['d', '=', undefined]], { d: undefined }],
+    ['an operator this face does not compile', [['d', 'starts_with', 'v']], { d: { $startsWith: 'v' } }],
+  ];
+  for (const [label, array, object] of REFUSED_AS_OBJECT) {
+    it(`${label}: ${JSON.stringify(array)} is refused, as ${JSON.stringify(object)} is`, async () => {
+      await expectRefusedOnBothExits(array);
+      await expectRefusedOnBothExits(object);
+    });
+  }
+});

packages/drivers/driver-memory/src/memory-analytics.ts

@@ -15,6 +15,11 @@ import {
   lowerFilterCondition,
   normalizeFilterComparandTypes,
 } from '@objectstack/spec/data';
+// The `FilterArray` spelling of `where`, lowered in front of that door by the
+// spec's ONE lowering of it — the pair the analytics `where` door and the engine
+// call: `isFilterAST` gates the shape, `parseFilterAST` lowers it. See
+// {@link lowerWhereFilterArray}. ⛔ No second FilterArray parser in this face.
+import { isFilterAST, parseFilterAST, VALID_AST_OPERATORS } from '@objectstack/spec/data';
 import type { InMemoryDriver } from './memory-driver.js';
 import {
   Logger,
@@ -51,6 +56,7 @@ import {
   assertFilterConditionShape,
   uncompilableCombinatorError,
   uncompilableFieldOperatorError,
+  unsupportedFilterError,
   unsupportedTimeGranularityError,
   type FilterFaceCapabilities,
 } from './filter-refusal.js';
@@ -938,9 +944,86 @@ function explicitDateRangeWindow(dateRange: readonly unknown[]): [string, string
   return [start as string, end as string];
 }
 
+/**
+ * Lower the `FilterArray` spelling of a cube `where` to the `FilterCondition`
+ * it spells, at the entry of {@link MemoryAnalyticsService.normalizeFilters} —
+ * so both spellings of one filter meet the same comparand doors, the same
+ * shared lowering and the same vocabulary gate after it.
+ *
+ * `AnalyticsQuery.where` is declared a `FilterCondition`; `FilterArray` is
+ * input-only authoring sugar (`spec/data/filter.zod.ts`). This face used to
+ * read only a non-array object `where`, so an array skipped every door and the
+ * flatten, and its predicate vanished: `[['d', '=', 'v1']]` aggregated EVERY
+ * row and the `generateSql()` echo carried no `WHERE`, while `{d: 'v1'}`
+ * answered its row. Fewer predicates means more rows — the widening class the
+ * vocabulary gate below already refuses for the object spelling.
+ *
+ * The analytics `where` door lowers the array spelling, and this face is an
+ * analytics face, so it LOWERS rather than refuses — through the same spec pair
+ * that door and the engine call, never a parser of its own. The three arrival
+ * answers are theirs:
+ *
+ * 1. `[]` is "no filter", not a failed filter — `parseFilterAST([])` is
+ *    `undefined`, and every door reads it that way.
+ * 2. An array `isFilterAST` accepts is lowered by `parseFilterAST`, which runs
+ *    the shared comparand-shape and comparand-type faces on what it returns;
+ *    `normalizeFilters` then runs its door on that condition as on any object.
+ * 3. Any other array is REFUSED `INVALID_FILTER` / 400 — `isFilterAST` gates
+ *    first so `parseFilterAST`'s lenient `$${op}` fallback cannot turn a
+ *    misspelled operator into a condition nothing compiles, and so a shape it
+ *    has no lowering for (the infix join, a list of scalars, a cube-style
+ *    `{member, operator, values}` list) is never read as no filter at all.
+ *
+ * Anything that is not an array is returned as it came.
+ */
+function lowerWhereFilterArray(where: unknown): unknown {
+  if (!Array.isArray(where)) return where;
+  // (1) `[]` is "no filter".
+  if (where.length === 0) return undefined;
+  // (3) Not a shape `parseFilterAST` can express.
+  if (!isFilterAST(where)) throw filterArrayNotLowerableError(where);
+  // (2) The declared path.
+  const condition = parseFilterAST(where);
+  if (!condition || typeof condition !== 'object' || Array.isArray(condition)) {
+    // Unreachable by construction — `isFilterAST` accepted the shape, so
+    // `parseFilterAST` has a lowering for it. Loud rather than silent: the
+    // failure mode of the two spec functions disagreeing is a dropped
+    // predicate, i.e. every row.
+    throw unsupportedFilterError(
+      `The analytics (cube) face received the filter array ${previewFilterArray(where)}, which ` +
+        `isFilterAST() accepted and parseFilterAST() lowered to ${previewFilterArray(condition)}. ` +
+        `It is refused rather than aggregating the UNFILTERED rows.`,
+    );
+  }
+  return condition;
+}
+
+/** A `where` array this face cannot lower — the analytics `where` door's refusal, in this face's words. */
+function filterArrayNotLowerableError(where: readonly unknown[]): Error {
+  return unsupportedFilterError(
+    `The analytics (cube) face received a 'where' array that is not a filter: ` +
+      `${previewFilterArray(where)}. A filter array is a comparison [field, operator, value], a ` +
+      `logical node ["and"|"or", ...conditions], or a list of those — it is INPUT-ONLY sugar (spec ` +
+      `'FilterArray'), lowered to a FilterCondition by @objectstack/spec parseFilterAST() at every ` +
+      `door, this one included. This value cannot be lowered, and an unapplied filter would have ` +
+      `aggregated the UNFILTERED rows. Recognised operators: ` +
+      `${[...VALID_AST_OPERATORS].sort().join(', ')}. Infix joins ([condA, "or", condB]) are NOT ` +
+      `one of the shapes — write the prefix form ["or", condA, condB].`,
+  );
+}
+
+/** The refused value for a message — never a second throw (a `bigint` has no JSON). */
+function previewFilterArray(value: unknown): string {
+  try {
+    return JSON.stringify(value) ?? String(value);
+  } catch {
+    return Array.isArray(value) ? 'an array' : typeof value;
+  }
+}
+
 /**
  * Memory-Based Analytics Service
- * 
+ *
  * Implements IAnalyticsService using InMemoryDriver's aggregation capabilities.
  * Provides a semantic layer (Cubes, Metrics, Dimensions) on top of in-memory data.
  * 
@@ -1646,14 +1729,20 @@ export class MemoryAnalyticsService implements IAnalyticsService {
    *    lowered filter it receives. So a `$between` reaches the gate as the two
    *    bounds it lowers to, and answers `find()`'s rows here too.
    *
-   * Not an array: `where` is declared a `FilterCondition`, and an array here is
-   * left to the reading this method has always given it.
+   * ## In front of the door: the `FilterArray` spelling
+   *
+   * An array `where` is lowered FIRST, by {@link lowerWhereFilterArray} — the
+   * spec's `isFilterAST` / `parseFilterAST` pair the analytics `where` door
+   * calls — so it reaches steps 1–3 as the `FilterCondition` it spells and
+   * answers what that object answers; `[]` is no filter, and an array with no
+   * lowering is refused `INVALID_FILTER` / 400. It used to skip all three steps
+   * and the flatten, and so answered every row.
    */
   private normalizeFilters(query: unknown): NormalizedCubeEntry[] {
     if (!query || typeof query !== 'object') return [];
 
     const out: NormalizedCubeEntry[] = [];
-    const where = (query as { where?: unknown }).where;
+    const where = lowerWhereFilterArray((query as { where?: unknown }).where);
 
     if (where && typeof where === 'object' && !Array.isArray(where)) {
       assertListComparandShapes(where);