feat(service-analytics,driver-memory): the shared filter lowering at the analytics seams and the cube face's new door (#5930 step 3)

.changeset/20810-analytics-seam-lowering.md

@@ -0,0 +1,21 @@
+---
+'@objectstack/service-analytics': minor
+---
+
+The analytics seams now run the one shared filter lowering (`lowerFilterCondition`, `@objectstack/spec/data`) that ADR-0053 D-D1, as amended, places at every seam that runs the shared comparand doors: after the doors and after filter-token resolution, so each face compiles one lowered condition — the `$between` split, the whole-day upper bound on a bare `YYYY-MM-DD`, the last supported day, and the NULL-polarity guards.
+
+Clause-②: yes
+
+**The read scope (`compileScopedFilterToSql`).** The scope is lowered at the compiler's entry, right after its placeholders resolve. It reads each column's declared type from the `declaredValueShape` option both of its consumers already pass, so the whole-day rule rewrites a declared `datetime` column and nothing else; with no declarations handed in, no column is read as `datetime`. Corrected answers, each now the rows `SqlDriver.find` returns for the same filter:
+
+- a bare-day `$lte` on a declared `datetime` column kept only the rows before that day and dropped the day itself. Rows at 10:00Z on 07-27, 07-28 and 07-29 under `{ signed_at: { $lte: '2026-07-28' } }` answered 07-27 alone; they now answer 07-27 and 07-28, compiled as `< '2026-07-29'`. This is the NativeSQL statement's read scope and the `/analytics/sql` echo's.
+- a bare-day `$between` on a declared `datetime` column answered no row for a one-day range, and now answers that day's rows.
+- a `{today}` (or any date-macro) upper bound is widened as the day it resolves to.
+
+An RLS `using` bound already reached the read scope lowered by the RLS compile seam, and answers as before. A declared `date` column compiles byte-identical to before.
+
+**The analytics `where` and draft-preview door.** The condition the door admits is lowered before either face reads it. The `where` → tree face (both strategies) reads each member's declared column type through the host's declared-type hook: a bare-day `$lte` on a `datetime` member now reaches the engine as `$lt` the next day, and the `/analytics/sql` echo prints that half-open bound — the statement the engine runs, where it used to print `<=` the named day. Rows are unchanged on every strategy. A nested-relation filter (`{ account: { region: 'NA' } }`) is spelled as the dotted member it has always compiled to before the lowering reads it, so a guard it adds under `$not` names that member.
+
+The draft preview (`queryDataset` with `previewDrafts`) now evaluates `$null` — the one operator the lowering emits that it did not — so a drafted chart filtered on `{ field: { $null: true } }` is answered instead of refused `INVALID_FILTER` / 400; `$exists` and `$empty` stay refused. Corrected answers: a row with no value now satisfies `$ne`, `$nin` and the negation of an equality even when the comparand is the text `"null"` or `"undefined"`, which this face used to compare as text against the missing value — the answer every data driver gives. Its bare-day bounds answer as before.
+
+Compiled SQL for `$ne`, `$nin`, `$notContains` and a `$not` operand now carries the lowering's NULL guard around each face's own copy of it: the same rows, a longer statement, until those copies are deleted.

.changeset/20810-memory-cube-face-door.md

@@ -0,0 +1,26 @@
+---
+'@objectstack/driver-memory': minor
+---
+
+`MemoryAnalyticsService` (the in-memory analytics cube face) now runs the two shared filter comparand doors every other analytics face runs, then the shared filter lowering of ADR-0053 D-D1 (as amended), before it compiles a query's `where`. It also compiles `$or` and `$null`, the two parts of the lowering's output it could not.
+
+Clause-②: yes (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) Nothing authorable changes spelling or type: no spec key, export name or stored row moves. What changes is which filter comparand values one runtime face of this package accepts, to the set every other face already accepts; which value the caller meant by a refused comparand is not something a ledger entry can decide. The package publishes (not `unpublished`); no ADR-0087 id covers a comparand check (not `registered` / `already-registered`); and the change is runtime behaviour, not a declaration (not `runtime-interface-only` / `type-surface-only`). -->
+
+**BREAKING**: `query()` and `generateSql()` now refuse, with `INVALID_FILTER` / 400, filter comparands they used to answer. Each is refused the same way by every other analytics face and by the query engine, so a filter that worked here worked nowhere else. Measured on a fixture where `d` is `'v1'`, `'v2'`, `null` and absent:
+
+- `undefined` in any comparand position — `{ d: undefined }` and `{ d: { $eq: undefined } }` answered the no-value rows, `{ d: { $ne: undefined } }` the valued ones, `{ d: { $in: ['v1', undefined] } }` row `v1`;
+- a `null` member of `$in` / `$nin` (`{ d: { $in: ['v1', null] } }` answered `v1` and both no-value rows), and a `null` under an ordering operator (`{ d: { $gt: null } }` answered no row);
+- a scalar where `$in` / `$nin` takes a list (`{ d: { $in: 'v1' } }`);
+- a plain object, a `Map` or a binary value as a comparand — `{ d: { $ne: { a: 1 } } }` and `{ d: new Map() }` answered EVERY row.
+
+The fix is to write the comparand you mean: `null` or `{ $null: true }` for "has no value", a list for `$in` / `$nin` (and `{ $null: true }` in a `$or` for "one of these, or no value"), a scalar for an ordering operator.
+
+Corrected answers, each now what the live query path (`find()`) returns:
+
+- a bigint comparand within 2^53 (`{ n: { $gt: 2n } }`) is read as its number and answered; beyond 2^53 it is refused `INVALID_FILTER` / 400. Both used to fail with an uncoded error.
+- `$between` is answered as its two bounds, with a bare-day maximum widened to the whole day before it is converted to the field's storage form; it was refused.
+- `$null` (true: no value; false: has a value) and `$or` (a `{}` branch is TRUE, `$or: []` is FALSE) are compiled on both exits; they were refused. `$not`, `$startsWith`, `$endsWith` and `$empty` stay refused. `ANALYTICS_FILTER_CAPABILITIES` names `$null` and the `$or` combinator accordingly.
+
+The `generateSql()` echo and the `query()` pipeline dump for `$ne`, `$nin` and `$notContains` now show the lowering's NULL escape around this face's own guard — `(d IS NULL OR (d IS NULL OR d != 'v1'))` — the same rows as before.

packages/drivers/driver-memory/src/memory-analytics-20661-lte-whole-day-first.test.ts

@@ -148,18 +148,20 @@ describe('[#20661] the siblings on this face', () => {
     expect(ids(result.rows)).toEqual(['r27', 'r28']);
   });
 
-  it('`$between` is refused on both exits, so its maximum never reaches a bound here', async () => {
-    // Not a lowering this face owns: widening `MONGO_TO_CUBE_OPERATOR` to take
-    // `$between` turns this red, and its maximum then owes the same whole-day
-    // order as the `lte` rows (`lteUpperBound`).
-    const { service } = await setup('declared');
-    const between = cubeQuery({ created_at: { $between: ['2026-07-01', '2026-07-28'] } });
-    for (const exit of [() => service.query(between), () => service.generateSql(between)]) {
-      const err = await exit().then(
-        () => undefined,
-        (e: unknown) => e as { code?: unknown; status?: unknown },
-      );
-      expect(err).toMatchObject({ code: 'INVALID_FILTER', status: 400 });
+  it('`$between` reaches the face as its two bounds, and its maximum is widened BEFORE it is converted', async () => {
+    // [ADR-0053 D-D1, amended — #5930 step 3] This row used to pin `$between`
+    // refused on both exits, and said what widening it would owe: the same
+    // whole-day order as the `lte` rows. The face's door now runs the shared
+    // lowering first, which splits the range and widens the AUTHORED maximum
+    // in the calendar-string domain (`$lt: '2026-07-29'`); the exit converts
+    // that bound to the storage form like any comparand. D-E3's order holds by
+    // construction, declared or not.
+    for (const declaration of ['undeclared', 'declared'] as const) {
+      const got = await answer(declaration, { created_at: { $between: ['2026-07-01', '2026-07-28'] } });
+      expect(got.cube, declaration).toEqual(got.find);
+      expect(got.cube, declaration).toEqual(['r27', 'r28']);
     }
+    expect((await answer('declared', { created_at: { $between: ['2026-07-01', '2026-07-28'] } })).echo)
+      .toBe("created_at >= '2026-07-01T00:00:00.000Z' AND created_at < '2026-07-29T00:00:00.000Z'");
   });
 });

packages/drivers/driver-memory/src/memory-analytics-echo-operator-coverage.test.ts

@@ -177,14 +177,27 @@ const ACCEPTED_CASES: Record<string, FilterCondition> = {
   $icontains: { name: { $icontains: 'industries' } },
   $notContains: { name: { $notContains: 'Industries' } },
   $exists: { name: { $exists: true } },
+  // [ADR-0053 D-D1, amended — #5930 step 3] `$null` joined the face's table:
+  // the shared lowering emits it. Row 6's NULL `name` makes it a real predicate.
+  $null: { name: { $null: true } },
 };
 
 /**
- * The spellings this face REFUSES — the complement, so the two sets are total.
- * [#20446] `$empty` joined `FILTER_OPERATORS`, and this cube face refuses it as
- * it refuses `$null`: a declared operator it has no lowering for.
+ * [ADR-0053 D-D1, amended — #5930 step 3] The spellings the face's door LOWERS
+ * before its vocabulary is asked: a `$between` reaches the face as the two
+ * bounds the shared lowering splits it into. Not in the face's table and not
+ * refused — so a third set, held to the same executed-echo invariant.
  */
-const REFUSED_OPERATORS = ['$between', '$startsWith', '$endsWith', '$null', '$empty'] as const;
+const LOWERED_CASES: Record<string, FilterCondition> = {
+  $between: { amount: { $between: [20, 40] } },
+};
+
+/**
+ * The spellings this face REFUSES — the complement, so the three sets are total.
+ * [#20446] `$empty` joined `FILTER_OPERATORS`, and this cube face refuses it: a
+ * declared operator it has no lowering for.
+ */
+const REFUSED_OPERATORS = ['$startsWith', '$endsWith', '$empty'] as const;
 
 describe('[#7117] the analytics echo renders the query it describes', () => {
   let db: any;
@@ -441,8 +454,8 @@ describe('[#7117] the analytics echo renders the query it describes', () => {
   // ── The vocabulary, enumerated so the two tables cannot drift apart ────────
 
   describe('the closed vocabulary, enumerated', () => {
-    it('the accepted and refused sets together are the whole Filter Protocol', () => {
-      expect(sortIds([...Object.keys(ACCEPTED_CASES), ...REFUSED_OPERATORS]))
+    it('the accepted, lowered and refused sets together are the whole Filter Protocol', () => {
+      expect(sortIds([...Object.keys(ACCEPTED_CASES), ...Object.keys(LOWERED_CASES), ...REFUSED_OPERATORS]))
         .toEqual(sortIds([...FILTER_OPERATORS]));
     });
 
@@ -451,7 +464,7 @@ describe('[#7117] the analytics echo renders the query it describes', () => {
         .toEqual(sortIds([...ANALYTICS_FILTER_CAPABILITIES.fieldOperators]));
     });
 
-    for (const [op, where] of Object.entries(ACCEPTED_CASES)) {
+    for (const [op, where] of Object.entries({ ...ACCEPTED_CASES, ...LOWERED_CASES })) {
       it(`${op}: running the echo returns exactly the rows the query returns`, async () => {
         const executed = await executedIds(where);
         const echoed = await echoIds(where);

packages/drivers/driver-memory/src/memory-analytics-filter-refusal.test.ts

@@ -24,6 +24,14 @@
  * Both public entry points are covered here — `query()` and `generateSql()` —
  * because both call `normalizeFilters` and a refusal on one only would leave the
  * other silently answering the old way.
+ *
+ * [ADR-0053 D-D1, amended 2026-09-30 — #5930 step 3] Three of those shapes are
+ * no longer refusals: the face now COMPILES `$or` and `$null` (the shared
+ * lowering emits both), and a `$between` reaches its vocabulary gate as the two
+ * bounds the lowering splits it into. Each is pinned below as answered with the
+ * rows the live query path gives it — the invariant this file has always held,
+ * refuse-or-agree, never a different row set. `$not`, `$startsWith` and
+ * `$endsWith` stay refused.
  */
 
 import { describe, it, expect, beforeEach } from 'vitest';
@@ -104,23 +112,30 @@ describe('[#5345] MemoryAnalyticsService — filters it cannot compile are refus
 
   // ── The combinators (`continue` #1) ────────────────────────────────────────
 
-  it('refuses $or instead of aggregating the whole table', async () => {
+  // [#5930 step 3] `$or` is compiled now, so the regression this section was
+  // written against — three rows asked for, five returned — is pinned as its
+  // answer: the rows the live query path returns, never the whole table.
+  const findCount = async (where: FilterCondition): Promise<number> => (await driver.find('deals', { where })).length;
+
+  it('compiles $or instead of aggregating the whole table', async () => {
     const where: FilterCondition = { $or: [{ stage: 'won' }, { stage: 'lost' }] };
-    const err = await expectRefusal(() => count(where), '$or', 'where.$or');
-    // The regression in one line: three rows were asked for, five were returned.
-    expect(err.message).toContain('WIDENS');
+    expect(await count(where)).toBe(3);
+    expect(await count(where)).toBe(await findCount(where));
   });
 
-  it('refuses a $or nested inside a $and it CAN compile', async () => {
+  it('compiles a $or nested inside a $and', async () => {
     const where: FilterCondition = { $and: [{ owner: 'u1' }, { $or: [{ stage: 'won' }, { stage: 'open' }] }] };
-    await expectRefusal(() => count(where), '$or', 'where.$and[1].$or');
+    expect(await count(where)).toBe(2);
+    expect(await count(where)).toBe(await findCount(where));
   });
 
-  it('refuses a $or that sits beside a compilable sibling key', async () => {
-    // The sharpest shape: the sibling `owner` lowered fine, so the query ran and
-    // returned a plausible-looking number computed without the $or at all.
+  it('compiles a $or that sits beside a sibling key, ANDed with it', async () => {
+    // The sharpest shape when `$or` was dropped: the sibling `owner` lowered
+    // fine, so the query ran and returned a plausible-looking number computed
+    // without the $or at all. Compiled, it narrows.
     const where: FilterCondition = { owner: 'u1', $or: [{ stage: 'won' }, { stage: 'lost' }] };
-    await expectRefusal(() => count(where), '$or');
+    expect(await count(where)).toBe(2);
+    expect(await count(where)).toBe(await findCount(where));
   });
 
   it('refuses $not — the shape an RLS read scope compiles to', async () => {
@@ -130,19 +145,31 @@ describe('[#5345] MemoryAnalyticsService — filters it cannot compile are refus
   });
 
   it('names the combinators it CAN compile, so the refusal is actionable', async () => {
-    const err = await expectRefusal(() => count({ $or: [{ stage: 'won' }] }));
-    expect(err.message).toContain('Supported combinators on this surface: $and');
+    const err = await expectRefusal(() => count({ $not: { stage: 'won' } }));
+    expect(err.message).toContain('Supported combinators on this surface: $and, $or');
   });
 
   // ── The unmapped operators (`continue` #2) ─────────────────────────────────
 
   const UNCOMPILABLE: Array<{ op: string; where: FilterCondition }> = [
-    { op: '$between', where: { amount: { $between: [100, 200] } } },
     { op: '$startsWith', where: { name: { $startsWith: 'al' } } },
     { op: '$endsWith', where: { name: { $endsWith: 'ta' } } },
-    { op: '$null', where: { closed_at: { $null: true } } },
   ];
 
+  // [#5930 step 3] Answered, not refused: `$null` joined this face's table, and
+  // a `$between` reaches it as its two bounds. Each agrees with the live path.
+  const NOW_COMPILED: Array<{ op: string; where: FilterCondition; rows: number }> = [
+    { op: '$between', where: { amount: { $between: [100, 200] } }, rows: 2 },
+    { op: '$null', where: { closed_at: { $null: true } }, rows: 2 },
+  ];
+
+  for (const { op, where, rows } of NOW_COMPILED) {
+    it(`compiles ${op}, with the rows the live query path returns`, async () => {
+      expect(await count(where)).toBe(rows);
+      expect(await findCount(where)).toBe(rows);
+    });
+  }
+
   for (const { op, where } of UNCOMPILABLE) {
     it(`refuses ${op} — declared by the Filter Protocol, not compilable by this face`, async () => {
       const err = await expectRefusal(() => count(where), op);
@@ -221,17 +248,17 @@ describe('[#5345] MemoryAnalyticsService — filters it cannot compile are refus
       () => service.generateSql(asQuery({
         cube: 'deals',
         measures: ['deals.count'],
-        where: { $or: [{ stage: 'won' }, { stage: 'lost' }] },
+        where: { $not: { stage: 'lost' } },
       })),
-      '$or',
+      '$not',
     );
     await expectRefusal(
       () => service.generateSql(asQuery({
         cube: 'deals',
         measures: ['deals.count'],
-        where: { amount: { $between: [1, 2] } },
+        where: { name: { $startsWith: 'al' } },
       })),
-      '$between',
+      '$startsWith',
     );
     const ok = await service.generateSql(asQuery({
       cube: 'deals',