fix(service-analytics)!: the analytics where door runs every arm of the shared comparand-shape face on the object spelling (#20010)

.changeset/20010-analytics-where-face-arms.md

@@ -0,0 +1,28 @@
+---
+"@objectstack/service-analytics": minor
+---
+
+fix(service-analytics)!: the analytics `where` door runs every arm of the shared comparand-shape face on the object spelling, not only the equality arm (#20010)
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (already-registered filter-between-blank-endpoint-refused) this change adds no new transition. It brings the analytics object-form `where`, its dataset and measure filters, and the draft-data preview under refusals the shared comparand-shape face already makes at every other door. The one arm whose transition is on the ledger is the blank `$between` endpoint (named). The null list member, the null `$between` endpoint and the null ordering comparand were ruled at the face with no ledger entry (their changesets declared no-migration-prescription: which explicit spelling matches the author's intent is an authoring decision no migration entry can perform). The non-list `$in` / `$nin` / `$between` refusal is the face's original rule. The `{ $field }` and one-bound `$between` endpoints were already refused on this door and change wording only. The table below is the author-facing remedy for each arm, not a mechanical rewrite. -->
+
+**BREAKING**: this narrows what the analytics faces of `@objectstack/service-analytics` accept. A caller `where`, a dataset `filter` or a measure `filter` that carries one of the shapes below compiled before this change, at any depth under `$and` / `$or` / `$not` or inside a nested relation. It is now refused with `INVALID_FILTER` / 400, carrying the shared face's own message, path and prescription. It ships as `minor` under the launch-window convention for accept-set narrowings.
+
+| you wrote | what it did before | write instead |
+|:--|:--|:--|
+| `{ stage: { $in: ['won', null] } }`, meaning "one of these, or empty" | `stage IN ('won', NULL)`: the empty rows were never matched | `{ $or: [{ stage: { $in: ['won'] } }, { stage: { $null: true } }] }` |
+| `{ stage: { $nin: ['won', null] } }`, meaning "has a value, and not one of these" | `stage IS NULL OR stage NOT IN ('won', NULL)`: only the empty rows | `{ $and: [{ stage: { $nin: ['won'] } }, { stage: { $null: false } }] }` |
+| `{ amount: { $gt: null } }` (or `$gte` / `$lt` / `$lte`) | `amount > NULL`: no row, while for `$lt` / `$lte` the draft preview charted every non-empty row | `{ amount: { $eq: null } }` for "has no value", `{ amount: { $ne: null } }` for "has a value" |
+| `{ amount: { $between: [null, 100] } }` | `amount >= NULL AND amount <= 100`: no row | `{ amount: { $lte: 100 } }` for a one-sided range; `$or` with `{ amount: { $null: true } }` to include the empty rows |
+| `{ amount: { $between: ['', 100] } }` | `amount >= '' AND amount <= 100`: the blank compared as a value, and the ObjectQL engine path accepted it | the bound you meant, or `{ amount: { $lte: 100 } }` for a one-sided range |
+| `{ stage: { $in: 'won' } }` or `{ stage: { $nin: 'won' } }` | laundered into a one-member list | `{ stage: 'won' }` / `{ stage: { $ne: 'won' } }`, or `{ stage: { $in: ['won'] } }` |
+
+The shared comparand-shape face in `@objectstack/spec` (`assertListComparandShapes`) is the one place that decides whether `$in` / `$nin` / `$between` received a list at all, for every driver. Three rulings put the null and blank positions on that same door: a null list member and a null `$between` endpoint (2026-08-31), a null ordering comparand (2026-09-01), and a blank `$between` endpoint (2026-09-20). The analytics `where` door met that face only for the `FilterArray` spelling (`['stage', 'in', ['won', null]]`), which was already refused. PR #20008 carried the face's equality arm to the object spelling. Every other arm now follows it: each field entry of the object-form `where` is handed to the face after the equality pass, so both spellings of one condition get the same refusal, byte for byte. This holds on the native SQL execute path, the `/analytics/sql` echo and the ObjectQL engine path. The draft-data preview runs the same gate, so a drafted chart refuses what the published chart refuses. Before, the preview charted rows for several of these shapes: `{ amount: { $lt: null } }` charted every non-empty row.
+
+Three shapes this door already refused now carry the face's wording instead of this package's own, the same wording the `FilterArray` spelling gets: a `$between` that is not a two-element list, a `$between` endpoint that is a `{ $field }` reference, and an `undefined` `$between` endpoint. Their verdict, code and status are unchanged.
+
+Who is affected: nothing in this repository's examples, seeds, docs or package sources authors any of the shapes. This was measured by a text scan over 4013 non-test files with positive controls. Stored datasets, dashboard widget filters and report runtime filters in a deployment were NOT measured. The authoring schema still admits the shapes, so such a document still publishes, and it is refused when it is charted.
+
+Not changed: `$ne` with a list is not judged by the face yet, so it compiles as before. `$in: []` / `$nin: []`, falsy list members (`0`, `''`, `false`), every non-null ordering comparand, a `{ $field }` in an ordering slot, and `null` in the equality slot (the has-no-value predicate) all compile as before. The comparand-TYPE face is not run on this door. An `undefined` comparand outside a `$between` endpoint keeps this package's own refusal.

packages/rest/src/analytics-filter-refusal-envelope.test.ts

@@ -194,9 +194,17 @@ describe('[#5352] POST /analytics/dataset/query — a filter refusal reaches the
       message: /carries a field constraint with zero operators/,
     },
     {
+      // [#20010] RE-WORDED, same verdict: still a real normalizer refusal,
+      // still 400 INVALID_FILTER through this seam. PR #20032 hands every field
+      // entry of an object-form `where` to the shared comparand-shape face
+      // (`assertListComparandShapes`) before any node is built, and the face's
+      // own rule answers first — a list operator takes a list, `$between` a
+      // two-element [min, max] (#5869, moved to the face by #9228). So the
+      // message is the face's, the same bytes the FilterArray spelling gets;
+      // the analytics "needs a two-element" sentence is no longer reached.
       name: 'a $between with one bound',
       runtimeFilter: { amount: { $between: [10] } },
-      message: /needs a two-element \[min, max\] array/,
+      message: /Operator "\$between" on field "amount" requires a \[min, max\] value array/,
     },
     {
       name: 'an unsupported top-level operator',

packages/services/service-analytics/src/__tests__/comparand-door-single-source.test.ts

@@ -124,9 +124,16 @@ const MATRIX: readonly Row[] = [
   { label: 'boolean', value: true,
     bindable: true, renderable: true,
     whereLike: OK, whereIn: OK, whereEq: OK, scopeLike: OK, scopeIn: OK, scopeEq: OK },
+  // [#20010] One cell of this row moved AFTER the #8186 measurement, on
+  // purpose: the `where` door now hands every field entry to the shared
+  // comparand-shape face, whose 2026-08-31 ruling (#13357) refuses "a `null`
+  // member of `$in` / `$nin`" at that door, for every driver. The object
+  // spelling used to compile it to `status IN (NULL)`; the FilterArray
+  // spelling was already refused (`where-face-arms-refusal.test.ts`). The
+  // read-scope cell is that door's own and is not this card's.
   { label: 'null', value: null,
     bindable: true, renderable: true,
-    whereLike: OK, whereIn: OK, whereEq: OK, scopeLike: OK, scopeIn: OK, scopeEq: OK },
+    whereLike: OK, whereIn: REFUSED_WHERE, whereEq: OK, scopeLike: OK, scopeIn: OK, scopeEq: OK },
   { label: 'Date', value: new Date('2026-01-01T00:00:00.000Z'),
     bindable: true, renderable: true,
     whereLike: OK, whereIn: OK, whereEq: OK, scopeLike: OK, scopeIn: OK, scopeEq: OK },
@@ -266,7 +273,13 @@ describe('[#8186] the comparand matrix is unchanged by the door reconciliation',
       'string', 'number', 'bigint', 'boolean', 'null', 'Date',
     ]);
     for (const row of doorTypes) {
-      for (const cell of [row.whereLike, row.whereIn, row.whereEq,
+      // [#20010] ONE exception, and it is not a TYPE verdict: `null` is an
+      // accepted comparand type, but not as an `$in` MEMBER — the SHAPE face's
+      // 2026-08-31 carve-out, which this door now runs. Every other position
+      // of every accepted type still accepts.
+      const whereIn = row.label === 'null' ? REFUSED_WHERE : OK;
+      expect(row.whereIn, `${row.label} $in`).toBe(whereIn);
+      for (const cell of [row.whereLike, row.whereEq,
                           row.scopeLike, row.scopeIn, row.scopeEq]) {
         expect(cell, row.label).toBe(OK);
       }

packages/services/service-analytics/src/__tests__/comparand-shape-refusal.test.ts

@@ -169,9 +169,21 @@ describe('[#5234] the analytics `where` door refuses an uncompilable comparand',
       expect(tree({ status: { $in: ['a', 'b'] } })).toEqual({
         kind: 'leaf', member: 'status', operator: 'in', values: ['a', 'b'],
       });
-      expect(tree({ status: { $in: ['a', null, 5, true] } })).toEqual({
-        kind: 'leaf', member: 'status', operator: 'in', values: ['a', null, 5, true],
+      expect(tree({ status: { $in: ['a', 5, true] } })).toEqual({
+        kind: 'leaf', member: 'status', operator: 'in', values: ['a', 5, true],
       });
+      // [#20010] RE-JUDGED. This list carried a `null` member, pinned as a
+      // legitimate `$in` member TYPE for this guard. It still is one for this
+      // guard, which asks only "can it bind". But the shared comparand-shape
+      // face refuses a null list member by its 2026-08-31 ruling (#13357: "A
+      // `null` member of `$in` / `$nin` … refused at this door"), and this door
+      // now runs that face first. So the list is refused whole, in the face's
+      // words and not this guard's.
+      const err = refusalOf(() => tree({ status: { $in: ['a', null, 5, true] } }));
+      expect(err.code).toBe('INVALID_FILTER');
+      expect(err.status).toBe(400);
+      expect(err.message.startsWith('Operator "$in" on field "status" does not accept null as a list member')).toBe(true);
+      expect(err.message).not.toContain('cannot be bound as a SQL parameter');
     });
 
     it('keeps every primitive LIKE comparand, including the two #5526 pinned', () => {

packages/services/service-analytics/src/__tests__/cross-field-reference-refusal.test.ts

@@ -286,23 +286,38 @@ describe("[#7598] the #5222 corpus's REFUSAL arm — routed, or refused at this
       .toContain('StringOperatorSchema');
     expect(refusalOf(() => tree({ amount: { $in: [{ $field: 'budget' }, 1] } })).message)
       .toContain('cannot be bound as a SQL parameter');
+    // [#20010] RE-JUDGED: the `$between` endpoint is now answered one step
+    // earlier, by the shared comparand-shape face this door hands every field
+    // entry to, in the face's words (its #7596 endpoint arm, enforced on the
+    // face itself). Still a third gate with its own wording, which is this case's
+    // point; the FilterArray spelling gets the same bytes.
     expect(refusalOf(() => tree({ amount: { $between: [{ $field: 'budget' }, 100] } })).message)
-      .toContain('may not be a field reference on any backend');
+      .toContain('does not accept a { "$field": … } reference as an endpoint');
     // …and the scalar position is not refused at all here any more.
     expect(tree({ amount: { $gt: { $field: 'budget' } } })).toEqual({
       kind: 'leaf', member: 'amount', operator: 'gt', values: [{ $field: 'budget' }],
     });
   });
 
-  it('the `$between` refusal names the laundering it prevents, not a bind failure', () => {
+  it('the `$between` refusal names the endpoint and the two-bound spelling, not a bind failure', () => {
     // The repair a `$between` author needs is different from the one a
     // read-scope author needs, so the two sentences are different (#5240 in the
     // direction that separates rather than merges).
+    //
+    // [#20010] RE-JUDGED. This pinned this door's own sentence, which named the
+    // laundering (`index 1`, `#7596`). The door now hands every field entry to
+    // the shared comparand-shape face before any leaf is built, and the face's
+    // endpoint arm answers first — the 2026-08-11 ruling (#7596), as the face
+    // records it: "The reference stays legal in the four ORDERING slots …
+    // the alternative this refusal prescribes". Same verdict, the face's words,
+    // which are the words the FilterArray spelling already got on this door.
+    // What the case protects is unchanged: the side is named, and the message
+    // points at the spelling that IS served.
     const err = refusalOf(() => tree({ amount: { $between: [0, { $field: 'budget' }] } }));
     expect(err.code).toBe('INVALID_FILTER');
     expect(err.status).toBe(400);
-    expect(err.message).toContain('index 1');
-    expect(err.message).toContain('#7596');
+    expect(err.message).toContain('at where.amount.$between[1], the MAX bound');
+    expect(err.message).not.toContain('cannot be bound');
     // It points at the spelling that IS served, rather than at "use a literal"
     // alone — the capability exists one operator away.
     expect(err.message).toContain('"$gte"');

packages/services/service-analytics/src/__tests__/filter-normalizer-undefined-comparand.test.ts

@@ -178,14 +178,18 @@ const MEASURED: Array<{ name: string; where: unknown; path: string; wasReadAs: s
  * refused as a whole before this gate runs, as the twin always did, and the row
  * moved to `where-equality-slot-list-refusal.test.ts` ("a list is diagnosed as
  * the list, not by one of its members").
+ *
+ * [#20010] And a third: a `$between` bound, `{d: {$between: [undefined, 5]}}`
+ * → `"d".$between[0]`. The door now hands every field entry to the shared
+ * comparand-shape face before this gate runs, and the face's 2026-09-20
+ * endpoint ruling (#19071) names this exact spelling: "`''` and `undefined`
+ * are refused, naming the blank side (MIN / MAX and the index)". So the bound
+ * is still refused, INVALID_FILTER / 400, but in the face's words, which are
+ * the words the FilterArray spelling already got; the row moved to
+ * `where-face-arms-refusal.test.ts`. Every other position below keeps this
+ * gate's sentence: the shape face judges `undefined` only as a range endpoint.
  */
 const BEYOND_THE_TABLE: Array<{ name: string; where: unknown; path: string; wasReadAs: string }> = [
-  {
-    name: "a $between bound — lowered to the leaf's comparand",
-    where: { d: { $between: [undefined, 5] } },
-    path: '"d".$between[0]',
-    wasReadAs: 'd gte [null] AND d lte [5]',
-  },
   {
     name: 'a member of a $nin list',
     where: { d: { $nin: [undefined] } },
@@ -238,38 +242,20 @@ const NULL_CONTROL: Array<{ name: string; where: unknown; tree: unknown }> = [
     where: { d: { $ne: null } },
     tree: { kind: 'leaf', member: 'd', operator: 'set', values: [] },
   },
-  {
-    name: '{$gt: null} → a real comparison binding NULL (#5526)',
-    where: { d: { $gt: null } },
-    tree: { kind: 'leaf', member: 'd', operator: 'gt', values: [null] },
-  },
-  {
-    name: '{$in: [null]} → a list member, not a predicate',
-    where: { d: { $in: [null] } },
-    tree: { kind: 'leaf', member: 'd', operator: 'in', values: [null] },
-  },
-  {
-    name: '{$nin: [null]} → null-safe guarded (#5298)',
-    where: { d: { $nin: [null] } },
-    tree: {
-      kind: 'or',
-      children: [
-        { kind: 'leaf', member: 'd', operator: 'notSet', values: [] },
-        { kind: 'leaf', member: 'd', operator: 'notIn', values: [null] },
-      ],
-    },
-  },
-  {
-    name: '{$between: [null, 5]} → lowered to two bounds',
-    where: { d: { $between: [null, 5] } },
-    tree: {
-      kind: 'and',
-      children: [
-        { kind: 'leaf', member: 'd', operator: 'gte', values: [null] },
-        { kind: 'leaf', member: 'd', operator: 'lte', values: [5] },
-      ],
-    },
-  },
+  // [#20010] RE-JUDGED. Four rows stood here: `{$gt: null}` → `gt [null]`
+  // (#5526's by-construction reading), `{$in: [null]}` → `in [null]`,
+  // `{$nin: [null]}` → the #5298-guarded `notIn [null]`, and
+  // `{$between: [null, 5]}` → `gte [null] AND lte [5]`. Each is a position the
+  // shared comparand-shape face refuses by ruling, for every driver: the null
+  // list member and the null `$between` endpoint by the 2026-08-31 ruling
+  // (#13357), the null ordering comparand by the 2026-09-01 ruling (#14080) —
+  // a position the face's docblock records #5332's landing had called one
+  // "no ruling covers", so #5526's reading of it was a consequence of deleting
+  // an encoder, never a ruling of this door's. The door now runs that
+  // face, so each shape is refused whole (INVALID_FILTER / 400, in the face's
+  // words, `where-face-arms-refusal.test.ts`) and is no longer an accepted
+  // position for this group to hold still. What the group protects is
+  // unchanged: none of the four is refused as an UNDEFINED comparand.
   {
     name: "{$contains: null} → LIKE '%null%' (#5526)",
     where: { d: { $contains: null } },

packages/services/service-analytics/src/__tests__/filter-operator-coverage.test.ts

@@ -196,8 +196,11 @@ describe('analytics filters — every authorable operator reaches the query (#41
   });
 
   it('a malformed $between throws rather than binding a half-open guess', () => {
+    // [#20010] The shared comparand-shape face answers this one first now, in
+    // its own words (the words the FilterArray spelling gets); this door's
+    // "two-element" sentence is no longer reached from `normalizeAnalyticsFilterTree`.
     expect(() => normalizeAnalyticsFilterTree({ where: { score: { $between: [10] } } })).toThrow(
-      /two-element/,
+      /requires a \[min, max\] value array/,
     );
   });
 });

packages/services/service-analytics/src/__tests__/filter-refusal-envelope.test.ts

@@ -94,9 +94,16 @@ const REFUSALS: Array<{
     issueBullet: true,
   },
   {
+    // [#20010] RE-WORDED, same input, same verdict, same envelope. The door now
+    // hands every field entry to the shared comparand-shape face before any
+    // leaf is built, and the face's arity arm answers first, in the words the
+    // FilterArray spelling already got. `fieldLeaves`' own "needs a
+    // two-element" check stays as that function's invariant and is no longer
+    // reached from this door. The row keeps #5352's bullet: the authoring
+    // mistake is the one #5352 named.
     name: '$between without exactly two bounds',
     where: { amount: { $between: [10] } },
-    message: /needs a two-element \[min, max\] array/,
+    message: /Operator "\$between" on field "amount" requires a \[min, max\] value array/,
     issueBullet: true,
   },
   // FLIPPED with the #5322 ruling: these two entries were "$and/$or with an