fix(spec)!: refuse an array under $ne at the shared comparand-shape face and at FieldOperatorsSchema.$ne

.changeset/19886-ne-array-comparand-refused-at-face-and-slot.md

@@ -0,0 +1,71 @@
+---
+"@objectstack/spec": minor
+---
+
+fix(spec)!: an ARRAY under `$ne` is refused at the shared comparand-shape face and at `FieldOperatorsSchema.$ne`, with one remedy text naming `$nin` (#19886)
+
+**BREAKING** — an accept-set narrowing at the runtime filter doors and at one published operator slot, shipped as `minor` under the repo's launch-window convention for accept-set narrowings. Ruled on #19886 (record 5805254639, ruling A, class 1): "The shared comparand-shape face refuses an array under `$ne` for every driver, and `FieldOperatorsSchema.$ne` refuses it at parse — one remedy text, naming the declared list-negation operator by its spec spelling". No alias, no grace window. The hand-migration prescription is registered under protocol major 18 as `filter-ne-array-comparand-refused`.
+
+## What changes
+
+`$ne` already declared its comparand as "a literal, or a { $field } reference to another column of the same table". An array is neither. Two doors now say so.
+
+**The shared comparand-shape face** (`assertListComparandShapes` in `@objectstack/spec/data`) refuses `{ field: { $ne: [...] } }` with `INVALID_FILTER` / 400. It refuses it at any depth under `$and` / `$or` / `$not`, and it refuses the empty array too. `parseFilterAST` lowers `['tags', 'ne', ['a']]` to that shape, and so do `!=`, `<>`, `neq`, `not_equals` and `notequals`. The face runs inside `parseFilterAST` and at the engine's lowering seam, so the refusal lands before any driver runs. The message is:
+
+```text
+Operator "$ne" on field "tags" requires a single comparable value, but received an array (["a"]) at where.tags.$ne. For "none of these values" use {"$nin": […]} (authoring: nin, not_in, notin). The filter was NOT applied, and an unapplied filter would have returned the UNFILTERED result set.
+```
+
+Its first sentence is `driver-memory`'s own wording for this condition. It names one remedy: `$nin`, the list-negation operator `FieldOperatorsSchema` declares ("Not in list"), with its authoring spellings.
+
+**The operator slot** `FieldOperatorsSchema.$ne` refuses an array on parse. So do its documentation copy `EqualityOperatorSchema.$ne` and the `NormalizedFilter` AST that validates against it. They print the same sentence without ` on field "…"` and ` at <path>`, because a slot cannot see either. The issue's own `path` carries the location instead (`$ne`, `$and.0.stage.$ne`).
+
+How each door answered `{ tags: { $ne: ['a'] } }` on `origin/main` `9e7824a4`, just before this change:
+
+| door | before | after |
+|:--|:--|:--|
+| the shared face, `parseFilterAST`, the engine's lowering seam | **passed** it to the driver, at every depth | `INVALID_FILTER` / 400, before any driver runs |
+| `FieldOperatorsSchema`, `EqualityOperatorSchema`, the `NormalizedFilter` AST | **parsed it green** | refused on parse at `$ne` |
+| the analytics `where` door (`normalizeAnalyticsFilterTree`) | **compiled it**: `{ stage: { $ne: ['won', 'lost'] } }` became `stage` not-set OR `stage` not-equals `['won', 'lost']`, and both analytics strategies render a not-equals member from its first value, so `'lost'` was dropped and its rows were counted | `INVALID_FILTER` / 400 with the face's sentence |
+| `driver-sql`, `driver-memory` | refused, 400, in their own words | unchanged; the face now answers first |
+| `driver-mongodb`, the formula evaluator | refused, since earlier stages of this card, in their own words | unchanged; the face now answers first |
+
+The analytics row's compiled tree was measured with the face's `$ne` arm removed, which is `main`'s face. Its rendering from the first value was read at source (`values[0]` in the native SQL strategy, `v0` in the ObjectQL strategy) and was not executed. A live `mongod`, MySQL, PostgreSQL and a live Turso server were NOT measured.
+
+So on the SQL family and `driver-memory` the verdict does not move (400 before and after). What moves is the text and the moment: the refusal now arrives at the face with the `$nin` remedy, before any driver runs. On the analytics `where` door the verdict does move: a query that used to answer with a silently widened row set is now refused.
+
+## What does NOT change
+
+- **A stored filter carrier still saves the shape.** `FilterConditionSchema`, which every stored filter parses through (a dataset filter and measure filter, a dashboard widget filter, a report `runtimeFilter`, a rollup filter and the rest), does not parse a field's operator map through `FieldOperatorsSchema`. Its own walk judges `$eq` and does not judge `$ne`. The ruling names the face and the operator slot, not that walk. So `DatasetSchema` with `filter: { stage: { $ne: ['won', 'lost'] } }` still parses green, and every query that uses it is refused at the face.
+- `$ne: null` is the has-a-value predicate and is untouched. Every scalar, a `Date` and a `{ $field }` reference are untouched too. `['amount', '!=', { $field: 'budget' }]` still lowers to `{ amount: { $ne: { $field: 'budget' } } }` and passes.
+- The list operators keep their arrays: `$in`, `$nin` and `$between`, including `$in: []` and `$nin: []`.
+- A field spec with no `$` key (`{ author: { tags: { $ne: ['a'] } } }`) is still not descended into.
+- The ordering operators carrying an array, a nested array inside `$in`, and a `{ $field }` referent to a multi-valued field are not judged here. They are the same class and are scoped separately on the card.
+- The drivers are untouched. A caller that hands a raw `FilterCondition` straight to a driver, without `parseFilterAST`, still gets that driver's own refusal.
+- `ViewFilterRule` already refused an array on `not_equals` at authoring time.
+- The published JSON Schema cannot state the check. `z.toJSONSchema()` has no projection for it, so `data/FieldOperators`, `data/EqualityOperator` and `data/NormalizedFilter` still read `{}` at `$ne`. The three sites are declared in `dropped-refinements.baseline.json`.
+- No key is added, removed or renamed, no exported symbol moves, and the operator vocabulary is unchanged.
+
+## FROM → TO
+
+| you wrote | write instead |
+|:--|:--|
+| `{ stage: { $ne: ['won', 'lost'] } }`, meaning "none of these values" | `{ stage: { $nin: ['won', 'lost'] } }` |
+| `[['stage', 'not_equals', ['won', 'lost']]]` (or `ne`, `!=`, `<>`, `neq`, `notequals`) | `[['stage', 'not_in', ['won', 'lost']]]` (or `nin`, `notin`) |
+| `{ stage: { $ne: ['won'] } }`, meaning one value | `{ stage: { $ne: 'won' } }` |
+
+On `driver-mongodb`, check what the query is supposed to return and do not assume the old rows were right. Before this card, MongoDB read `$ne` against an array as "not equal to that array and not holding it as an element", and `$nin` does not reproduce that. On the analytics `where` door, the old answer dropped every member after the first, so a chart built on it counted rows its filter named.
+
+## Who is affected, measured
+
+Nothing shipped authors the shape. Each count below was read against the named tree, with a control:
+
+- **This repository**, `origin/main` `9e7824a4`. `$ne` followed by an array literal in `packages/**`, `examples/**`, `apps/**`, `scripts/**`, `content/**` and `skills/**` has 20 hits. All of them are tests, refusal code, comments or migration prose. The control, `$in` followed by an array literal in `packages/**` and `examples/**`, has 901 hits. The FilterArray triple on `ne`, `!=`, `<>`, `neq`, `not_equals` or `notequals` carrying an array has 0 hits. A CEL `!=` against a list literal in `packages/platform-objects/**`, `examples/**` and `packages/qa/**` (tests excluded) has 0 hits. `$ne` fed by a variable in non-test source has 11 sites, and none of them builds a list. Each passes a list's first member, a stored-form value, a named constant, a `{ $field }` reference, or a caller's comparand passed through: the better-auth adapter's `ne`, and the CEL lowering, which refuses a list under `!=` before it emits.
+- **objectui** at the `.objectui-sha` pin `f8a9d0fb05`: 2 hits, both in its own refusal test. The dataset filter builder gives `notEquals` a scalar arity, and the rollup summary editor gives only `in` / `notIn` a list. The control, `$in` with an array, has 46 hits.
+- **cloud** `main` `48d7066`: 0 hits. The control has 33 hits.
+
+Deployed stacks were NOT measured. Every query that carries the shape is refused with `INVALID_FILTER` / 400 naming the field, the path and `$nin`, so a test suite that exercises the query finds each one. A clean re-save of a stored carrier does not find them, because the carrier schema still accepts the shape. Exercise stored filters, or grep them.
+
+Clause-②: no (narrowing) — nothing is widened. One comparand shape in one slot, which `$ne`'s published description already excluded, is refused at the face and at the operator slot.
+
+<!-- adr-0087: registered filter-ne-array-comparand-refused -->

packages/services/service-analytics/src/__tests__/where-equality-slot-list-refusal.test.ts

@@ -201,8 +201,23 @@ describe('[#19888] the neighbouring shapes compile exactly as before', () => {
     });
   });
 
-  it('$ne with a list is not judged by this gate (not ruling 乙’s)', () => {
-    expect(() => tree({ stage: { $ne: ['won'] } })).not.toThrow(/requires a single comparable value/);
+  it('$ne with a list is not judged by this gate (not ruling 乙’s) — the shared face refuses it in the $ne sentence', () => {
+    // [#19886] RE-JUDGED. This pinned `.not.toThrow(/requires a single
+    // comparable value/)`, which was true while no face judged `$ne`. Ruling A
+    // on #19886 (record 5805254639) put the refusal on the shared face, and
+    // this door runs that face, so the shape is refused here with the face's
+    // own `$ne` sentence. What the row protects is unchanged: THIS door's
+    // equality-slot gate does not claim `$ne`, so the words are the `$ne`
+    // arm's (`$nin` remedy), never the equality slot's (`$in` remedy).
+    const where = { stage: { $ne: ['won'] } };
+    const err = refusalOf(() => tree(where));
+    expectEnvelope(err);
+    expect(err.message).toBe(refusalOf(() => assertListComparandShapes(where)).message);
+    expect(err.message.startsWith('Operator "$ne" on field "stage" requires a single comparable value')).toBe(true);
+    expect(err.message).toContain('For "none of these values" use {"$nin": […]}');
+    expect(err.message).not.toContain(REMEDY);
+    expect(err.message.startsWith(EQ('stage'))).toBe(false);
+    expect(err.message.startsWith(IMPLICIT('stage'))).toBe(false);
   });
 });
 

packages/spec/dropped-refinements.baseline.json

@@ -3,7 +3,7 @@
   "measured": {
     "zod": "4.4.3",
     "publishedSchemasWithDroppedRefinements": 207,
-    "droppedRefinementSites": 574,
+    "droppedRefinementSites": 577,
     "refinementSitesThatDidProject": 369,
     "refinementSitesWithNoJsonFormToCompare": 0
   },
@@ -607,7 +607,8 @@
     },
     "data/EqualityOperator": {
       "sites": [
-        "$eq"
+        "$eq",
+        "$ne"
       ]
     },
     "data/Field": {
@@ -623,6 +624,7 @@
         "$between.items[1]",
         "$eq",
         "$in",
+        "$ne",
         "$nin"
       ]
     },
@@ -689,6 +691,7 @@
         "lazy.$not.options[0].valueType.$between.items[1]",
         "lazy.$not.options[0].valueType.$eq",
         "lazy.$not.options[0].valueType.$in",
+        "lazy.$not.options[0].valueType.$ne",
         "lazy.$not.options[0].valueType.$nin"
       ]
     },

packages/spec/src/data/filter-comparand-refusal-text.ts

@@ -16,6 +16,11 @@
  * `LIST_COMPARAND_OPERATORS` docblock records. A module both import breaks the
  * tie. It imports nothing.
  *
+ * [#19886] The `$ne` slot is the second tenant, for the same reason. Ruling A
+ * (record 5805254639) refuses an array under `$ne` at the shared face and at
+ * `FieldOperatorsSchema.$ne` with "one remedy text", so that sentence is
+ * assembled here too, by {@link arrayInequalityComparandMessage}.
+ *
  * ## Why it is NOT in the `data` barrel
  *
  * Nothing here is a contract a consumer calls; the two refusals are, and they
@@ -37,6 +42,18 @@
  */
 export const IN_OPERATOR_SPELLINGS: readonly string[] = ['in'];
 
+/**
+ * [#19886] The authoring spellings that lower to `$nin`.
+ *
+ * `$nin` is the list-negation operator `FieldOperatorsSchema` declares ("Not
+ * in list"), and it is what the `$ne` refusal prescribes. So, as with
+ * {@link IN_OPERATOR_SPELLINGS}, the face's `LIST_COMPARAND_OPERATORS` reads its
+ * `$nin` row from here, and the refusal of a malformed `$nin` and the `$ne`
+ * refusal (which prescribes `$nin`) name one spelling list.
+ * `filter-comparand-shape.test.ts` reconciles the list against `AST_OPERATOR_MAP`.
+ */
+export const NIN_OPERATOR_SPELLINGS: readonly string[] = ['nin', 'not_in', 'notin'];
+
 /**
  * [#19757] The ARRAY-CONTAINMENT operator the equality-slot refusal prescribes,
  * with its authoring spelling: `$contains`, which `FieldOperatorsSchema`
@@ -135,11 +152,86 @@ export function arrayEqualityComparandMessage(
   site: ArrayEqualityComparandSite,
 ): string {
   const position = site.op
-    ? `Operator "${site.op}"${site.field === undefined ? '' : ` on field "${site.field}"`}`
+    ? operatorPosition(site.op, site.field)
     : `The implicit-equality comparand on field "${site.field}"`;
-  const location = site.path === undefined ? '' : ` at ${site.path}`;
+  return singleValueSlotMessage(position, value, site.path, ARRAY_EQUALITY_COMPARAND_REMEDY);
+}
+
+/**
+ * [#19886] The ONE remedy for an array under `$ne`: the declared list-negation
+ * operator, by its spec spelling and its authoring spellings, then the sentence
+ * a caller cannot infer from a status code.
+ *
+ * Ruling A on #19886 (record 5805254639) asks for exactly this: "one remedy
+ * text, naming the declared list-negation operator by its spec spelling", read
+ * off `FieldOperatorsSchema`, where `$nin` is the operator declared as "Not in
+ * list". ⛔ Nothing else is offered. `$notContains` is declared on a STRING
+ * comparand and is not a list operator, and a second remedy here would be an
+ * invented one. `$nin` is also the remedy the formula evaluator and
+ * `driver-mongodb` already name for the same shape.
+ *
+ * Written like {@link ARRAY_EQUALITY_COMPARAND_REMEDY}: no field wrapper and
+ * `…` for the value, because the field is the subject of the sentence in front
+ * of this one.
+ */
+export const ARRAY_INEQUALITY_COMPARAND_REMEDY: string =
+  `For "none of these values" use {"$nin": […]} (authoring: ${NIN_OPERATOR_SPELLINGS.join(', ')}). ` +
+  `The filter was NOT applied, and an unapplied filter would have returned the ` +
+  `UNFILTERED result set.`;
+
+/**
+ * [#19886] Where the refused array under `$ne` sits, as far as the calling door
+ * can see. The same rules as {@link ArrayEqualityComparandSite}'s `$eq` arm:
+ * `FieldOperatorsSchema.$ne` judges one field's operator map and cannot see the
+ * field, and the face passes `path` because a zod refinement cannot know it.
+ */
+export type ArrayInequalityComparandSite = { readonly field?: string; readonly path?: string };
+
+/**
+ * [#19886] The refusal of an ARRAY under `$ne`, as both doors print it — the
+ * shared comparand-shape face and `FieldOperatorsSchema.$ne`.
+ *
+ * The leading sentence is `driver-memory`'s `arrayComparandError` for the same
+ * operator, word for word, so one condition keeps one wording across packages
+ * (#5240's rule). Then the received list, bounded by {@link shapePreview}, the
+ * location when the door knows it, and {@link ARRAY_INEQUALITY_COMPARAND_REMEDY}.
+ *
+ * As with {@link arrayEqualityComparandMessage}, the two doors print the same
+ * characters for a given value, except that the face adds ` at <path>` and names
+ * the field. `filter-ne-array-schema-door.test.ts` pins exactly that.
+ */
+export function arrayInequalityComparandMessage(
+  value: unknown,
+  site: ArrayInequalityComparandSite,
+): string {
+  return singleValueSlotMessage(
+    operatorPosition('$ne', site.field),
+    value,
+    site.path,
+    ARRAY_INEQUALITY_COMPARAND_REMEDY,
+  );
+}
+
+/** `Operator "$op"`, plus ` on field "f"` when the door can see the field. */
+function operatorPosition(op: string, field: string | undefined): string {
+  return `Operator "${op}"${field === undefined ? '' : ` on field "${field}"`}`;
+}
+
+/**
+ * The sentence both single-value-slot refusals share: the position, the words
+ * `driver-memory` uses for this condition, the received list, the location and
+ * the slot's own remedy. One template, so the `$eq` and `$ne` refusals cannot
+ * drift apart in their first sentence.
+ */
+function singleValueSlotMessage(
+  position: string,
+  value: unknown,
+  path: string | undefined,
+  remedy: string,
+): string {
+  const location = path === undefined ? '' : ` at ${path}`;
   return (
     `${position} requires a single comparable value, but received an array ` +
-    `(${shapePreview(value)})${location}. ${ARRAY_EQUALITY_COMPARAND_REMEDY}`
+    `(${shapePreview(value)})${location}. ${remedy}`
   );
 }