diff --git a/.changeset/19757-equality-slot-array-refused.md b/.changeset/19757-equality-slot-array-refused.md index 5cd08308867..efd175fece0 100644 --- a/.changeset/19757-equality-slot-array-refused.md +++ b/.changeset/19757-equality-slot-array-refused.md @@ -32,7 +32,7 @@ After this change, every row above that goes through a platform door gets the 40 - Every scalar equality comparand is untouched. That includes `null`: `{ field: null }` and `{ field: { $eq: null } }` are the has-no-value predicate. - A `{ $field }` reference on an equality spelling still lowers to `$eq` and passes. - A field spec with no `$` key (`{ author: { tags: ['a'] } }`) is still not descended into. -- The schema doors are not touched. `FilterConditionSchema` still parses `{ field: [...] }`, because `FieldOperatorsSchema.$eq` is `z.any()`. A document carrying the shape therefore still publishes, and is refused when it is queried. +- This change does not touch the schema doors. A separate change in this release does: `FilterConditionSchema` and `FieldOperatorsSchema.$eq` now refuse the same shape when a document is saved, with this refusal's sentence (the location is carried by the issue's path instead). Its changeset and the ADR-0087 entry `filter-equality-array-comparand-refused-at-save` describe it. - `ViewFilterRule` already refused an array on every scalar view operator at authoring time. ## FROM → TO diff --git a/.changeset/19888-analytics-implicit-array.md b/.changeset/19888-analytics-implicit-array.md index 0bc913fb8da..d819b051478 100644 --- a/.changeset/19888-analytics-implicit-array.md +++ b/.changeset/19888-analytics-implicit-array.md @@ -27,4 +27,4 @@ Ruling 乙 of #19757 refuses a list in the equality slot at the shared comparand Each list is now handed to the shared face's equality arm before any node is built. Both spellings of one condition therefore get the same refusal, with the same wording, path and `$in` prescription. The draft-data preview runs the same gate, so a drafted chart refuses what the published chart refuses. Before, it compared each row against the list's string form. -Who is affected: nothing in this repository's examples, seeds or docs authors the shape (measured over `examples/**`, `packages/**` and the fenced code in the docs). Stored datasets, dashboard widget filters, report runtime filters and measure filters in a deployment were NOT measured. The authoring schema still admits the shape, so such a document still publishes, and it is refused when it is charted. The refusal names the field and the path. `$ne` with a list is not part of the ruling and is not judged here. The list operators (`$in`, `$nin`, `$between`) keep their lists, and every scalar, `null` included, compiles as before. +Who is affected: nothing in this repository's examples, seeds or docs authors the shape (measured over `examples/**`, `packages/**` and the fenced code in the docs). Stored datasets, dashboard widget filters, report runtime filters and measure filters in a deployment were NOT measured. In this release the authoring schema refuses the shape too, when such a document is saved (a separate change in `@objectstack/spec`, ADR-0087 entry `filter-equality-array-comparand-refused-at-save`). One position is judged only here: a list inside a nested relation, which this door flattens to a dotted member. A document carrying that still publishes, and it is refused when it is charted. The refusal names the field and the path. `$ne` with a list is not part of the ruling and is not judged here. The list operators (`$in`, `$nin`, `$between`) keep their lists, and every scalar, `null` included, compiles as before. diff --git a/.changeset/19889-filter-schema-door-array-equality.md b/.changeset/19889-filter-schema-door-array-equality.md new file mode 100644 index 00000000000..d27f04252d3 --- /dev/null +++ b/.changeset/19889-filter-schema-door-array-equality.md @@ -0,0 +1,46 @@ +--- +"@objectstack/spec": minor +--- + +fix(spec)!: a filter carrying an array in the equality slot is refused when it is saved, in the query face's own words (#19889) + +**BREAKING** — an accept-set narrowing of published authoring schemas, shipped as `minor` under the repo's launch-window convention for accept-set narrowings. Ruled on #19889 (record 5805248669, letter A). The hand-migration prescription is registered under protocol major 18 as `filter-equality-array-comparand-refused-at-save`. + +## What changes + +`FilterConditionSchema` now refuses an ARRAY in the equality slot at parse: the implicit form `{ field: [...] }` and the explicit form `{ field: { $eq: [...] } }`, the empty array included, at any depth under `$and` / `$or` / `$not`. `FieldOperatorsSchema.$eq` refuses an array comparand too, and so do its documentation copy `EqualityOperatorSchema.$eq` and the `NormalizedFilter` AST that validates against it. + +The comparand-shape face (`assertListComparandShapes`) refuses both shapes on every query, from the equality-slot change earlier in this release. Before this change the schema accepted them, so a dataset or dashboard filter carrying one published clean and then failed every query that used it. Measured on `origin/main` `a0920b42dc`: `DatasetSchema.safeParse` with `filter: { stage: ['won', 'lost'] }` answered `success: true`, and so did a measure `filter` of `{ stage: { $eq: ['won', 'lost'] } }`. + +The schema door prints the face's sentence: the field, the received list, and the two operators a list in that slot stood in for. Both doors import it from one builder. The face adds `at where.`. The schema door leaves that out, because the issue's `path` already says where it is (`filter.stage`, `measures.0.filter.stage.$eq`). + +Every schema that carries a `FilterCondition` refuses it on parse. That covers the dataset `filter` and measure `filter`, the dashboard widget `filter` and options-source `filter`, the report and joined-report-block `runtimeFilter`, the field `relatedListFilter` and rollup `summaryOperations.filter`, the solution-blueprint summary `filter`, the analytics query `where`, the dataset selection `runtimeFilter`, the query `where` and `having`, the data-engine aggregate call's `having`, the aggregation `filter`, and the query-filter `where`. So `defineStack`, `os validate` and a save through the metadata protocol (`422 INVALID_METADATA`) refuse such a document at the filter's path. + +Two request doors parse these carriers, and they now answer before the analytics compiler does. The REST dataset selection (its `runtimeFilter`) and the analytics query body (its `where`) answer `VALIDATION_FAILED` / 400 with the sentence at the field. Before, the compiler answered `INVALID_FILTER` / 400. + +## What does NOT change + +- **Nothing stored is rewritten, and nothing is dropped.** The parse fails and strips nothing. The read path does not re-validate stored rows, so a stored document keeps loading, and its next save is refused. Such a filter has failed every query since the equality-slot change, so the refusal is a repair. +- **The reach is the face's, and no wider.** A field spec with no `$` key, such as the nested-relation condition `{ account: { region: ['a'] } }`, is not judged, because the face does not judge it either. The analytics `where` door does refuse that shape when a dataset or measure filter is charted. +- **The data-engine calls' `where` option still parses.** Its type is a union whose first arm is an open record. The face refuses the shape when the call runs. +- **`$ne` carrying an array is not judged.** +- The list operators keep their arrays, `$in: []` and `$nin: []` included. Every scalar, `null`, a `Date` and a `{ $field }` reference pass as before. +- **The published JSON Schema cannot state the check.** `z.toJSONSchema()` has no projection for it, so `data/FieldOperators`, `data/EqualityOperator` and `data/NormalizedFilter` still read `{}` at `$eq`. The three sites are declared in `dropped-refinements.baseline.json`. + +## FROM → TO + +| you wrote | write instead | +|:--|:--| +| `{ stage: ['won', 'lost'] }`, meaning "one of these values" | `{ stage: { $in: ['won', 'lost'] } }` | +| `{ tags: ['a'] }`, meaning "the stored list holds `a`" on a multi-value field | `{ tags: { $contains: 'a' } }` | +| `{ tags: ['a', 'b'] }`, meaning "the stored list holds `a` or `b`" | `{ $or: [{ tags: { $contains: 'a' } }, { tags: { $contains: 'b' } }] }` | +| `{ stage: ['won'] }`, meaning one value | `{ stage: 'won' }` | +| `{ stage: { $eq: [...] } }` | any of the rows above | + +## Who is affected, measured + +Nothing shipped in this repository carries the shape. A brace-matched scan of every filter-carrier literal (`filter`, `where`, `runtimeFilter`, `having`, `relatedListFilter`) in `packages/**`, `examples/**`, `apps/**`, `content/docs/**` and `skills/**` read 4709 literals across 7785 files and found 20 field entries whose value opens an array. Sixteen are test fixtures, and four are not filter carriers (a realtime subscription filter and a plugin-permission filter). A grep for `$eq` followed by an array found 24 lines: prose, MongoDB aggregation expressions, and one door-refusal conformance row. Deployed datasets, dashboards and reports were NOT measured. Validating each stack, or re-saving each document, finds every instance the surface above lists. + +Clause-②: no (narrowing) — nothing is widened. No key is added, removed or renamed, no exported symbol moves, and the operator vocabulary is unchanged. One comparand shape in one slot, which every query door already refused, is now refused on save as well. + + diff --git a/packages/spec/dropped-refinements.baseline.json b/packages/spec/dropped-refinements.baseline.json index e3660dc1a76..77a87e632cc 100644 --- a/packages/spec/dropped-refinements.baseline.json +++ b/packages/spec/dropped-refinements.baseline.json @@ -2,8 +2,8 @@ "description": "Shrink-only ledger of every PUBLISHED JSON Schema that is STILL WIDER than the Zod type it was generated from, because a rule written as `.refine()` reaches the runtime and not the file (#18670). `z.toJSONSchema()` has no arm for a `custom` check: a plain record, the same record with a `.refine()`, and the same record with an ABORTING `.refine()` all project byte-identically (measured on zod 4.4.3, the version packages/spec resolves). So a document one of these files ACCEPTS can still be refused at parse time, and an author -- or an AI -- validating against packages/spec/json-schema/** finds out a release later. Each `sites` path is a position under that schema at which a refinement is dropped; the same paths are written onto the artifact itself as `x-dropped-refinements`. Item 2 closed the first patterns: a refinement DECLARED through the closed list in src/shared/refinement-projection.ts is emitted into the published file, reads `projected` rather than `dropped`, and its row LEAVES this ledger in the same PR -- which is why the ledger shrinks and never grows on a repair. Every refinement outside that closed list stays here, and adding an arm to the list is a public-contract decision, not a refactor. Hand-edited on purpose and with no `gen:` script: a generator would let a new gap be admitted by running a command instead of by a decision, which is the silence this ledger exists to end. Adding, removing or moving a site fails packages/spec/scripts/build-schemas.ts until the line moves with it, and the failure prints the corrected entry in full. ⛔ Do not delete or weaken a refinement to shorten this file -- the runtime rule is correct; it is the projection that is silent, and the remedy is to teach the closed list a NAMED pattern, never to drop the rule.", "measured": { "zod": "4.4.3", - "publishedSchemasWithDroppedRefinements": 206, - "droppedRefinementSites": 571, + "publishedSchemasWithDroppedRefinements": 207, + "droppedRefinementSites": 574, "refinementSitesThatDidProject": 369, "refinementSitesWithNoJsonFormToCompare": 0 }, @@ -605,6 +605,11 @@ "where.options[1].lazy" ] }, + "data/EqualityOperator": { + "sites": [ + "$eq" + ] + }, "data/Field": { "sites": [ "", @@ -616,6 +621,7 @@ "sites": [ "$between.items[0]", "$between.items[1]", + "$eq", "$in", "$nin" ] @@ -681,6 +687,7 @@ "sites": [ "lazy.$not.options[0].valueType.$between.items[0]", "lazy.$not.options[0].valueType.$between.items[1]", + "lazy.$not.options[0].valueType.$eq", "lazy.$not.options[0].valueType.$in", "lazy.$not.options[0].valueType.$nin" ] diff --git a/packages/spec/src/data/filter-comparand-refusal-text.ts b/packages/spec/src/data/filter-comparand-refusal-text.ts new file mode 100644 index 00000000000..2c3a3694b02 --- /dev/null +++ b/packages/spec/src/data/filter-comparand-refusal-text.ts @@ -0,0 +1,145 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#19889] The words the two filter doors share — the comparand-shape FACE + * (`./filter-comparand-shape.ts`, which every query crosses) and the SCHEMA + * door (`./filter.zod.ts`, which every stored filter crosses on save). + * + * ## Why a module of its own + * + * The 2026-09-24 ruling on #19889 (letter A, record 5805248669) refuses an + * ARRAY in the EQUALITY slot at the schema door "with the SAME remedy text the + * shared compile face emits — one constant, two doors". The face already + * refused it (#19757, ruling 乙). So the text has to live where BOTH doors can + * import it, and neither door can be that place: `filter.zod.ts` imports the + * face, and the face cannot import `filter.zod.ts` back — the cycle the face's + * `LIST_COMPARAND_OPERATORS` docblock records. A module both import breaks the + * tie. It imports nothing. + * + * ## Why it is NOT in the `data` barrel + * + * Nothing here is a contract a consumer calls; the two refusals are, and they + * are already published (`assertListComparandShapes`, and the schemas + * themselves). Exporting the text would widen `@objectstack/spec/data` for no + * reader, so this file stays out of `./index.ts`, as + * `./currency-fraction-digits.ts` does. Consumers read the words where the + * refusals put them: the `Error` message at the face, the issue message at the + * schema door. + */ + +/** + * The authoring spellings that lower to `$in`. + * + * The face's `LIST_COMPARAND_OPERATORS` reads its `$in` row from here, so the + * refusal of a malformed `$in` and the equality-slot refusal (which prescribes + * `$in`) name one spelling list. `filter-comparand-shape.test.ts` reconciles + * the list against `AST_OPERATOR_MAP`. + */ +export const IN_OPERATOR_SPELLINGS: readonly string[] = ['in']; + +/** + * [#19757] The ARRAY-CONTAINMENT operator the equality-slot refusal prescribes, + * with its authoring spelling: `$contains`, which `FieldOperatorsSchema` + * declares as a MEMBERSHIP test on a `multiple: true` / JSON-stored column + * ("the stored list holds this value"). The other half of the prescription is + * `$in`, read off {@link IN_OPERATOR_SPELLINGS}. Reconciled by pin in + * `filter-comparand-shape.test.ts`. + */ +export const ARRAY_CONTAINMENT_OPERATOR = { op: '$contains', spellings: ['contains'] } as const; + +/** + * A short, bounded rendering of the offending value. + * + * Bounded because the value came off the wire and a filter comparand can be + * arbitrarily large; the message is for a human reading a 400, not a dump. + * + * The whole message has a second, harder bound: `rest-server.ts` TRUNCATES a + * declared-4xx message at `CLIENT_MESSAGE_MAX` (500) before it reaches the + * client (#5423). Everything a caller needs in order to act — operator, field, + * received value, position, corrected shape — is therefore front-loaded, and + * `filter-comparand-shape.test.ts` pins the assembled length under that bound so + * a later edit cannot silently push the tail off the wire. + * + * Moved here from the face (#19889) so the schema door renders a received list + * with the same characters the face does. + */ +export function shapePreview(value: unknown): string { + let text: string; + try { + text = JSON.stringify(value) ?? String(value); + } catch { + text = String(value); + } + return text.length > 60 ? `${text.slice(0, 59)}…` : text; +} + +/** + * The ONE remedy for an array in the equality slot: the two operators an author + * holding a list was reaching for, by their spec spellings and their authoring + * spellings, then the sentence a caller cannot infer from a status code. + * + * - `$in` — "one of these values" (authoring spelling `in`). + * - `$contains` — "the stored list holds a value" on a multi-value field + * (authoring spelling `contains`), and an `$or` of those for any-of. + * + * The corrected shapes are written WITHOUT the field wrapper and with `…` for + * the value: the field is the subject of the sentence in front of this one, + * and the 500-char client bound buys more as prescription than as a second + * echo of the field name or the received list. + * + * The closing sentence is the face's, and it is kept at the schema door too: + * a refused filter is one that was NOT applied, and the reason for refusing + * rather than dropping it is that a dropped filter shows MORE rows than the + * author asked for — the ruling's own ground for never dropping one silently. + */ +export const ARRAY_EQUALITY_COMPARAND_REMEDY: string = + `For "one of these values" use {"$in": […]} (authoring: ${IN_OPERATOR_SPELLINGS.join(', ')}); ` + + `for "the stored list holds a value" on a multi-value field, ` + + `{"${ARRAY_CONTAINMENT_OPERATOR.op}": "…"} ` + + `(authoring: ${ARRAY_CONTAINMENT_OPERATOR.spellings.join(', ')}), an $or of those for any-of. ` + + `The filter was NOT applied, and an unapplied filter would have returned the ` + + `UNFILTERED result set.`; + +/** + * Where the refused array sits, as far as the calling door can see. + * + * - `op: '$eq'` — the explicit spelling. The `field` is optional because one + * door cannot know it: `FieldOperatorsSchema.$eq` is judged inside a + * single field's operator map, whose key belongs to the enclosing record. + * - no `op` — the implicit form `{ field: [...] }`, which only exists as a + * field entry, so the field is always known. + * - `path` — the face's dotted location (`where.stage`). The schema door + * passes none: a zod refinement cannot see where it sits in the document, + * and the issue it raises carries that location as its own `path`. + */ +export type ArrayEqualityComparandSite = + | { readonly op: '$eq'; readonly field?: string; readonly path?: string } + | { readonly op?: undefined; readonly field: string; readonly path?: string }; + +/** + * The refusal of an ARRAY as an EQUALITY comparand, as both doors print it. + * + * The leading sentence is `driver-memory`'s `arrayComparandError` verbatim, so + * one condition keeps one wording across packages (#5240's rule, applied + * across packages). Then the received list, bounded by {@link shapePreview}; + * then the location when the door knows it; then + * {@link ARRAY_EQUALITY_COMPARAND_REMEDY}. + * + * For a given field, value and operator the two doors therefore print the same + * characters, with one difference: the face adds ` at ` and the schema + * door does not, because the schema door's issue carries that location as its + * own `path`. `filter-equality-array-schema-door.test.ts` pins exactly that. + */ +export function arrayEqualityComparandMessage( + value: unknown, + site: ArrayEqualityComparandSite, +): string { + const position = site.op + ? `Operator "${site.op}"${site.field === undefined ? '' : ` on field "${site.field}"`}` + : `The implicit-equality comparand on field "${site.field}"`; + const location = site.path === undefined ? '' : ` at ${site.path}`; + return ( + `${position} requires a single comparable value, but received an array ` + + `(${shapePreview(value)})${location}. ${ARRAY_EQUALITY_COMPARAND_REMEDY}` + ); +} diff --git a/packages/spec/src/data/filter-comparand-shape.test.ts b/packages/spec/src/data/filter-comparand-shape.test.ts index 9482bb5823c..777f9769542 100644 --- a/packages/spec/src/data/filter-comparand-shape.test.ts +++ b/packages/spec/src/data/filter-comparand-shape.test.ts @@ -594,15 +594,17 @@ describe('the list-comparand shape door (#5869) runs inside parseFilterAST (#922 it('LIT CONTROL — every array-valued operator the vocabulary declares keeps its array', () => { // Read off the enforced schema rather than listed: the operators whose - // declared comparand ACCEPTS an array. `$eq` / `$ne` are in that set only - // because both are `z.any()` there — `$eq` is this arm's subject and - // `$ne` is not judged at this door (see the todo below) — so the loop - // covers exactly the list operators, and a fourth array-valued operator - // added to the schema lands in it without an edit here. + // declared comparand ACCEPTS an array. `$ne` is in that set only because + // it is `z.any()` there and is not judged at this door (see the todo + // below) — so the loop covers exactly the list operators, and a fourth + // array-valued operator added to the schema lands in it without an edit + // here. [#19889] `$eq` left the set when the schema door began refusing an + // array there too (ruling A, record 5805248669), in this face's words — + // `filter-equality-array-schema-door.test.ts` pins that door. const arrayValued = Object.keys(FieldOperatorsSchema.shape).filter((op) => FieldOperatorsSchema.safeParse({ [op]: ['a', 'b'] }).success); - expect(arrayValued.sort()).toEqual(['$between', '$eq', '$in', '$ne', '$nin']); - for (const op of arrayValued.filter((o) => o !== '$eq' && o !== '$ne')) { + expect(arrayValued.sort()).toEqual(['$between', '$in', '$ne', '$nin']); + for (const op of arrayValued.filter((o) => o !== '$ne')) { expect(parseFilterAST({ tags: { [op]: ['a', 'b'] } }), op).toEqual({ tags: { [op]: ['a', 'b'] } }); } // The empty lists stay the declared predicates they are. diff --git a/packages/spec/src/data/filter-comparand-shape.ts b/packages/spec/src/data/filter-comparand-shape.ts index ef772a616c0..ff16f10303e 100644 --- a/packages/spec/src/data/filter-comparand-shape.ts +++ b/packages/spec/src/data/filter-comparand-shape.ts @@ -283,6 +283,13 @@ * (`arrayComparandError`), kept verbatim so one condition keeps one wording * across the platform (#5240's rule, applied across packages). * + * [#19889] The SCHEMA door refuses the same shape on save (ruling A, record + * 5805248669: "one constant, two doors"): `FilterConditionSchema`'s implicit + * form and `$eq`, and `FieldOperatorsSchema.$eq`. So the whole sentence — the + * position, the received list, the prescription — is assembled in + * `./filter-comparand-refusal-text.ts`, which both doors import; this door adds + * only its `at ` location and its envelope. + * * ## Refusal envelope * * Every refusal carries `code: 'INVALID_FILTER'` and `status: 400` (ADR-0112 @@ -297,6 +304,12 @@ * @see https://github.com/objectstack-ai/objectstack/issues/19757 (the equality-slot arm) */ +import { + IN_OPERATOR_SPELLINGS, + arrayEqualityComparandMessage, + shapePreview, +} from './filter-comparand-refusal-text'; + /** * The operators whose comparand `FieldOperatorsSchema` declares as a list, with * the authoring spellings that lower to each. @@ -310,8 +323,10 @@ * cycle. `filter-comparand-shape.test.ts` reconciles the two sets, so a new * membership spelling cannot land with this message left behind. */ -const LIST_COMPARAND_OPERATORS: ReadonlyMap = new Map([ - ['$in', ['in']], +const LIST_COMPARAND_OPERATORS: ReadonlyMap = new Map([ + // [#19889] The `$in` row is read from the shared refusal text, so the operator + // the equality-slot refusal prescribes and this row cannot name two lists. + ['$in', IN_OPERATOR_SPELLINGS], ['$nin', ['nin', 'not_in', 'notin']], ['$between', ['between']], ]); @@ -334,15 +349,6 @@ const ORDERING_COMPARAND_OPERATORS: ReadonlyMap = new ['$lte', ['<=', 'lte', 'less_than_or_equal', 'lessthanorequal', 'lessorequal']], ]); -/** - * [#19757] The ARRAY-CONTAINMENT operator the equality-slot refusal prescribes, - * with its authoring spelling: `$contains`, which `FieldOperatorsSchema` - * declares as a MEMBERSHIP test on a `multiple: true` / JSON-stored column - * ("the stored list holds this value"). The other half of the prescription is - * `$in`, read off {@link LIST_COMPARAND_OPERATORS}. Reconciled by pin. - */ -const ARRAY_CONTAINMENT_OPERATOR = { op: '$contains', spellings: ['contains'] } as const; - /** What a caller most likely meant when they wrote a scalar. */ const SCALAR_ALTERNATIVE: ReadonlyMap = new Map([ ['$in', '"=" ($eq)'], @@ -396,28 +402,6 @@ function describeOperand(value: unknown): string { return typeof value; } -/** - * A short, bounded rendering of the offending value. - * - * Bounded because the value came off the wire and a filter comparand can be - * arbitrarily large; the message is for a human reading a 400, not a dump. - * - * The whole message has a second, harder bound: `rest-server.ts` TRUNCATES a - * declared-4xx message at `CLIENT_MESSAGE_MAX` (500) before it reaches the - * client (#5423). Everything a caller needs in order to act — operator, field, - * received value, position, corrected shape — is therefore front-loaded, and - * the test file pins the assembled length under that bound so a later edit - * cannot silently push the tail off the wire. - */ -function shapePreview(value: unknown): string { - let text: string; - try { - text = JSON.stringify(value) ?? String(value); - } catch { - text = String(value); - } - return text.length > 60 ? `${text.slice(0, 59)}…` : text; -} /** * The wire envelope every filter refusal on the platform carries — ADR-0112 @@ -681,6 +665,10 @@ function nullOrderingComparandError( * equality spellings (`=`, `==`, `equals`, `eq`) are not listed: "the * implicit-equality comparand" names the slot, and a received list of 60 * characters leaves no room for them inside the bound. + * + * [#19889] The sentence itself is `arrayEqualityComparandMessage` + * (`./filter-comparand-refusal-text.ts`), the one the schema door prints on + * save; this door passes its `path` and wraps the envelope. */ function arrayEqualityComparandError( context: string | undefined, @@ -689,19 +677,9 @@ function arrayEqualityComparandError( path: string, op?: '$eq', ): Error { - const position = op - ? `Operator "${op}" on field "${field}"` - : `The implicit-equality comparand on field "${field}"`; - const inSpellings = LIST_COMPARAND_OPERATORS.get('$in') ?? []; return invalidFilterComparandError( context, - `${position} requires a single comparable value, but received an array ` + - `(${shapePreview(value)}) at ${path}. For "one of these values" use {"$in": […]} ` + - `(authoring: ${inSpellings.join(', ')}); for "the stored list holds a value" on a ` + - `multi-value field, {"${ARRAY_CONTAINMENT_OPERATOR.op}": "…"} (authoring: ` + - `${ARRAY_CONTAINMENT_OPERATOR.spellings.join(', ')}), an $or of those for any-of. ` + - `The filter was NOT applied, and an unapplied filter would have returned the ` + - `UNFILTERED result set.`, + arrayEqualityComparandMessage(value, op ? { op, field, path } : { field, path }), ); } diff --git a/packages/spec/src/data/filter-equality-array-schema-door.test.ts b/packages/spec/src/data/filter-equality-array-schema-door.test.ts new file mode 100644 index 00000000000..c92c307c985 --- /dev/null +++ b/packages/spec/src/data/filter-equality-array-schema-door.test.ts @@ -0,0 +1,266 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#19889] An ARRAY in the EQUALITY slot is refused at the SCHEMA door, in the + * comparand-shape face's own words. + * + * Ruled 2026-09-24 (letter A, record 5805248669): + * + * 1. `FilterConditionSchema` (implicit equality) and `FieldOperatorsSchema.$eq` + * refuse an array comparand at parse, with the SAME remedy text the shared + * compile face emits — one constant, two doors. + * 4. Pins: implicit and `$eq` arrays refused; scalars and `{ $field }` still + * pass; the parse-door message equals the compile-face message. + * + * Measured before the change, on `origin/main` `a0920b42dc`: + * `DatasetSchema.safeParse` with `filter: { stage: ['won', 'lost'] }`, and with + * a measure `filter` of `{ stage: { $eq: ['won', 'lost'] } }`, both answered + * `success: true`, and so did `FilterConditionSchema` and + * `FieldOperatorsSchema` on the bare shapes — while `assertListComparandShapes` + * refused both with `INVALID_FILTER` / 400. So a stored filter published clean + * and then failed every query that used it. + * + * ## What would make these pins worthless, and what stops it + * + * - A door that refuses EVERYTHING passes every REFUSE row. §4 runs each + * control through BOTH doors and requires both to accept, so an over-wide + * arm goes red there, and so does an arm that drifts off the face's reach. + * - "Same message" read as "similar message". §3 compares the two doors' + * strings for EQUALITY, after removing only the one clause the face adds and + * the schema door cannot know (`at `; the issue's own `path` carries + * the location instead). §3 also proves that clause is really there, so the + * removal is not vacuous. + */ + +import { describe, expect, it } from 'vitest'; + +import { StandardErrorCode } from '../api/errors.zod'; +import { DashboardSchema } from '../ui/dashboard.zod'; +import { DatasetSchema } from '../ui/dataset.zod'; +import { ReportSchema } from '../ui/report.zod'; +import { assertListComparandShapes } from './filter-comparand-shape'; +import { + EqualityOperatorSchema, + FieldOperatorsSchema, + FilterConditionSchema, + NormalizedFilterSchema, + parseFilterAST, +} from './filter.zod'; + +type Issue = { path: PropertyKey[]; message: string }; +type Parsed = { success: boolean; error?: { issues: readonly Issue[] } }; + +/** The one issue at `path` (dot-joined), failing loudly on none or several. */ +function issueAt(result: Parsed, path: string): Issue { + expect(result.success, `expected a refusal at ${path}`).toBe(false); + const issues = (result.error?.issues ?? []).filter((i) => i.path.join('.') === path); + expect( + issues, + `issues raised: ${JSON.stringify((result.error?.issues ?? []).map((i) => i.path))}`, + ).toHaveLength(1); + return issues[0]!; +} + +/** The face's refusal of `where`, failing loudly when the face accepts it. */ +function faceRefusal(where: unknown): Error & { code?: string; status?: number } { + try { + assertListComparandShapes(where); + } catch (error) { + return error as Error & { code?: string; status?: number }; + } + throw new Error(`the face accepted ${JSON.stringify(where)}`); +} + +/** Each equality-slot shape, with its zod issue path and the face's own location. */ +const REFUSED: ReadonlyArray = [ + ['implicit, two members', { stage: ['won', 'lost'] }, 'stage', 'where.stage'], + ['implicit, one member', { stage: ['won'] }, 'stage', 'where.stage'], + ['implicit, EMPTY — still an array in this slot', { stage: [] }, 'stage', 'where.stage'], + ['$eq, two members', { stage: { $eq: ['won', 'lost'] } }, 'stage.$eq', 'where.stage.$eq'], + ['$eq, EMPTY', { stage: { $eq: [] } }, 'stage.$eq', 'where.stage.$eq'], + ['implicit, under $and', { $and: [{ tags: ['a'] }] }, '$and.0.tags', 'where.$and[0].tags'], + ['implicit, under $or', { $or: [{ stage: 'won' }, { tags: ['a'] }] }, '$or.1.tags', 'where.$or[1].tags'], + ['$eq, under $not', { $not: { tags: { $eq: ['a'] } } }, '$not.tags.$eq', 'where.$not.tags.$eq'], +]; + +// --------------------------------------------------------------------------- +// §1 FilterConditionSchema refuses both spellings, at the slot's own path +// --------------------------------------------------------------------------- + +describe('#19889 §1 — FilterConditionSchema refuses an array in the equality slot', () => { + it.each(REFUSED)('refuses %s', (_label, where, issuePath) => { + const issue = issueAt(FilterConditionSchema.safeParse(where), issuePath); + expect(issue.message).toMatch(/requires a single comparable value, but received an array/); + }); + + it('refuses every equality slot in one document, not only the first', () => { + const result = FilterConditionSchema.safeParse({ stage: ['won'], owner: { $eq: ['a', 'b'] } }); + expect(result.success).toBe(false); + expect(result.error!.issues.map((i) => i.path.join('.')).sort()).toEqual(['owner.$eq', 'stage']); + }); +}); + +// --------------------------------------------------------------------------- +// §2 FieldOperatorsSchema.$eq refuses — the enforced copy and the documented one +// --------------------------------------------------------------------------- + +describe('#19889 §2 — the $eq operator slot refuses an array', () => { + it.each([ + ['FieldOperatorsSchema (the enforced copy)', FieldOperatorsSchema], + ['EqualityOperatorSchema (the documentation copy)', EqualityOperatorSchema], + ])('%s refuses $eq: [...] at $eq', (_label, schema) => { + const issue = issueAt(schema.safeParse({ $eq: ['won', 'lost'] }), '$eq'); + expect(issue.message).toMatch(/^Operator "\$eq" requires a single comparable value/); + expect(issueAt(schema.safeParse({ $eq: [] }), '$eq').message).toMatch(/received an array \(\[\]\)/); + }); + + it('the normalized AST — which validates field conditions against FieldOperatorsSchema — refuses it', () => { + expect(NormalizedFilterSchema.safeParse({ $and: [{ stage: { $eq: ['won'] } }] }).success).toBe(false); + // Control: the same member with a scalar is a normalized field condition. + expect(NormalizedFilterSchema.safeParse({ $and: [{ stage: { $eq: 'won' } }] }).success).toBe(true); + }); +}); + +// --------------------------------------------------------------------------- +// §3 The parse-door message IS the compile-face message +// --------------------------------------------------------------------------- + +describe('#19889 §3 — one text, two doors', () => { + it.each(REFUSED)('%s — the schema door prints the face\'s sentence', (_label, where, issuePath, facePath) => { + const face = faceRefusal(where); + // The face is the ADR-0112 class-1 refusal it has been since #19757. + expect(face.code).toBe(StandardErrorCode.enum.INVALID_FILTER); + expect(face.status).toBe(400); + // The one clause only the face can write is really there, exactly once… + const location = ` at ${facePath}.`; + expect(face.message.split(location)).toHaveLength(2); + // …and removing it leaves the schema door's message, character for character. + const schema = issueAt(FilterConditionSchema.safeParse(where), issuePath); + expect(schema.message).toBe(face.message.replace(location, '.')); + }); + + it('the FilterArray spelling reaches the face through parseFilterAST with the same sentence', () => { + let lowered: Error | undefined; + try { + parseFilterAST([['stage', '=', ['won', 'lost']]]); + } catch (error) { + lowered = error as Error; + } + expect(lowered?.message).toBe(faceRefusal({ stage: ['won', 'lost'] }).message); + }); + + it('the operator slot prints the same sentence, less the field it cannot see', () => { + const face = faceRefusal({ stage: { $eq: ['won', 'lost'] } }).message; + const slot = issueAt(FieldOperatorsSchema.safeParse({ $eq: ['won', 'lost'] }), '$eq').message; + expect(face).toContain(' on field "stage"'); + expect(slot).toBe(face.replace(' on field "stage"', '').replace(' at where.stage.$eq.', '.')); + }); + + it('the remedy the author is handed on save names both operators and the unapplied-filter reason', () => { + const message = issueAt(FilterConditionSchema.safeParse({ tags: ['a', 'b'] }), 'tags').message; + expect(message).toContain('For "one of these values" use {"$in": […]} (authoring: in)'); + expect(message).toContain('{"$contains": "…"} (authoring: contains), an $or of those for any-of.'); + expect(message).toMatch(/The filter was NOT applied, .*UNFILTERED result set\.$/); + // Inside the 500-char client bound the face's messages are held to. + const longest = issueAt( + FilterConditionSchema.safeParse({ close_date: ['aaaaaaaaaaaaaaaaaaaa', 'bbbbbbbbbbbbbbbbbbbbbbb', 'cccccccccccccccccccc'] }), + 'close_date', + ).message; + expect(longest.length).toBeLessThan(500); + }); +}); + +// --------------------------------------------------------------------------- +// §4 CONTROLS — both doors accept, so the arm is exactly the face's +// --------------------------------------------------------------------------- + +describe('#19889 §4 — what stays accepted, at BOTH doors', () => { + const day = new Date('2026-07-01T00:00:00.000Z'); + it.each([ + ['a string', { stage: 'won' }], + ['a number, zero', { amount: 0 }], + ['a boolean', { active: false }], + ['the empty string', { stage: '' }], + ['null — the has-no-value predicate', { stage: null }], + ['$eq: null', { stage: { $eq: null } }], + ['a Date', { closed_at: day }], + ['$eq: a Date', { closed_at: { $eq: day } }], + ['$eq: a { $field } reference', { amount: { $eq: { $field: 'budget' } } }], + ['$eq: a scalar', { stage: { $eq: 'won' } }], + ['$in keeps its list', { stage: { $in: ['won', 'lost'] } }], + ['$in: [] stays the declared predicate', { stage: { $in: [] } }], + ['$nin keeps its list', { stage: { $nin: ['lost'] } }], + ['$between keeps its pair', { amount: { $between: [1, 9] } }], + ['$ne carrying an array — not this ruling', { stage: { $ne: ['won'] } }], + ['an array inside a nested-relation spec — the face never descends one', { owner: { region: ['NA'] } }], + ['$eq array inside a nested-relation spec — likewise', { owner: { region: { $eq: ['NA'] } } }], + ])('%s', (_label, where) => { + expect(() => assertListComparandShapes(where)).not.toThrow(); + const result = FilterConditionSchema.safeParse(where); + expect(result.success, JSON.stringify(result.error?.issues)).toBe(true); + // Accepted means KEPT: the door returns the document it was given. + expect(result.data).toEqual(where); + }); + + it('the { $field } reference an equality spelling lowers to passes both doors', () => { + const lowered = parseFilterAST([['amount', '=', { $field: 'budget' }]]); + expect(lowered).toEqual({ amount: { $eq: { $field: 'budget' } } }); + expect(FilterConditionSchema.safeParse(lowered).success).toBe(true); + expect(FieldOperatorsSchema.safeParse({ $eq: { $field: 'budget' } }).success).toBe(true); + }); + + it('the operator slot accepts every scalar and null', () => { + for (const comparand of ['won', 0, false, '', null, day]) { + expect(FieldOperatorsSchema.safeParse({ $eq: comparand }).success, String(comparand)).toBe(true); + } + }); +}); + +// --------------------------------------------------------------------------- +// §5 The stored carriers refuse on save, naming the path +// --------------------------------------------------------------------------- + +describe('#19889 §5 — a stored filter carrier refuses the shape on save', () => { + const dataset = (extra: Record) => ({ + name: 'deals_ds', + label: 'Deals', + object: 'deal', + dimensions: [{ name: 'stage', field: 'stage', type: 'string' }], + measures: [{ name: 'deal_count', aggregate: 'count' }], + ...extra, + }); + + it('DatasetSchema refuses at filter.stage', () => { + const issue = issueAt(DatasetSchema.safeParse(dataset({ filter: { stage: ['won', 'lost'] } })), 'filter.stage'); + expect(issue.message).toMatch(/^The implicit-equality comparand on field "stage"/); + }); + + it('DatasetSchema refuses a measure filter at measures.0.filter.stage.$eq', () => { + const doc = dataset({ + measures: [{ name: 'deal_count', aggregate: 'count', filter: { stage: { $eq: ['won', 'lost'] } } }], + }); + const issue = issueAt(DatasetSchema.safeParse(doc), 'measures.0.filter.stage.$eq'); + expect(issue.message).toMatch(/^Operator "\$eq" on field "stage"/); + }); + + it('a dashboard widget filter and a report runtimeFilter refuse at their own paths', () => { + const dashboard = { + name: 'sales', + label: 'Sales', + widgets: [{ id: 'won_deals', type: 'metric', dataset: 'deals', values: ['total'], filter: { stage: ['won'] } }], + }; + issueAt(DashboardSchema.safeParse(dashboard), 'widgets.0.filter.stage'); + const report = { + name: 'pipeline', label: 'Pipeline', type: 'summary', + dataset: 'sales', rows: ['stage'], values: ['revenue'], + runtimeFilter: { stage: { $eq: ['won'] } }, + }; + issueAt(ReportSchema.safeParse(report), 'runtimeFilter.stage.$eq'); + }); + + it('CONTROL — the same documents with $in publish, and keep their filter', () => { + const parsed = DatasetSchema.safeParse(dataset({ filter: { stage: { $in: ['won', 'lost'] } } })); + expect(parsed.success, JSON.stringify(parsed.error?.issues)).toBe(true); + expect(parsed.data!.filter).toEqual({ stage: { $in: ['won', 'lost'] } }); + }); +}); diff --git a/packages/spec/src/data/filter.zod.ts b/packages/spec/src/data/filter.zod.ts index f44efc43bb8..be90963c907 100644 --- a/packages/spec/src/data/filter.zod.ts +++ b/packages/spec/src/data/filter.zod.ts @@ -3,6 +3,10 @@ import { z } from 'zod'; import { closedObject } from '../shared/strict-object'; import { assertListComparandShapes } from './filter-comparand-shape'; +// [#19889] The equality-slot refusal's words, shared with the comparand-shape +// face so the save door and the query door print one sentence (ruling A, +// record 5805248669: "one constant, two doors"). +import { arrayEqualityComparandMessage } from './filter-comparand-refusal-text'; import { normalizeFilterComparandTypes } from './filter-comparand-type'; import { bareDateRangePresetComparandMessage, isDateRangePresetName } from './date-range-presets'; // [#19514] The text-comparand door this package publishes for the @@ -257,13 +261,40 @@ const NE_DESCRIPTION = + '`{ "$ne": null }` is the HAS-A-VALUE predicate — the same question `{ "$exists": true }` ' + 'asks: it matches rows whose field holds a value, never rows that merely carry the key.'; +/** + * [#19889] The `$eq` slot, shared by the documentation copy + * ({@link EqualityOperatorSchema}) and the enforced copy + * (`FieldOperatorsSchema`) — the two share the CODE, the pairing + * `orderingComparandSchema` gives the ordering slots, so the published + * reference cannot describe a slot the enforced copy narrows. + * + * Open (`z.any()`) EXCEPT for an ARRAY, refused by ruling (#19889, letter A, + * record 5805248669): an array is not an equality comparand. The comparand-shape + * face has refused it on every query since ruling 乙 (#19757); this is the same + * refusal at the schema door, in the same words + * (`arrayEqualityComparandMessage`, one text for both doors). The field is not + * named here because this slot cannot see it — the key belongs to the record + * enclosing the operator map — and the issue's own `path` says where it is. + * `null`, every scalar, a `Date` and a `{ $field }` reference are not arrays and + * pass exactly as before; `$ne` is not judged (the ruling names equality). + * + * ⚠️ `z.toJSONSchema()` has no arm for a custom check, so the published JSON + * Schema still reads `{}` here. The site is declared in + * `dropped-refinements.baseline.json`. + */ +const equalityComparandSchema = () => + z.any().superRefine((value, ctx) => { + if (!Array.isArray(value)) return; + ctx.addIssue({ code: 'custom', message: arrayEqualityComparandMessage(value, { op: '$eq' }) }); + }).optional().describe(EQ_DESCRIPTION); + /** * Comparison operators for equality and inequality checks. * Supported data types: Any */ export const EqualityOperatorSchema = lazySchema(() => z.object({ /** Equal to (default) - SQL: = | MongoDB: $eq */ - $eq: z.any().optional().describe(EQ_DESCRIPTION), + $eq: equalityComparandSchema(), /** Not equal to - SQL: <> or != | MongoDB: $ne */ $ne: z.any().optional().describe(NE_DESCRIPTION), @@ -1415,8 +1446,10 @@ export const FieldOperatorsSchema = lazySchema(() => z.object({ // Equality. Both slots read the SAME description constants the documentation // copy ({@link EqualityOperatorSchema}) reads — the pairing the ordering and // set slots below already use, extended to every remaining family so no - // operator can be described in one copy and blank in the other. - $eq: z.any().optional().describe(EQ_DESCRIPTION), + // operator can be described in one copy and blank in the other. [#19889] + // `$eq` is built from the same `equalityComparandSchema` factory the + // documentation copy uses: it refuses an ARRAY comparand, in the face's words. + $eq: equalityComparandSchema(), $ne: z.any().optional().describe(NE_DESCRIPTION), // Ordering. `string` is in the union for the reason {@link ComparisonOperatorSchema} @@ -1553,8 +1586,36 @@ function isPlainFilterNode(value: unknown): value is Record { /** * Walk one condition node and report every comparand this authoring door * refuses — the bare date-range PRESET names in an ordering position (#8793), - * and the `$icontains` comparands the platform's own conformance table declares - * refused (#19514). + * the `$icontains` comparands the platform's own conformance table declares + * refused (#19514), and an ARRAY in the EQUALITY slot (#19889). + * + * ## The equality-slot arm answers the FACE, in the face's words (#19889) + * + * Ruled 2026-09-24 (letter A, record 5805248669): `FilterConditionSchema` + * (implicit equality) and `FieldOperatorsSchema.$eq` refuse an array comparand + * at parse, "with the SAME remedy text the shared compile face emits — one + * constant, two doors". The compile face (`assertListComparandShapes`) has + * refused `{ field: [...] }` and `{ field: { $eq: [...] } }` on every query since + * ruling 乙 (#19757), so before this arm a stored dataset, widget or report + * filter carrying either shape published clean and failed every query that + * used it, for a different person, later. + * + * - **The words** are `arrayEqualityComparandMessage` + * (`./filter-comparand-refusal-text.ts`), which the face prints too. The one + * difference is the location: the face appends `at where.`, this door + * does not, because a refinement cannot see where it sits in the document and + * the issue's own `path` carries that (`filter.stage`, + * `measures.0.filter.stage.$eq`). + * - **The reach** is the face's, not wider. This node's own field entries are + * judged, and `$and` / `$or` / `$not` members by their own pass, as the face + * walks them. A field spec with NO `$` key is a nested-relation or + * deep-equality condition the face never descends, so the arm does not fire + * inside one (`depth` > 0): refusing there would refuse a document the query + * door accepts, which is the split this arm exists to close. + * - **Not dropped.** The refusal fails the parse; nothing is stripped from the + * document. A dropped filter would show MORE rows than the author asked for. + * - ⛔ `$ne` is not judged (the ruling names equality), and the list operators + * keep their lists, `$in: []` / `$nin: []` included. * * Descends non-`$` keys only (operator specs and nested relations): the * `$and` / `$or` / `$not` members are re-parsed by {@link FilterConditionSchema} @@ -1606,7 +1667,22 @@ function checkFilterConditionComparands( for (const [key, value] of Object.entries(node)) { if (key.startsWith('$')) continue; // $and/$or/$not re-parse; other $ keys stay unjudged - if (!isPlainFilterNode(value)) continue; // implicit equality / arrays — not judged + // [#19889] The EQUALITY slot, implicit form `{ field: [...] }` — the empty + // list included. Judged on this node's OWN field entries only (`depth` 0), + // the face's exact reach: its walk never descends a field spec that has no + // `$` key, so an array inside a nested-relation condition is not refused + // there and is not refused here. See the docblock's third arm. + if (Array.isArray(value)) { + if (depth === 0) { + ctx.addIssue({ + code: 'custom', + path: [...path, key], + message: arrayEqualityComparandMessage(value, { field: key }), + }); + } + continue; + } + if (!isPlainFilterNode(value)) continue; // a scalar implicit-equality comparand — not judged const hasOperatorKeys = Object.keys(value).some((k) => k.startsWith('$')); if (!hasOperatorKeys) { // Nested relation / deep equality — the schema does not re-parse these, @@ -1615,6 +1691,19 @@ function checkFilterConditionComparands( continue; } for (const [op, comparand] of Object.entries(value)) { + // [#19889] The EQUALITY slot, explicit form `{ field: { $eq: [...] } }`, + // on the same reach as the implicit form above. `null`, every scalar and a + // `{ $field }` reference are not arrays and pass. + if (op === '$eq') { + if (depth === 0 && Array.isArray(comparand)) { + ctx.addIssue({ + code: 'custom', + path: [...path, key, op], + message: arrayEqualityComparandMessage(comparand, { op: '$eq', field: key }), + }); + } + continue; + } if (op === FILTER_TEXT_COMPARAND_OPERATOR && isRefusedTextComparand(comparand)) { ctx.addIssue({ code: 'custom', @@ -1765,15 +1854,17 @@ export const FilterConditionSchema: z.ZodType $or: z.array(FilterConditionSchema).optional(), $not: FilterConditionSchema.optional(), }) - // Two comparand refusals ride one walk — see its docblock for why. [#8793] + // Three comparand refusals ride one walk — see its docblock for why. [#8793] // Bare date-range preset names are refused from ordering comparands (the // § 3.35 block above carries the ruling, the measured defect and the // ordering-only boundary); [#19514] `$icontains` comparands are refused on // the two shapes `FILTER_TEXT_CASES` already declares refused, so the door - // stops admitting the document its own conformance table says will 400. The - // refinement judges this node's own field entries; `$and` / `$or` / `$not` - // members re-enter the schema and are judged by their own pass with nested - // issue paths. + // stops admitting the document its own conformance table says will 400; + // [#19889] an ARRAY in the equality slot, implicit or `$eq`, is refused in + // the comparand-shape face's own words, so a stored filter the query door + // refuses is refused on save. The refinement judges this node's own field + // entries; `$and` / `$or` / `$not` members re-enter the schema and are + // judged by their own pass with nested issue paths. ).superRefine((node, ctx) => checkFilterConditionComparands(node, ctx)) ); diff --git a/packages/spec/src/migrations/entries/semantic/18.filter-equality-array-comparand-refused-at-save.ts b/packages/spec/src/migrations/entries/semantic/18.filter-equality-array-comparand-refused-at-save.ts new file mode 100644 index 00000000000..b8d4e7610fe --- /dev/null +++ b/packages/spec/src/migrations/entries/semantic/18.filter-equality-array-comparand-refused-at-save.ts @@ -0,0 +1,83 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import type { SemanticMigration } from '../../types.js'; + +// The SCHEMA door's half of filter-equality-array-comparand-refused. That entry +// refuses the shape at the runtime filter doors, where a query is compiled; this +// one refuses the same shape where a filter is SAVED, in the same words, so a +// stored filter stops publishing clean and failing later for someone else. +// Recorded as its own entry because the surface is different (every schema that +// carries a FilterCondition, and the $eq operator slot) and because what an +// upgrading author sees changes at a different moment: on save, not on query. +export const entry: SemanticMigration = { + id: 'filter-equality-array-comparand-refused-at-save', + // No backticks in `surface` — build-upgrade-guide renders it inside a code + // span already, and a nested backtick would close it. + surface: + 'data.FilterCondition and the $eq slot of data.FieldOperators — an ARRAY as an EQUALITY ' + + 'comparand, now refused when the document is PARSED: the implicit form { field: [...] } and ' + + 'the explicit form { field: { $eq: [...] } }, the empty array included, on every schema ' + + 'that carries a FilterCondition — a dataset filter and a dataset measure filter, a ' + + 'dashboard widget filter and an options-source filter, a report and joined-report-block ' + + 'runtimeFilter, a field relatedListFilter and a rollup summaryOperations filter, a ' + + 'solution-blueprint summary filter, an analytics query where, a dataset selection ' + + 'runtimeFilter, a query where and having, the data-engine aggregate call\'s having, an ' + + 'aggregation filter and a query-filter where ' + + '— plus FieldOperatorsSchema.$eq, its documentation copy EqualityOperatorSchema.$eq, and ' + + 'the NormalizedFilter AST that validates against it', + replacement: + 'the operator the list was standing in for, exactly as in ' + + 'filter-equality-array-comparand-refused. "One of these values" is $in: ' + + '{ field: { $in: ["a", "b"] } } (authoring spelling "in"). "The stored multi-value field ' + + 'holds this value" is $contains with ONE member: { field: { $contains: "a" } } (authoring ' + + 'spelling "contains"), and an $or of those for any-of. A filter that meant a single value ' + + 'writes that value: { field: "a" }. The list operators ($in / $nin / $between) keep their ' + + 'arrays, empty lists included; every scalar equality comparand, null above all, a Date and ' + + 'a { $field } reference are untouched; and $ne is NOT judged by this entry', + reason: + 'Ruling on #19889 (record 5805248669, letter A): FilterConditionSchema (implicit equality) ' + + 'and FieldOperatorsSchema.$eq refuse an array comparand at parse, with the SAME remedy ' + + 'text the shared compile face emits — one constant, two doors; a stored filter carrying ' + + 'the shape is refused loudly on its next save, and never silently dropped, because a ' + + 'dropped filter shows MORE rows than intended. Measured on origin/main a0920b42dc before ' + + 'the change: a dataset whose filter was { stage: ["won", "lost"] }, and one whose measure ' + + 'filter was { stage: { $eq: ["won", "lost"] } }, both parsed GREEN, as did ' + + 'FilterConditionSchema and FieldOperatorsSchema on the bare shapes, while the shared ' + + 'comparand-shape face refused both with INVALID_FILTER / 400. So such a document ' + + 'published clean and then failed every query that used it, for a different person, ' + + 'later. The schema door now prints the face\'s own sentence, from one builder both doors ' + + 'import; the only difference is that the face appends the location (at where.stage) and ' + + 'the schema door does not, because its issue carries the location as its path ' + + '(filter.stage, measures.0.filter.stage.$eq). The reach is the face\'s and no wider: the ' + + 'field entries of a condition and of every $and / $or / $not member, but NOT a field spec ' + + 'with no $ key (a nested-relation or deep-equality condition), which the face never ' + + 'descends either. ⚠️ Three positions therefore still refuse only at execution. (1) A list ' + + 'inside a nested-relation condition, { account: { region: ["a"] } }: the analytics where ' + + 'door flattens that to the dotted member account.region and refuses it when a dataset or ' + + 'measure filter is charted. (2) The where option of the data-engine calls (find, count, ' + + 'update, delete, aggregate, vector find): its type is a union whose first arm is an open ' + + 'record, so it parses and the face refuses it when the call runs. (3) $ne carrying a ' + + 'list, which no ruling has decided. Two request doors parse these carriers and now answer ' + + 'the shape before the analytics compiler does: the REST dataset selection (its ' + + 'runtimeFilter) and the analytics query body (its where) refuse with VALIDATION_FAILED / ' + + '400 and this sentence at the field, one step ahead of the compiler\'s INVALID_FILTER / ' + + '400. Metadata AT REST is not rewritten and this entry adds no D2 conversion, for the ' + + 'reason the runtime entry gives: an array on equality has no single honest value. The ' + + 'read path does not re-validate stored rows, so a stored document keeps loading; what ' + + 'changes is that re-saving it through the metadata protocol (422 INVALID_METADATA), ' + + 'defineStack or os validate is refused at the filter\'s path. Such a filter has failed ' + + 'every query since the runtime entry, and on the SQL family before it at the top level, ' + + 'so the refusal is a repair and not a loss. ADR-0049 / ADR-0087 / ADR-0112.', + acceptanceCriteria: + 'Validate every stack and re-save every stored document that carries a filter: os validate ' + + 'or defineStack, and a save through the metadata protocol, report each array in an ' + + 'equality slot by path with the field, the received list and both remedies, so the sweep ' + + 'is mechanical for the carriers listed in the surface. Decide per filter what it meant — ' + + 'one of these values ($in), the stored list holds a value ($contains, an $or of them for ' + + 'several), or one value — and re-check what the surface is supposed to show rather than ' + + 'assuming the old rows were right: on most backends the filter had been failing every ' + + 'query. ⛔ A clean re-save is NOT a complete sweep for the three positions the reason ' + + 'names: grep nested-relation conditions and data-engine where options for a field whose ' + + 'value is a list, and exercise them, where the runtime doors refuse with INVALID_FILTER ' + + '/ 400 naming the field and the path.', +}; diff --git a/packages/spec/src/migrations/entries/semantic/18.filter-equality-array-comparand-refused.ts b/packages/spec/src/migrations/entries/semantic/18.filter-equality-array-comparand-refused.ts index d6f055a3634..eac947138cb 100644 --- a/packages/spec/src/migrations/entries/semantic/18.filter-equality-array-comparand-refused.ts +++ b/packages/spec/src/migrations/entries/semantic/18.filter-equality-array-comparand-refused.ts @@ -45,11 +45,21 @@ export const entry: SemanticMigration = { + 'named proxy, over ["a"], "a", ["a","b"], ["b","a"], [["a"],"x"], [["a"]], "b" and [] ' + 'selected ["a"], [["a"],"x"] and [["a"]]. The service-analytics filter normalizer read the ' + 'FilterArray form as MEMBERSHIP: ["stage", "=", ["won", "lost"]] charted as stage IN ' - + '(won, lost); that form now gets the refusal too, while its OBJECT form, which that ' - + 'normalizer does not route through the shared face, still reads as membership. A live mongod, MySQL, PostgreSQL and a live ' + + '(won, lost), and its OBJECT form read the same list four ways: { stage: [...] } as IN, ' + + '$eq with a list as its first member alone, $eq with an empty list as no predicate, and ' + + 'an empty implicit list as the FALSE constant. A live mongod, MySQL, PostgreSQL and a live ' + 'Turso server were NOT measured. So one stored filter was a 400 on most backends and a ' + 'silent, differently-shaped row set on one. The shared face now refuses it with ' + 'INVALID_FILTER / 400 before any driver runs, naming the field, the path and both remedies. ' + + 'Which doors refuse it at this release, and with what: the shared face, inside ' + + 'parseFilterAST and at the engine lowering seam, with INVALID_FILTER / 400; the analytics ' + + 'where door in BOTH spellings, the FilterArray form through parseFilterAST and the OBJECT ' + + 'form because that door hands each equality-slot list to the shared face before it builds ' + + 'a node, with the same INVALID_FILTER / 400 and the same sentence (that door alone also ' + + 'refuses a list inside a nested-relation condition, which it flattens to a dotted member); ' + + 'and, on SAVE, the schema door (FilterConditionSchema and the $eq operator slot), with the ' + + 'same sentence as a parse issue at the filter\'s own path, which is the sibling entry ' + + 'filter-equality-array-comparand-refused-at-save. ' + 'The ruling records the hosted product as running on the SQL family, where the top-level ' + 'shape was already a 400, so the population that can observe a change is self-hosted ' + 'driver-mongodb, plus any filter nested under a combinator on the SQL family (a 500 ' @@ -65,9 +75,11 @@ export const entry: SemanticMigration = { + 'what it meant: one of these values ($in), the stored list holds a value ($contains, an ' + '$or of them for several), or one value. Each is refused at query time with INVALID_FILTER ' + '/ 400 naming the field and the path, so a test suite that exercises the query finds ' - + 'every one. A dashboard or dataset filter written as the FilterArray sugar with an array on ' - + 'equality charted as membership through the analytics normalizer; $in is the spelling that ' - + 'charts the same rows. On driver-mongodb re-check what the query is supposed to return rather than ' + + 'every one; a stored carrier is also refused on save, which is the sibling entry ' + + 'filter-equality-array-comparand-refused-at-save. A dashboard or dataset filter written as ' + + 'the FilterArray sugar with an array on equality, or as the implicit object form, charted ' + + 'as membership through the analytics normalizer; $in is the spelling that charts the same ' + + 'rows. On driver-mongodb re-check what the query is supposed to return rather than ' + 'assuming the old rows were right: the old answer was MongoDB array equality, which ' + 'neither $in nor $contains reproduces.', }; diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index 30cf33582d0..f6e4baf8695 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -8811,11 +8811,21 @@ const step18: MigrationStep = { + 'named proxy, over ["a"], "a", ["a","b"], ["b","a"], [["a"],"x"], [["a"]], "b" and [] ' + 'selected ["a"], [["a"],"x"] and [["a"]]. The service-analytics filter normalizer read the ' + 'FilterArray form as MEMBERSHIP: ["stage", "=", ["won", "lost"]] charted as stage IN ' - + '(won, lost); that form now gets the refusal too, while its OBJECT form, which that ' - + 'normalizer does not route through the shared face, still reads as membership. A live mongod, MySQL, PostgreSQL and a live ' + + '(won, lost), and its OBJECT form read the same list four ways: { stage: [...] } as IN, ' + + '$eq with a list as its first member alone, $eq with an empty list as no predicate, and ' + + 'an empty implicit list as the FALSE constant. A live mongod, MySQL, PostgreSQL and a live ' + 'Turso server were NOT measured. So one stored filter was a 400 on most backends and a ' + 'silent, differently-shaped row set on one. The shared face now refuses it with ' + 'INVALID_FILTER / 400 before any driver runs, naming the field, the path and both remedies. ' + + 'Which doors refuse it at this release, and with what: the shared face, inside ' + + 'parseFilterAST and at the engine lowering seam, with INVALID_FILTER / 400; the analytics ' + + 'where door in BOTH spellings, the FilterArray form through parseFilterAST and the OBJECT ' + + 'form because that door hands each equality-slot list to the shared face before it builds ' + + 'a node, with the same INVALID_FILTER / 400 and the same sentence (that door alone also ' + + 'refuses a list inside a nested-relation condition, which it flattens to a dotted member); ' + + 'and, on SAVE, the schema door (FilterConditionSchema and the $eq operator slot), with the ' + + 'same sentence as a parse issue at the filter\'s own path, which is the sibling entry ' + + 'filter-equality-array-comparand-refused-at-save. ' + 'The ruling records the hosted product as running on the SQL family, where the top-level ' + 'shape was already a 400, so the population that can observe a change is self-hosted ' + 'driver-mongodb, plus any filter nested under a combinator on the SQL family (a 500 ' @@ -8831,12 +8841,93 @@ const step18: MigrationStep = { + 'what it meant: one of these values ($in), the stored list holds a value ($contains, an ' + '$or of them for several), or one value. Each is refused at query time with INVALID_FILTER ' + '/ 400 naming the field and the path, so a test suite that exercises the query finds ' - + 'every one. A dashboard or dataset filter written as the FilterArray sugar with an array on ' - + 'equality charted as membership through the analytics normalizer; $in is the spelling that ' - + 'charts the same rows. On driver-mongodb re-check what the query is supposed to return rather than ' + + 'every one; a stored carrier is also refused on save, which is the sibling entry ' + + 'filter-equality-array-comparand-refused-at-save. A dashboard or dataset filter written as ' + + 'the FilterArray sugar with an array on equality, or as the implicit object form, charted ' + + 'as membership through the analytics normalizer; $in is the spelling that charts the same ' + + 'rows. On driver-mongodb re-check what the query is supposed to return rather than ' + 'assuming the old rows were right: the old answer was MongoDB array equality, which ' + 'neither $in nor $contains reproduces.', }, + // The SCHEMA door's half of filter-equality-array-comparand-refused. That entry + // refuses the shape at the runtime filter doors, where a query is compiled; this + // one refuses the same shape where a filter is SAVED, in the same words, so a + // stored filter stops publishing clean and failing later for someone else. + // Recorded as its own entry because the surface is different (every schema that + // carries a FilterCondition, and the $eq operator slot) and because what an + // upgrading author sees changes at a different moment: on save, not on query. + { + id: 'filter-equality-array-comparand-refused-at-save', + // No backticks in `surface` — build-upgrade-guide renders it inside a code + // span already, and a nested backtick would close it. + surface: + 'data.FilterCondition and the $eq slot of data.FieldOperators — an ARRAY as an EQUALITY ' + + 'comparand, now refused when the document is PARSED: the implicit form { field: [...] } and ' + + 'the explicit form { field: { $eq: [...] } }, the empty array included, on every schema ' + + 'that carries a FilterCondition — a dataset filter and a dataset measure filter, a ' + + 'dashboard widget filter and an options-source filter, a report and joined-report-block ' + + 'runtimeFilter, a field relatedListFilter and a rollup summaryOperations filter, a ' + + 'solution-blueprint summary filter, an analytics query where, a dataset selection ' + + 'runtimeFilter, a query where and having, the data-engine aggregate call\'s having, an ' + + 'aggregation filter and a query-filter where ' + + '— plus FieldOperatorsSchema.$eq, its documentation copy EqualityOperatorSchema.$eq, and ' + + 'the NormalizedFilter AST that validates against it', + replacement: + 'the operator the list was standing in for, exactly as in ' + + 'filter-equality-array-comparand-refused. "One of these values" is $in: ' + + '{ field: { $in: ["a", "b"] } } (authoring spelling "in"). "The stored multi-value field ' + + 'holds this value" is $contains with ONE member: { field: { $contains: "a" } } (authoring ' + + 'spelling "contains"), and an $or of those for any-of. A filter that meant a single value ' + + 'writes that value: { field: "a" }. The list operators ($in / $nin / $between) keep their ' + + 'arrays, empty lists included; every scalar equality comparand, null above all, a Date and ' + + 'a { $field } reference are untouched; and $ne is NOT judged by this entry', + reason: + 'Ruling on #19889 (record 5805248669, letter A): FilterConditionSchema (implicit equality) ' + + 'and FieldOperatorsSchema.$eq refuse an array comparand at parse, with the SAME remedy ' + + 'text the shared compile face emits — one constant, two doors; a stored filter carrying ' + + 'the shape is refused loudly on its next save, and never silently dropped, because a ' + + 'dropped filter shows MORE rows than intended. Measured on origin/main a0920b42dc before ' + + 'the change: a dataset whose filter was { stage: ["won", "lost"] }, and one whose measure ' + + 'filter was { stage: { $eq: ["won", "lost"] } }, both parsed GREEN, as did ' + + 'FilterConditionSchema and FieldOperatorsSchema on the bare shapes, while the shared ' + + 'comparand-shape face refused both with INVALID_FILTER / 400. So such a document ' + + 'published clean and then failed every query that used it, for a different person, ' + + 'later. The schema door now prints the face\'s own sentence, from one builder both doors ' + + 'import; the only difference is that the face appends the location (at where.stage) and ' + + 'the schema door does not, because its issue carries the location as its path ' + + '(filter.stage, measures.0.filter.stage.$eq). The reach is the face\'s and no wider: the ' + + 'field entries of a condition and of every $and / $or / $not member, but NOT a field spec ' + + 'with no $ key (a nested-relation or deep-equality condition), which the face never ' + + 'descends either. ⚠️ Three positions therefore still refuse only at execution. (1) A list ' + + 'inside a nested-relation condition, { account: { region: ["a"] } }: the analytics where ' + + 'door flattens that to the dotted member account.region and refuses it when a dataset or ' + + 'measure filter is charted. (2) The where option of the data-engine calls (find, count, ' + + 'update, delete, aggregate, vector find): its type is a union whose first arm is an open ' + + 'record, so it parses and the face refuses it when the call runs. (3) $ne carrying a ' + + 'list, which no ruling has decided. Two request doors parse these carriers and now answer ' + + 'the shape before the analytics compiler does: the REST dataset selection (its ' + + 'runtimeFilter) and the analytics query body (its where) refuse with VALIDATION_FAILED / ' + + '400 and this sentence at the field, one step ahead of the compiler\'s INVALID_FILTER / ' + + '400. Metadata AT REST is not rewritten and this entry adds no D2 conversion, for the ' + + 'reason the runtime entry gives: an array on equality has no single honest value. The ' + + 'read path does not re-validate stored rows, so a stored document keeps loading; what ' + + 'changes is that re-saving it through the metadata protocol (422 INVALID_METADATA), ' + + 'defineStack or os validate is refused at the filter\'s path. Such a filter has failed ' + + 'every query since the runtime entry, and on the SQL family before it at the top level, ' + + 'so the refusal is a repair and not a loss. ADR-0049 / ADR-0087 / ADR-0112.', + acceptanceCriteria: + 'Validate every stack and re-save every stored document that carries a filter: os validate ' + + 'or defineStack, and a save through the metadata protocol, report each array in an ' + + 'equality slot by path with the field, the received list and both remedies, so the sweep ' + + 'is mechanical for the carriers listed in the surface. Decide per filter what it meant — ' + + 'one of these values ($in), the stored list holds a value ($contains, an $or of them for ' + + 'several), or one value — and re-check what the surface is supposed to show rather than ' + + 'assuming the old rows were right: on most backends the filter had been failing every ' + + 'query. ⛔ A clean re-save is NOT a complete sweep for the three positions the reason ' + + 'names: grep nested-relation conditions and data-engine where options for a field whose ' + + 'value is a list, and exercise them, where the runtime doors refuse with INVALID_FILTER ' + + '/ 400 naming the field and the path.', + }, // One entry for two doors on purpose: the two vocabularies spell one operator // and the rows being answered are one pair. Splitting it would put half the // prescription in front of an author who wrote the other spelling.