fix(formula)!: refuse an ordering comparison whose stored operand holds a list or an object

.changeset/19886-stored-list-ordering-refused.md

@@ -0,0 +1,40 @@
+---
+"@objectstack/formula": minor
+"@objectstack/plugin-security": minor
+"@objectstack/spec": patch
+---
+
+A row-level write check that orders a field against a bound (`>`, `>=`, `<`, `<=`) is refused with `INVALID_FILTER` / 400 when that field holds a list or an object on the record being written, instead of comparing the list's string form and admitting the write (#19886).
+
+**BREAKING** — an accept-set narrowing, shipped by `@objectstack/formula` and `@objectstack/plugin-security` as `minor` under the repo's launch-window convention for accept-set narrowings. The hand-migration prescription is registered under protocol major 18 as `rls-predicate-stored-list-ordering-refused`.
+
+Clause-②: no (narrowing)
+
+**Security fix for RLS write checks.** Measured through the real plugin-security on driver-sql and driver-memory: `record.tags > 'a'`, with `tags` a `json` field holding `['m']`, compared `'m' > 'a'` and admitted and stored the write. `record.meta < 'a'` with `meta` holding `{ a: 1 }` compared `'[object Object]' < 'a'` and did the same, and so did a `multiple` lookup. `using` stands in as the check for a policy that declares no `check`, so the same predicate in `using` was enforced the same way on writes. No shipped row-level or sharing-rule predicate orders a field at all.
+
+What changes:
+
+- `@objectstack/formula`: `matchesFilterCondition` refuses `$gt` / `$gte` / `$lt` / `$lte` and `$between` on a field whose value on the record is a list or a plain object, whatever the comparand, with the same `INVALID_FILTER` / 400 and the same message as the stage 2d refusals. The refusal is per record: a record whose `json` field holds one scalar is compared as before. `null` and `Date` values are unchanged, and so is every equality against a stored list (`$eq`, `$ne`, implicit equality, `$in`, `$nin`). `$between` is not produced by the CEL lowering, so it reaches this only through a filter passed to `matchesFilterCondition` directly.
+- `@objectstack/plugin-security`: a check insert or by-id update whose post-image holds a list or an object in an ordered field is refused 400 and stores nothing. That includes a by-id update that edits another field of a row whose stored `json` column holds a list, because the post-image merges the stored row.
+- `@objectstack/spec`: the migration registry carries the entry. The stage 2a entry `rls-predicate-array-comparand-refused` now ends "Scalar != and ==, null, Date comparands, and { $field } references between single-valued columns evaluate exactly as before", which is true since stage 2d.
+
+Three moves, named:
+
+1. **The write check now matches driver-sql's read.** driver-sql refuses every ordering comparison, and `$between`, on a column it stores as JSON text, by declared type (400, #7398). The in-process check now refuses the same predicate on the same row (400).
+2. **driver-memory's read parts from the write check.** driver-memory, a test driver, compares a stored list element by element on a read and keeps returning those rows (`record.tags > 'a'` reads a row holding `['m']`), while the check now refuses writing it. This is declared on #15104, as for stage 2d's `{ $field }` half.
+3. **A list written into a scalar field under an ordering check now answers 400.** `status: ['m']` into a `text` field under `record.status > 'a'`, or `amount: [500]` into a `number` field under `record.amount > 10`, was admitted, and driver-sql stored it as the text `'["m"]'` / `'[500]'`. It is now refused before anything is stored.
+
+**The explain answer.** `security/explain` evaluates the business RLS predicate in-process on the fetched record, so it now answers `INVALID_FILTER` / 400 where the record holds a list or an object under an ordering predicate (this stage). It already answered 400 for a `{ $field }` comparison against a list-holding column (stage 2d). For both, per operation:
+
+| explain operation | driver | the enforced operation answers | same as explain's 400? |
+|---|---|---|---|
+| `read` | driver-sql | 400 `INVALID_FILTER` (the driver's refusal) | yes |
+| `update` | driver-memory | 400 `INVALID_FILTER` (the post-image check) | yes |
+| `update` | driver-sql | 403 `PERMISSION_DENIED`: the pre-image gate fails closed on the driver's 400 | no — both deny |
+| `read` | driver-memory | the rows its element-wise read admits | no — the test driver's read |
+
+Explain itself is unchanged.
+
+**What to change.** Order a single-valued column (`record.priority > 2`), or test membership in the list with `in` (`record.status in ['open', 'pending']`). A `json` or `multiple` field has no ordering.
+
+<!-- adr-0087: registered rls-predicate-stored-list-ordering-refused -->

packages/formula/src/matches-filter-array-comparand.test.ts

@@ -34,6 +34,15 @@
  * the rows above. The third is a `{ $field }` comparison whose column holds a
  * list: the shape is legal, so it is judged on the record — refused when either
  * compared column holds a list (or an object) on the record being judged.
+ *
+ * [#19886 stage 2e] The mirror of the first 2d row, with the list on the
+ * RECORD's side and a literal bound — also judged on the record:
+ *
+ * | shape                                                  | before                                        | after        |
+ * |---|---|---|
+ * | `{ tags: { $gt: 'a' } }`, `tags` holding `['m']`       | `'m' > 'a'` → `true`, the write **ALLOWED**   | throws → 400 |
+ * | `{ meta: { $lt: 'a' } }`, `meta` holding `{ a: 1 }`    | `'[object Object]' < 'a'` → `true`, **ALLOWED** | throws → 400 |
+ * | `{ tags: { $between: ['a', 'z'] } }`, `['m']`          | `true`                                        | throws → 400 |
  */
 
 import { describe, it, expect } from 'vitest';
@@ -235,3 +244,124 @@ describe('[#19886 stage 2d] a { $field } comparison whose column holds a list is
     }
   });
 });
+
+describe('[#19886 stage 2e] an ordering comparison whose STORED operand holds a list or an object is refused on that record', () => {
+  const m = matchesFilterCondition;
+
+  /** What a `json` column or a `multiple` lookup holds on a post-image. */
+  const STORED: Array<[string, unknown]> = [
+    ['a one-element list', ['m']],
+    ['a multi-element list', ['a', 'z']],
+    ['an empty list', []],
+    ['a list of lists', [['m']]],
+    ['a plain object', { a: 1 }],
+    ['an empty object', {}],
+  ];
+
+  /** Every ordering operator, each with a bound the coerced string form used to satisfy or not. */
+  const ORDERINGS: Array<[string, Record<string, unknown>]> = [
+    ['$gt', { $gt: 'a' }],
+    ['$gte', { $gte: 'a' }],
+    ['$lt', { $lt: 'z' }],
+    ['$lte', { $lte: 'z' }],
+    ['$between', { $between: ['a', 'z'] }],
+  ];
+
+  for (const [storedName, stored] of STORED) {
+    for (const [opName, spec] of ORDERINGS) {
+      it(`${opName} on a field holding ${storedName}: INVALID_FILTER / 400`, () => {
+        const err = refusalOf({ tags: spec }, { tags: stored });
+        expect(err.code).toBe('INVALID_FILTER');
+        expect(err.status).toBe(400);
+      });
+    }
+  }
+
+  it('the refusal reaches every depth the evaluation reaches, and a negation cannot turn it into an answer', () => {
+    const record = { owner: 'u1', tags: ['m'] };
+    const leaf = { tags: { $gt: 'a' } };
+    for (const filter of [
+      { $and: [{ owner: 'u1' }, leaf] },
+      { $or: [{ owner: 'nobody' }, leaf] },
+      { $not: leaf },
+      { $and: [{ $or: [{ $not: leaf }] }] },
+    ]) {
+      const err = refusalOf(filter, record);
+      expect(err.code).toBe('INVALID_FILTER');
+      expect(err.status).toBe(400);
+    }
+  });
+
+  it('a verdict already decided without the field does not reach it — judged per record, like the stage 2d refusal', () => {
+    // Unlike the shape refusals, which judge the authored filter before any
+    // record, this one judges a VALUE, so it fires where evaluation reads it.
+    // Where it is not read, the answer does not depend on it either way.
+    const record = { owner: 'u1', tags: ['m'] };
+    expect(m(record, { $or: [{ owner: 'u1' }, { tags: { $gt: 'a' } }] })).toBe(true);
+    expect(m(record, { owner: 'nobody', tags: { $gt: 'a' } })).toBe(false);
+  });
+
+  it('whatever the comparand: a number, a Date, null or a { $field } reference', () => {
+    const record = { tags: ['m'], status: 'a' };
+    for (const spec of [
+      { $gt: 5 },
+      { $lt: new Date('2026-01-01T00:00:00.000Z') },
+      { $gte: null },
+      { $lte: { $field: 'status' } },
+    ]) {
+      const err = refusalOf({ tags: spec }, record);
+      expect(err.code).toBe('INVALID_FILTER');
+      expect(err.status).toBe(400);
+    }
+  });
+
+  it('a list written into a scalar field is judged the same way (a text or number field under an ordering check)', () => {
+    for (const [record, filter] of [
+      [{ status: ['m'] }, { status: { $gt: 'a' } }],
+      [{ amount: [500] }, { amount: { $gt: 10 } }],
+    ] as const) {
+      const err = refusalOf(filter, record);
+      expect(err.code).toBe('INVALID_FILTER');
+      expect(err.status).toBe(400);
+    }
+  });
+
+  it('the SAME filter over a record holding one scalar is compared, not refused — the refusal is per record', () => {
+    expect(m({ tags: 'm' }, { tags: { $gt: 'a' } })).toBe(true);
+    expect(m({ tags: 'a' }, { tags: { $gt: 'a' } })).toBe(false);
+    expect(m({ tags: 'm' }, { tags: { $between: ['a', 'z'] } })).toBe(true);
+    expect(m({ tags: 5 }, { tags: { $lte: 10 } })).toBe(true);
+  });
+
+  it('null, a missing field and a Date are untouched: no value is false, and a Date is a value', () => {
+    for (const record of [{ tags: null }, {}]) {
+      expect(m(record, { tags: { $gt: 'a' } })).toBe(false);
+      expect(m(record, { tags: { $lt: 'z' } })).toBe(false);
+      expect(m(record, { tags: { $between: ['a', 'z'] } })).toBe(false);
+    }
+    const at = new Date('2026-06-01T00:00:00.000Z');
+    expect(m({ at }, { at: { $gt: '2026-01-01T00:00:00.000Z' } })).toBe(true);
+    expect(m({ at }, { at: { $lt: '2026-01-01T00:00:00.000Z' } })).toBe(false);
+    expect(m({ at }, { at: { $between: ['2026-01-01T00:00:00.000Z', '2026-12-31T00:00:00.000Z'] } })).toBe(true);
+  });
+
+  it('equality against a stored list keeps the answer stage 2a pinned — only ordering moved', () => {
+    const record = { tags: ['m'] };
+    expect(m(record, { tags: 'm' })).toBe(false);
+    expect(m(record, { tags: { $eq: 'm' } })).toBe(false);
+    expect(m(record, { tags: { $ne: 'm' } })).toBe(true);
+    expect(m(record, { tags: { $in: ['m'] } })).toBe(false);
+    expect(m(record, { tags: { $nin: ['m'] } })).toBe(true);
+    expect(m(record, { tags: { $exists: true } })).toBe(true);
+    expect(m(record, { tags: { $null: false } })).toBe(true);
+  });
+
+  it('the message withholds the field and the stored value', () => {
+    const err = refusalOf({ secret_scope_column: { $gt: 'a' } }, {
+      secret_scope_column: ['usr_member_one', 'usr_member_two'],
+    });
+    for (const secret of ['secret_scope_column', 'usr_member_one', 'usr_member_two']) {
+      expect(err.message).not.toContain(secret);
+    }
+  });
+});

packages/formula/src/matches-filter.ts

@@ -156,9 +156,12 @@ function emptyFieldConstraintError(field: string, path: string): Error {
  * # What stays exactly as it was
  *
  * `$in` / `$nin` (the list operators — this is what they are FOR), scalars,
- * `null`, `Date`, and `{ $field }` references. The refusal reads the AUTHORED
- * comparand, never a resolved one: a `{ $field }` reference whose column
- * happens to hold an array is untouched.
+ * `null`, `Date`, and `{ $field }` references between single-valued columns.
+ * The shape refusal reads the AUTHORED comparand, never a resolved one. The
+ * two record-side refusals that reuse this error are judged per record in
+ * {@link evalOp}: [stage 2d] a `{ $field }` column holding a list or an object
+ * ({@link assertComparableReference}), and [stage 2e] a list or an object stored
+ * under an ordering operator ({@link ORDERING_OPERATORS}).
  *
  * # Why the message names nothing from the filter
  *
@@ -175,8 +178,9 @@ function arrayComparandError(): Error {
     'A single-value comparison in this filter received an array as its comparand: an array ' +
       'under "$ne", or an array in the equality position ({ "field": [ ... ] } or "$eq"), or ' +
       'under an ordering operator ("$gt", "$gte", "$lt", "$lte"), or as a member of an "$in" / ' +
-      '"$nin" list, or a column on either side of a { "$field" } comparison that holds a list or ' +
-      'an object. A list is not one comparable value. For "one of these values" use "$in", and ' +
+      '"$nin" list, or a column that holds a list or an object on either side of a { "$field" } ' +
+      'comparison or on the compared side of an ordering operator or "$between". A list is not ' +
+      'one comparable value. For "one of these values" use "$in", and ' +
       'for "none of these values" use "$nin" — the list operators the filter protocol declares; ' +
       'an ordering comparison takes one bound. It is refused rather than evaluated, because this ' +
       'evaluator compares strictly and no stored value ever equals an array: "$ne" matched EVERY ' +
@@ -211,6 +215,39 @@ const ARRAY_REFUSED_OPERATORS = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte'] as
  */
 const LIST_MEMBER_OPERATORS = ['$in', '$nin'] as const;
 
+/**
+ * [#19886 stage 2e] The ordering operators, whose STORED operand is judged on
+ * the record: a list or a plain object there is refused with
+ * {@link arrayComparandError}, per record, whatever the comparand — the mirror
+ * of stage 2d's array-comparand refusal with the list on the record's side.
+ *
+ * `record.tags > 'a'` on a `json` column or a `multiple` lookup lowers to
+ * `{ tags: { $gt: 'a' } }`, a legal shape; the list arrives on the post-image.
+ * `order` then compared the list's JavaScript string form: `['m'] > 'a'` is
+ * `'m' > 'a'`, and `{ a: 1 } < 'a'` is `'[object Object]' < 'a'` — both `true`.
+ * Measured through the real plugin-security write check on driver-sql and
+ * driver-memory, such a `check` admitted and stored the list-holding write.
+ * `$between` is `>=` and `<=` over the same `order`, and coerced the same way.
+ *
+ * The norm it follows is the production read driver's: driver-sql refuses every
+ * ordering comparison, and `$between`, against a column it stores as JSON text,
+ * by DECLARED type (#7398, `JSON_COLUMN_INCOMPATIBLE_OPERATORS`), because such a
+ * comparison "can never mean what the caller wrote". This evaluator has no
+ * schema, so it judges the VALUE: a record whose json column holds one scalar is
+ * compared as before. `null` (no value; `order` is never reached) and `Date` (a
+ * comparand, not an object map) are untouched, and so is equality — a scalar
+ * `$eq` / `$ne` / implicit equality against a stored list keeps the answer stage
+ * 2a pinned.
+ *
+ * What this does NOT align: driver-memory's read, a frozen test driver, compares
+ * a stored list element by element and keeps returning those rows, so its write
+ * check and its read part here (declared on #15104, as for stage 2d's `$field`
+ * half). And a list written into a scalar column (`amount: [500]` into a
+ * `number`) under an ordering check is refused here too, where it was admitted
+ * and stored stringified.
+ */
+const ORDERING_OPERATORS = ['$gt', '$gte', '$lt', '$lte', '$between'] as const;
+
 /** One comparable value: not a list, and not a plain object (a `Date` is a value). */
 function isNonScalarValue(value: unknown): boolean {
   return Array.isArray(value) || isOperatorMap(value);
@@ -347,6 +384,12 @@ function evalOp(actual: unknown, op: string, raw: unknown, record: Record<string
   // [#19886 stage 2d] A `{ $field }` comparison whose column holds a list or an
   // object ON THIS RECORD — either side. See {@link assertComparableReference}.
   if (isFieldReference(raw)) assertComparableReference(actual, op, raw, record);
+  // [#19886 stage 2e] An ordering comparison whose STORED operand holds a list
+  // or an object on this record, whatever the comparand. See
+  // {@link ORDERING_OPERATORS}.
+  if ((ORDERING_OPERATORS as readonly string[]).includes(op) && isNonScalarValue(actual)) {
+    throw arrayComparandError();
+  }
   const v = resolveValue(raw, record);
   // [#14104] An offset reference whose base is NULL is FALSE for every
   // operator — see {@link NO_OFFSET_BASE}. Before the switch, so `$ne`'s