Repository navigation
Commit bf37b99
Fixes #20025
Clause-②: no
## What this changes
`service-analytics` compiles its own SQL for the text operators, in
`text-match-sql.ts` `textMatchPredicateSql`, and three faces call it
with `dialect: 'sqlite'`:
- `compileScopedFilterToSql` (`read-scope-sql.ts`), the read scope,
reached directly and through both strategies' scope merge;
- `NativeSQLStrategy.buildFilterClause`, the executed `where`;
- `ObjectQLStrategy.buildFilterClauseSql`, the `/analytics/sql` echo,
which its caller runs on the same database.
Its `sqlite` arm emitted `GLOB` for every shape. SQLite's `glob()` reads
its pattern AND the stored value as C strings, so each is cut at its
first U+0000. `driver-sql` left that construct in #19999 (the
comparand's cut) and #20024 (the stored value's cut); this package
re-emits the driver's construct table rather than importing it, and its
copy had kept `GLOB`.
The `sqlite` arm now mirrors `driver-sql`'s `textMatchPredicate` /
`sqliteLengthAwareTextMatch` on `main`, cell for cell:
| shape | SQLite predicate |
|---|---|
| `contains` (`$contains`, `$notContains`, `$icontains`) | `instr(col,
?) > 0` |
| `ends` (`$endsWith`), non-empty comparand | `coalesce(substr(CAST(col
AS BLOB), -length(CAST(? AS BLOB))), CAST(col AS BLOB)) = CAST(? AS
BLOB)` |
| `ends`, empty comparand | `instr(col, ?) > 0` |
| `starts` (`$startsWith`), comparand without U+0000 | `col GLOB ?`,
unchanged byte for byte |
| `starts`, comparand holding U+0000 | `instr(col, ?) = 1` |
- This is the wider split the card's suggested shape predates. `glob()`
also cuts the stored value, so every `contains` / `ends` comparand
moves, not only a comparand holding U+0000. Only `$startsWith` without
U+0000 stays on `GLOB`, because neither cut can change that answer and
it is the one shape an index serves.
- The fold is still `lower()` on both sides, ASCII-only (#4706 Q1 = A).
The four case-exact operators stay case-sensitive (Q2 = A); `instr()`
and the BLOB comparison are byte-wise.
- The negation is `NOT (…)`, which is NULL for a NULL value, as `NOT
GLOB` was. So the NULL-safe `$notContains` wrapper and the `$not`
totalisation compose unchanged (#5298).
- Every comparand refusal, and the non-text-column constant, still runs
before the text arm. Neither door is touched.
- The suffix arm binds its comparand twice, in placeholder order,
through the existing `TextMatchBind`. The `LIKE` arms already bind two
values, so none of the three call sites changes.
- Nothing is escaped on the new constructs. `instr()` and the BLOB
suffix have no pattern language, so `*`, `?` and `[` bind as written.
- `text-match-sql.ts`'s header now describes the `sqlite` arm as it is,
and names the new parity suite beside
`text-operator-case-exactness.test.ts`.
## Measured before the fix: the face table
Executed, not read, at `980bc05e5b`. That is `main` after PR #20046
landed, and it moved nothing in `text-match-sql.ts` since the dispatch
base `2274894cc4`. It ran on better-sqlite3 (SQLite 3.53.4) and sql.js
(3.49.1), each through its driver's own `execute()`. Every cell was
compared with `@objectstack/formula`'s `matchesFilterCondition`.
The grid: stored values with U+0000 at the start, middle and end,
without it, `''` and NULL; comparands with and without U+0000; each leaf
bare and under `$not`. All five faces answered alike, and so did both
engines: the direct read scope, the read scope through
`NativeSQLStrategy` and through the echo, the native `where`, and the
echo `where`.
| operator | comparand without U+0000 | comparand holding U+0000 |
|---|---|---|
| `$contains` | narrows → **matches JS** | widens → **matches JS** |
| `$notContains` | widens → **matches JS** | narrows → **matches JS** |
| `$startsWith` | matches JS (unchanged, still `GLOB`) | widens →
**matches JS** |
| `$endsWith` | widens and narrows → **matches JS** | widens → **matches
JS** |
| `$icontains` | narrows → **matches JS** | widens → **matches JS** |
| each of the above under `$not` | the opposite direction → **matches
JS** | the opposite direction → **matches JS** |
- **In this PR's pin grid** (`text-match-sqlite-nul.test.ts`, 128
cells), every face on both engines answered 88 cells differently from JS
at base, and answers 0 at head.
- **The ObjectQL execute face** goes through the engine to `driver-sql`,
which is already fixed on `main`. It is the control, and it answered all
128 like JS at base and at head.
- **The `unknown` arm on SQLite** emits `LIKE`, and SQLite's `LIKE` cuts
at U+0000 too. That was measured on both engines: `'a'` + U+0000 + `'b'`
is not `LIKE '%b%'`, and `'plain'` is `LIKE '%'` + U+0000 + `'%'`. The
arm is untouched. It is the residue for dialects nothing answered for,
PostgreSQL-like ones among them, and PostgreSQL has no `instr()`.
Choosing a construct there would be a dialect guess. It is reported to
the seat as a finding and noted in the header.
- **`$like` / `$ilike`** are refused on every analytics face, before
this arm: `READ_SCOPE_COMPILE_FAILED` on the read scope,
`INVALID_FILTER` / 400 on the `where`. This package compiles no
raw-pattern operator, so there is nothing to swap.
## Controllability (triage's point 1)
- **Comparand side: no caller-controlled producer measured.** The RLS
compile path (`RLSCompiler.compileFilter` → `compileCelToFilter`)
resolves a string method's argument from one of two places:
- a literal in the admin-authored policy;
- a string attribute of `current_user` (id, organization id, email).
The ids are system-generated. The email is refused at every auth entry
measured when it holds U+0000, and it is read-only on the user object.
Membership sets are arrays, which the string methods refuse.
`plugin-sharing` emits id / owner membership scopes, not text operators.
Sharing-rule criteria are admin-authored and evaluated through
`engine.find`, the driver face that is already fixed.
- **Stored-value side: a caller-controlled producer exists.** It is any
persona that holds create or edit on an object with a text field. The
engine's write path stores U+0000 unchanged under a non-system context
(executed). Reaching a read scope also needs a deployment-authored
`using` predicate with a string method over such a field. No shipped
example authors one.
## Tests (at `d000746b1d`)
- **New `src/__tests__/text-match-sqlite-nul.test.ts`, 29 tests.** It
covers:
- the 128-cell grid on every face, on better-sqlite3 and sql.js, each
through its driver's `execute()`. That is the transport `plugin.ts`
uses, and a bare sql.js bind would cut a comparand holding U+0000 before
any construct saw it;
- each cell equal to the JS answer AND to `driver-sql`'s own rows on the
same engine. That is the parity pin that keeps the two construct tables
from drifting;
- literal pins for the headline cells;
- the compiled constructs per shape, including `$startsWith` without
U+0000 emitting `GLOB` and its escaped pattern byte for byte on all
three compilers.
It was committed first, red at base (`ced01cbad7`): 14 failed and 15
passed; each face on each engine had 88 of 128 cells differ;
`driver-sql` had 0.
- **Respelled pins, never loosened.** Four existing files asserted
`GLOB` for shapes that no longer compile to it:
`text-operator-case-exactness.test.ts`, `icontains-dialect-sql.test.ts`,
`sql-dialect-vocabulary.test.ts` and
`text-operator-non-text-column.test.ts`.
- Each now names the exact per-operator construct.
- The read-scope and echo loops check a per-operator regex plus "not `
LIKE `", where they used one `/GLOB/`.
- The `$endsWith` / `$startsWith` compiled-text rows gained SQL
assertions they did not have.
- `pnpm --filter @objectstack/service-analytics test`: 121 files, 2718
tests passed. `typecheck` (`tsc --noEmit`, whose program lists the
touched test files) exit 0.
### Ablations: each negative pin, one shape at a time, through
`scripts/ablation-replace.mjs`
The subject is `src/text-match-sql.ts`, imported relatively by every
face, so vitest reads source and no `dist/` is involved. Every leg
landed on disk (the anchor count went from 1 to 0 and the blob changed)
and was restored with its blob equal to HEAD (`e913134711cb`) and `git
diff HEAD` empty.
| leg | mutation | red |
|---|---|---|
| A1 | `contains` back on `GLOB` | 21 tests; 60/128 cells per face and
engine |
| A2 | `ends` back on `GLOB` | 15 tests; 22/128 cells |
| A3 | `starts` holding U+0000 back on `GLOB` | 11 tests; 6/128 cells |
| A4 | `starts` without U+0000 moved off `GLOB` | 4 compiled-text tests
(the byte-for-byte pins); rows unchanged, as expected |
| A5 | empty `ends` sent to the suffix construct | 11 tests; 2/128 cells
(`''` bare and under `$not`) |
| A6 | `coalesce()` removed from the suffix | 14 tests; 12/128 cells
(`''` under `$not`) |
The first A6 attempt was a no-op. Its replacement text already occurred
inside its own anchor, so the tool refused the leg and ran nothing. It
was re-run with a non-overlapping anchor (the row above).
`EXPLAIN QUERY PLAN` over an indexed TEXT column, on both engines:
`$contains` / `$endsWith` scanned under `GLOB` and still scan, and
`$startsWith` without U+0000 keeps its `SEARCH … USING INDEX`.
## Gates (at `d000746b1d`)
- **The dispatch-gates derivation**: `node scripts/pm/dispatch-gates.mjs
--repo objectstack-ai/objectstack --commands`, run from the real diff.
It gave 60 families, identical to the dispatch-time list.
- 58 exit 0.
- 2 are NOT MEASURED: `check:dual-build-cjs-loads` and
`check:type-check-debt` both exited 3 with PREREQUISITE NOT MET, because
they read the whole repo's built closure and this worktree built only
the package's own. CI builds it first.
- `--ran` reconciliation: 60 derived, 58 run, 2 NOT MEASURED, 0 unrun.
- `GITHUB_TOKEN=… node scripts/check-issue-citations.mjs`: every
citation this change adds resolves.
- **Lint, a declared narrowing.** ESLint ran on the 6 changed TS files
(`--no-inline-config --format json`): 6 files, 0 errors, 0 warnings.
- The population is the config's `packages/**/*.{ts,…}` block; none of
the six is in `NEVER_LINTED`.
- Type-aware linting is not enabled: `--print-config` shows
`parserOptions` without `project`. So this diff cannot move a verdict on
an untouched file.
- The full `pnpm lint` is CI's.
## Acceptance notes
- **Surface.** The claim named `text-operator-case-exactness.test.ts`
and new test files. Three more existing test files carried `GLOB` pins
for the moved shapes and are respelled here (listed above). No call
site, refusal door, driver, spec or formula file is touched.
- **Stale prose, no assertion reads it.** These comments still say the
SQLite arm is `GLOB` for every shape: `like-pattern.ts` (the
`$icontains` note), `read-scope-sql.ts` (the `textMatch` and
`$icontains` docblocks), `strategies/native-sql-strategy.ts`,
`strategies/objectql-strategy.ts` and `strategies/types.ts`. They ride
the next PR that touches those files. Carrier: none.
- **Observation: `$icontains: ''`.** It is not a U+0000 cell, and this
change does not move it.
- These compilers, and `AnalyticsService.query` in-process, answer every
non-NULL row for it, as they did under `GLOB '**'`.
- The spec's `FILTER_TEXT_CASES` declares it refused (`INVALID_FILTER`),
and `driver-sql` refuses it.
- Reachability through the REST door is not measured. It is reported to
the seat with its evidence.
- `main` moved during this run only by `3557f85fa5` (a `driver-turso`
README), which touches nothing here. No merge was needed.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Evb5jFDZGKQE9KG4jbMfMF)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
1 parent 1872e52 commit bf37b99
7 files changed
Lines changed: 568 additions & 52 deletions
File tree
- .changeset
- packages/services/service-analytics/src
- __tests__
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
28 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
246 | 247 | | |
247 | 248 | | |
248 | 249 | | |
249 | | - | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
250 | 255 | | |
251 | | - | |
| 256 | + | |
252 | 257 | | |
253 | | - | |
254 | | - | |
| 258 | + | |
| 259 | + | |
255 | 260 | | |
256 | 261 | | |
257 | | - | |
| 262 | + | |
258 | 263 | | |
259 | | - | |
| 264 | + | |
260 | 265 | | |
261 | 266 | | |
262 | | - | |
| 267 | + | |
263 | 268 | | |
264 | 269 | | |
265 | 270 | | |
| |||
290 | 295 | | |
291 | 296 | | |
292 | 297 | | |
| 298 | + | |
| 299 | + | |
293 | 300 | | |
294 | | - | |
| 301 | + | |
295 | 302 | | |
296 | | - | |
| 303 | + | |
297 | 304 | | |
298 | 305 | | |
299 | 306 | | |
| |||
455 | 462 | | |
456 | 463 | | |
457 | 464 | | |
458 | | - | |
459 | | - | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
460 | 468 | | |
461 | 469 | | |
462 | 470 | | |
| |||
465 | 473 | | |
466 | 474 | | |
467 | 475 | | |
468 | | - | |
469 | | - | |
| 476 | + | |
| 477 | + | |
470 | 478 | | |
471 | 479 | | |
472 | 480 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
417 | 417 | | |
418 | 418 | | |
419 | 419 | | |
420 | | - | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
421 | 424 | | |
422 | 425 | | |
0 commit comments