Skip to content

Commit 81e69ca

Browse files
fix(service-analytics)!: the read scope, the where tree and the draft preview take the shared lowering's bound and NULL guards; their own copies are deleted (#21553)
Fixes #21417 Clause-②: no (narrowing) #5930 step 4 (`domain:services`) covers three faces: - **F9**, the analytics read scope; - **F10**, the analytics `where` tree, its native compiler and its `/analytics/sql` echo; - **F11**, the draft preview. On all three, the shared lowering (`lowerFilterCondition`, `@objectstack/spec/data`) is now the one source of the whole-day bound, the `$between` split and the NULL-polarity guards, and each face's own copy is deleted. With F11 in, nothing remains open on the card. Claim: the PM's `Claim:` comment 5962872882 and its revision 1, 5963878819 (Q1 = A: one PR; Q2 = A: F11 in, with one `analytics-service.ts` call site). Branch `claude/issue-21417-analytics-faces-delete-copies`. Base `0b8239111`, with `origin/main` merged twice (no rebase). ## Each face and its typed column reader | Face | Copy deleted | Reader that now carries the rule | |:--|:--|:--| | F10, native SQL strategy | `buildFilterClause`'s bare-day `lte` arm; the `dateRange` window arm's own widening; the `$not`-operand rewrite and the #5298 leaf wrap with their polarity tables (`filter-normalizer.ts`) | `declaredDatetimeLowering(ctx, target, 'type-blind')`, the context's `declaredFieldType` hook, which the plugin answers from the engine registry (`sourceFieldMeta`). A column the hook names no type for is read type-blind (ADR-0053 D-D1 item 7), because this face is the last seam before its statement runs. The window is the `{ $gte, $lte }` pair, lowered by the same reader (item 8). | | F10, ObjectQL strategy (engine hand-off and echo) | the echo's own window widening | the same hook, read `'as-written'` for a column it names no type for: the engine's `where` seam reads the object's own field map, and applies item 7 itself to an object with none. | | F9, read scope | the `$not`-operand rewrite and its three polarity tables; the `IS NULL OR` wrap on `$ne` / `$nin` / `$notContains`. F9 never had a whole-day copy (measured). | `ReadScopeCompileOptions.declaredValueShape`, filled by both consumers from the context's `declaredValueShape` hook. | | F11, draft preview | `lteBound` (the `$lte` and `$between` predicates); the window's own bare-day, last-day and `'~'`-suffix readings | `declaredPreviewLowering`, over the drafted object's declared types. `queryDataset`'s preview branch passes `sourceFieldMeta(dataset.object, field)?.type`; that one call site is the only `analytics-service.ts` edit. A declared `datetime` is rewritten, any other declared type is compared as written, and a column with no declared type is read type-blind (item 7). The window is the `{ $gte, $lte }` pair, matched by the same `matchesWhere` as the `where` (item 8). | The `$between` split in `fieldLeaves` stays. After the lowering it only meets a column declared something other than `datetime`, where it is structural: inclusive at both ends, which is the typed drivers' comparison. ## Measured, before and after **F9 and F10** were measured through the plugin's own composition over a real `ObjectQL` engine with `SqlDriver`. The databases were SQLite (better-sqlite3) and PostgreSQL 16.14 (a private server, session timezone America/New_York, stopped and deleted afterwards). Rows: - `signed_at` (datetime): `2026-07-27T10Z`, `2026-07-28T00Z`, `2026-07-28T10Z`, `2026-07-29T10Z`, null; - `due_on` (date): the same days; - `note` (text): `'2026-07-27'`, `'2026-07-28'`, `'2026-07-28 late'`, `'n'`, null. **F11** was measured through the production path: `AnalyticsServicePlugin`'s own composition (`sourceFieldMeta` from the engine registry, `draftRowsResolver` from a `protocol` service returning the same rows as a pending seed draft), then `queryDataset` with `previewDrafts`, over SQLite. "Before" is the same harness on the pre-F11 sources (`d2f452b88`); it reproduces the round-1 preview column cell for cell. The `note` cells. Every move lands on the engine's answer: | `note` cell | native before | preview before | after: native = preview = engine = ObjectQL | |:--|:--|:--|:--| | `$lte '2026-07-28'` | r1, r2, r3 | r1, r2, r3 | r1, r2 | | `$lte '9999-12-31'` (the carrier note's cell) | r1, r2, r3, r4 | r1, r2, r3 | r1, r2, r3 | | `$between ['2026-07-28', '2026-07-28']` | r2, r3 | r2, r3 | r2 | | `$between ['2026-07-28', '9999-12-31']` | r2, r3, r4 | r2, r3 | r2, r3 | | `$not: { $lte '2026-07-28' }` | r4, r5 | r4, r5 | r3, r4, r5 | | window `['2026-07-28', '2026-07-28']` | r2, r3 | r2, r3 | r2 | | window `['2026-07-28', '9999-12-31']` | r2, r3, r4 | r2, r3, r4 | r2, r3 | Native results are identical on SQLite and PostgreSQL. These are unchanged: - every `datetime` and `date` cell, on all three faces; - every cell of the ObjectQL face; - every read-scope cell; - every native cell of a host with no typed reader. **A host with no typed reader** reads every column type-blind, as ADR-0053 D-D1 item 7 prescribes for a seam that cannot read declarations. The engine lane's seams already do this (`engine.ts` `declaredDatetimeLowering`, the RLS seam). It does not read every column as "not datetime": read that way, the native face would have lost the last day of every `datetime` bound (measured: `datetime $lte a day` went from r1, r2, r3 to r1, r2). - On the native face, such a host keeps its old answers. - On the preview, one pair of cells moves. A preview built without `sourceFieldMeta`, or an object the registry does not hold yet, now answers `note $lte '9999-12-31'` with r1, r2, r3, r4 (was r1, r2, r3) and the `$between` to that day with r2, r3, r4 (was r2, r3). - Why it moved: the deleted `lteBound` sniffed instants on the last day. Every other type-blind seam does not, and this host now answers as they do. - These cells, where "answer as the typed drivers" cannot hold without a reader, are pinned by name. **The preview window's full-timestamp end** (the deleted `'~'` reading, "that instant and its own sub-values"). On a `datetime` column over SQLite: - A canonical end (`…T10:00:00.000Z`) answers as the engine, before and after. - An end spelled shorter than the stored value is now compared as text, exactly as the preview's `where` already compared the same `{ $gte, $lte }` pair (that column did not move). The cells that moved: | end | engine | preview before | preview after | |:--|:--|:--|:--| | `…T10:00`, main rows | r1, r2, r3 | r1, r2, r3 | r1, r2 | | `…T10:00:00`, main rows | r1, r2, r3 | r1, r2, r3 | r1, r2 | | `…T10:00`, rows at 09:59:59.999 / 10:00:00.000 / 10:00:00.500 / 10:00:30 / 10:01 | s1, s2 | s1, s2, s3, s4 | s1 | | `…T10:00:00`, same rows | s1, s2 | s1, s2, s3 | s1 | | `…T09:59`, same rows | none | s1 | none | Across the 10 full-timestamp `datetime` cells, 4 differed from the engine before this change and 5 differ after. Every difference, before and after, comes from comparing an instant written in a different spelling as text. The seat ruled that kind of defect out of this card ("a separate coercion defect, same class as #21505 … ⛔ do not build a coercion fix in this card"). See Acceptance notes. **The preview window over `Date` rows** (the BSON storage form): it is now compared as instants, like the `where`. Run against the pre-F11 evaluator, the new matrix's `dateRange` assertions on the BSON row population fail 10 times, and the evaluator answers them by the `Date`'s display text. They are green after. **SQL text that moved, rows unchanged:** - Every NULL guard prints once where it printed two or three nested copies. - A native window on a declared `date` prints the inclusive bound. - The `/analytics/sql` echo prints a window on a declared `date` inclusive, and a window on an undeclared column as the bound handed to the engine. - The echo prints a resolved preset that stops before its end with that end bound. Before, it printed the inclusive operator with no value bound (measured at base on `this_year`). ## The temporal conformance matrix, on every face `TEMPORAL_CASES`, with its token and `dateRange` axes, answers unchanged on every face: - through the native strategy, over a context with no hook and with the declared-type hook; - through the native and the ObjectQL faces of the plugin's composition; - through the read scope; - through the preview evaluator, read with the declared types and with none. The diff against the base sources is empty: the face files at `0b8239111` (F9/F10) and the preview at `d2f452b88` (F11) were checked out in this tree and restored afterwards, with the blob equal to HEAD. The one exception is the preview's `dateRange` on BSON `Date` rows, which is new coverage. It was red on the pre-F11 evaluator (above). ## Pins `analytics-faces-one-lowering.test.ts` (new): - **Enumeration:** no analytics source file uses a whole-day helper (`nextUtcCalendarDay`, `isUnboundedAbove`, `UNBOUNDED_ABOVE`, `lteBound`), and none uses a NULL-polarity copy's name; both holder maps are empty. A file "uses" a name when it imports, declares or calls it. A positive control proves the scan reaches the seams that call `lowerFilterCondition`. - **One source:** the native compiler and the echo emit, for each declared type, exactly the bound the lowering hands them. Every null predicate in F10's tree and F9's SQL is one the lowering wrote. - **Parity:** the 18-cell table, checked against `engine.find`: - on the native and ObjectQL faces, on SQLite and PostgreSQL (`OS_TEST_POSTGRES_URL`; a named skip without it); - on the read scope, on SQLite; - on the draft preview, through `queryDataset` with `previewDrafts` and the live path wired to throw. This covers `note lte 2026-07-28`, `between`, `$not` and the window cells, plus the undeclared-host row. - **No reader:** the type-blind answers of the native face and the preview, and the ObjectQL face handing the engine the bound as written. - **Matrix:** as above. `preview-temporal-conformance.test.ts` now runs the matrix through `evaluateAnalyticsQueryOverRows` under both readers, on both row populations. `preview-evaluator.test.ts` pins that `matchesWhere` compares the bound it is handed, and that the evaluator gives the whole day on a declared `datetime`, compares as written on a declared `date` or text column, and reads type-blind with no reader. Against the base sources, the new file is red on 21 tests for F9/F10 (enumeration 2, one-source 5, the text cells 7 on each database). Against the pre-F11 preview sources it is red on 7 more: the whole-day enumeration, the preview text cells `$lte`, `$between` and window one day, `$not`, window to the last day, and the undeclared-host row. Both runs are green on the matrix, the read scope and the reader-less native rows. Fifteen existing pin files moved from the stacked guards to the single guard: these are the step-3 rows marked "until the copy's deletion card". No id-set assertion changed. ## Ablations Each direction was predicted before the run. Each mutation went through `scripts/ablation-replace.mjs`, and every restore was proven by the blob equalling HEAD and `git diff HEAD` being empty. The pins import `src/` by relative path. | Ablation | Predicted | Observed | |:--|:--|:--| | A1 restore the native `lte` copy (import plus arm) | enumeration whole-day pin red; one-source native rows red; the seven text cells red on both databases; matrix green | 18 red, exactly those | | A2 unwire the native reader | native datetime cells red; native matrix red under both readers; reader-less datetime cells red; text and date cells green | 62 red, exactly those | | A3 unwire the read scope's reader | F9 parity and F9 matrix datetime rows red | 16 red: those plus the step-3 read-scope seam pins | | A4 unwire the echo's reader | one-source echo rows and the echo window pins red | 7 red: those plus the step-3 echo pin | | A5 restore the read scope's `IS NULL OR` wrap | enumeration NULL-polarity pin and the F9 one-source count red | 2 red, exactly those | | A6 restore `lteBound` (import, function, `$lte`, `$between` max) | whole-day enumeration red; preview text `$lte` a day, `$between` one day, `$not`, window one day red; the last-day text cells green (instant sniff); the matcher pin red | 6 red, exactly those | | A7a drop the declared type at the `analytics-service.ts` call site | the seven preview text cells red; everything else green | 7 red, exactly those | | A7b the preview reader answers "not datetime" | preview datetime cells (`$lte` a day, `$between` one day, `$not`, window one day) red, last-day ones green; the undeclared-host row red; both matcher pins red; the matrix's datetime bare-day cases red under both readers | 69 red: those (5 + 2 + 60) plus two more datetime bare-day preview pins in files not named, `date-range-array-arm-arity` (#17124 control) and the step-3 `where-door` F11 pin | ## Gates The pushed head `e08db24240` has tree `a8afd8be53`. The runs below were made on that same tree as local commit `cf643cce9e`. Before the first push, the three unpublished commits' co-author trailer was reworded to the model-free pair. Every rewritten commit's tree equals its original's, so no measurement below moved. - **Derived gates** (at `cf643cce9e`): `dispatch-gates --commands` derives 64 families. All 64 were run with exit codes, all 0, and `--ran` reconciles 64 derived / 64 run / 0 NOT MEASURED. - **Full `@objectstack/service-analytics` `test`** (at `c466dc77c8`, rewritten as `bbabcc6978`; the later commit only edits the changeset): 174 files, 4142 passed, 247 skipped. - **`typecheck`:** green. It covers the test files (171 listed). - **Downstream, on the rebuilt `dist`, all green:** - rest `analytics-*`: 22 files, 291 tests; - runtime `analytics-*`: 6 files, 41 tests; - driver-memory date-range conformance; - client; - dogfood `analytics-*`: 8 files, 70 tests. - **ESLint:** the run was narrowed to the 27 changed code files. The config's population is `**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}` minus its ignores, with no `parserOptions.project`, so type-aware linting is off and this diff cannot move any untouched file's result. ESLint's JSON reports 27 files, 0 errors and 0 warnings. Size: 28 files, +1418 / -1107. ## Acceptance notes - `assertDefinedComparands` in F9 and F10 is a refusal door, not the lowering's meaning, and is kept. - The package README's `sourceFieldMeta` paragraph names `$empty` as the reason to wire it. It is now also the reader that tells a `datetime` column from a text one, on the native face and the preview. No sentence is false, so it is not edited here. - F9 binds a bare-day comparand as written, with no storage-form coercion; this is filed as #21505. - The preview compares a temporal comparand as written, which is the same class of defect, and it is left as it is: - `{ signed_at: { $ne: '2026-07-28' } }` answers r1 to r5 on the preview and r1, r3, r4, r5 on the engine, before and after (the reader does not move it); - the full-timestamp window ends above also belong to this class. - The seat answered the report's question on the `'~'` deletion with A: keep it. The window is the `{ $gte, $lte }` pair (item 8), and the remaining off-engine cells belong to the draft-preview part of #21505. - `docs/design/predicate-compilation-convergence.md` §4.1's face table still lists F9, F10 and F11 under "Deleted later". The F7 row was marked retired by its own deletion card (#21242). That file is outside this claim's surface, so it is named here for the seat rather than edited. - The PostgreSQL parity cells were not re-run at this head. The F11 commits touch only the preview evaluator and its call site, which those cells do not execute, and the `main` merged since touches no analytics, ObjectQL or SQL-driver source (three test files in `driver-sql` and `objectql`, nothing else there). --- _Generated by [Claude Code](https://claude.ai/code/session_01DiCSbmJrkzNhuEAier4VoJ)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent aa0d4b9 commit 81e69ca

28 files changed

Lines changed: 1418 additions & 1107 deletions
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
"@objectstack/service-analytics": minor
3+
---
4+
5+
fix(service-analytics)!: the analytics read scope, the `where` tree and the draft preview take the shared lowering's bound and NULL guards; their own whole-day and NULL-polarity copies are deleted (ADR-0053 D-D1 items 7 to 9)
6+
7+
Clause-②: no (narrowing)
8+
9+
<!-- adr-0087: not-required (no-migration-prescription) a change of how the native analytics strategy and the draft preview answer an ordering comparison on a column its host declares neither datetime nor date, and of how the draft preview reads a window end, not of anything an author writes: no spec key, spelling, export or stored shape moves. AnalyticsQuerySchema, CubeSchema, DatasetSchema and every RLS policy parse and save as before, the package index exports the same names with the same types, and no stored row is read or rewritten. What moves is the row set a bare-day upper bound selects on such a column, which now equals the engine's own answer for the same filter, and the row set the draft preview selects for a window, which now equals what it selects for the same bounds written as a where; so there is nothing for objectstack migrate meta to rewrite. The other categories are closed on facts: the package publishes (not unpublished); no ADR-0087 id covers a filter's bound semantics and this diff adds none (not registered / already-registered); and the change is runtime behaviour, not a declaration (not runtime-interface-only / type-surface-only). -->
10+
11+
**BREAKING**: this narrows the rows the native analytics strategy and the draft preview (`queryDataset` with `previewDrafts`) select for a bare-day upper bound on a column the host declares as neither `datetime` nor `date` — a `text` column, for example. It ships as `minor` under the launch-window convention for answer narrowings. No export, published type, accepted input or error code changes.
12+
13+
**What is deleted.** The native SQL strategy no longer reads a bare `YYYY-MM-DD` `$lte`, a `$between` maximum or an explicit `dateRange` end as "through that whole day" on every column, and no longer drops such a bound on `9999-12-31` whatever the column holds. The whole-day rule is applied once, by the shared `lowerFilterCondition` (`@objectstack/spec/data`), with the column's declared type, the reader the plugin already wires from the engine's registry (`sourceFieldMeta`): a declared `datetime` column keeps the whole day, and every other declared column is compared as written, as the engine compares it. The `/analytics/sql` echo renders the same lowering.
14+
15+
**The native face now agrees with the engine.** Measured through `AnalyticsService.query` (what `POST /api/v1/analytics/query` relays) in the plugin's own composition, on SQLite and on PostgreSQL 16, over a `text` column `note` holding `'2026-07-27'`, `'2026-07-28'`, `'2026-07-28 late'`, `'n'` and no value:
16+
17+
- `{ note: { $lte: '9999-12-31' } }` counted every row with a value (4). It now counts 3, the rows the engine's `find` returns: `'n'` sorts above `'9999-12-31'`.
18+
- `{ note: { $lte: '2026-07-28' } }` counted 3, the `'2026-07-28 late'` row included. It now counts 2.
19+
- `$between ['2026-07-28', '2026-07-28']` and a `dateRange` window of the same day counted 2; they now count 1. Their negation through `$not` gains the row the bound lost.
20+
21+
On a declared `datetime` or `date` column every answer is unchanged, on both strategies.
22+
23+
**A host with no typed reader** (a strategy context with no `declaredFieldType` hook, or an `AnalyticsService` built without `sourceFieldMeta`) reads every column type-blind, as ADR-0053 D-D1 item 7 prescribes for a seam that cannot read declarations: its native answers do not move. Pass `sourceFieldMeta` (the README shows how) to get the engine's answer on a non-temporal column.
24+
25+
**The `/analytics/sql` echo.** A `dateRange` window on a declared `date` column now prints the inclusive `<=` the engine runs, where it printed `<` the next day; on a column the host names no type for, it prints the bound the ObjectQL strategy hands the engine, as written. A preset window that stops before its end (`today`, `this_month`, …) now prints `<` its end instant with that instant bound, where it printed `<=` with no value bound. The NULL guards print once where they printed two or three nested copies of the same guard; every row set is unchanged.
26+
27+
**The draft preview now agrees with the engine too.** `queryDataset` with `previewDrafts` evaluates drafted seed rows in memory; it kept its own whole-day copy, read on every column. It now hands the evaluator the drafted object's declared types (`sourceFieldMeta`), and the shared lowering applies the rule with them: a declared `datetime` column keeps the whole day, any other declared column is compared as written, and a column the host names no type for is read type-blind (ADR-0053 D-D1 item 7). Measured through the plugin's own composition over the same rows, five of the preview's `note` cells moved, each onto the engine's answer: `$lte` a day 3 to 2, `$between` and a window of one day 2 to 1, a window to `9999-12-31` 3 to 2, and the `$not` gains the row. Its `$lte` and `$between` to `9999-12-31` already gave the engine's answer and are unchanged. Every `datetime` and `date` cell is unchanged.
28+
29+
- A preview window is now the `{ $gte, $lte }` pair the ObjectQL strategy hands the engine, matched like the same bounds in a `where`. Its end used to be read with a `'~'` suffix ("that instant and its own sub-values"), a reading no other face gives. Measured on a `datetime` column over SQLite, a canonical end (`…T10:00:00.000Z`) answers as before and as the engine. An end spelled shorter than the stored value is compared as text, as the preview's `where` already compared it: an end of `…T10:00` or `…T10:00:00` now leaves out the row stored at exactly that instant (the engine keeps it), and leaves out the rows inside that minute or second (the engine leaves them out too; the old reading kept them). Write a window end in full (`2026-07-28T10:00:00.000Z`) to get the engine's rows on the preview.
30+
- A window over rows that hold a `Date` (the BSON storage form a MongoDB-backed draft reads back) is compared as instants, like the preview's `where`; it was compared as the `Date`'s display text.
31+
- A host that wires no `sourceFieldMeta` (or an object the registry does not hold yet) reads every column type-blind. On a `text` column holding a value that sorts above `'9999-12-31'` (`'n'`), a `$lte` or `$between` maximum of `9999-12-31` now keeps that row, as every other type-blind seam does; the deleted copy left it out.
32+
33+
**Unchanged.** Every answer on a declared `datetime` or `date` column, on the native strategy, the ObjectQL strategy and the draft preview; every answer of the ObjectQL strategy; every answer of the read scope.

0 commit comments

Comments
 (0)