Repository navigation
Commit e5cf27d
fix(objectql,core): a per-aggregation filter and having read a temporal comparand by the column's storage rule — one rule in core, shared with both drivers' where (#20202)
Fixes #20176
Clause-②: yes (widening)
A per-aggregation `filter` (`aggregations[i].filter`) and `having` on
`engine.aggregate` now read a temporal comparand by the column's storage
rule — the rule both drivers already apply to the same comparand in a
`where`. The rule lives in one place, `temporalStorageForm(value, kind)`
in `@objectstack/core`, and `driver-sql` and `driver-memory` call it
instead of each carrying a copy. Every `where` answer is byte-identical
before and after, and so is every refusal PR #20174 added.
## What changed
- **`@objectstack/core`** — new `utils/temporal-storage-form.ts`,
exported from the index: `temporalStorageForm(value, kind)` for `kind`
`'datetime' | 'date' | 'time'`. It is the body that `driver-sql`'s
`canonicalUtcDatetime` / `toDateOnly` / `canonicalTimeOfDay` and
`driver-memory`'s `storageDatetimeValue` / `storageDateValue` /
`storageTimeValue` each carried word for word. It is total,
operator-blind and does not map arrays. `temporal-comparand.ts` docs now
name it.
- **`driver-sql`** — the three functions' bodies call the core rule;
`storageDatetimeValue` still adds MySQL's literal spelling on top (the
dialect half stays in the driver). No other line moved.
- **`driver-memory`** — `coerceTemporalValue` calls the core rule; the
three private copies are deleted (nothing else imported them).
- **`@objectstack/objectql`**
- `having-filter.ts` `checkCondition` takes the column's temporal kind.
When it has one, the row's value and every comparand of `$eq` / `$ne` /
the four orderings / `$between` / `$in` / `$nin` / implicit equality are
put in the storage form first. A bare `YYYY-MM-DD` as the upper bound of
a `datetime` (`$lte`, a `$between` max) means the whole day — the
exclusive bound at the next day's midnight — from the spec's
`nextUtcCalendarDay` (ADR-0053 D-D), as
`SqlDriver.calendarDayUpperBoundRewrite` and `driver-memory`'s `$lte`
arm do. Presence tests, text operators and `{ $field }` references are
read as before. `matchesHaving`, `applyHaving` and
`matchesAggregationFilter` thread a class map down;
`declaredFieldClasses(fields)` is new (the object's declared fields,
classified by the same spec sets #20127's `aggregatedRowColumnClasses`
uses).
- `in-memory-aggregation.ts` — `applyInMemoryAggregation(rows, ast,
timezone?, fields?)`: the optional declared field map reaches each
per-aggregation filter. Without it the function answers as before.
- `engine.ts` `ObjectQL.aggregate` — reads the declaration once, keeps
`having`'s aggregated column classes (already computed for #20127's
`addDays` rule) and hands them to both `applyHaving` doors, and hands
the field map to `applyInMemoryAggregation`.
## The rows, measured
Real `InMemoryDriver` and real `SqlDriver` (better-sqlite3,
schema-synced), through `engine.aggregate` and through `POST
/api/v1/data/:object/query` (JSON round-tripped body), on a populated (6
rows, 3 groups) and an empty object; the per-aggregation filter on the
second of two counts, the `where` twin beside it. Base `cfe2387a3b` vs
head (code identical to `b48f417154`). Every empty-object cell is 0 /
`[]` at base and head.
| # | shape | base, populated (all 4 driver × door cells) | head |
`where` twin |
|:--|:--|:--|:--|:--|
| 1 | ISO instant `$gte` on a `date` | 1 | 3 | 3 |
| 2 | ISO instant `$eq` on a `date` | 0 | 2 | 2 |
| 3 | bare day `$lte` on a `datetime` | 2 | 3 | 3 |
| 4 | bare day as `$between` max on a `datetime` | 2 | 3 | 3 |
| 5 | epoch-ms `$gt` on a `datetime` | 0 | 3 | 3 |
| 6 | `Date` 10:00 on a `date`, `$gte` / `$lt` / `$eq` | 1 / 5 / 0 | 3 /
3 / 2 | 3 / 3 / 2 |
| 7 | `Date` `$gt` on a `time` | 0 | 3 | 3 |
| 8 | `having` ISO instant `$gte` on `max(date)`, both doors | c2 | c2,
c3 | the date rule: c2, c3 |
| control | zone-naive `'2026-02-01 09:00'` `$gt` on a `datetime` | 4 |
4 | 4 |
Rows 6 and 7 are in-process only on the engine door; over REST the
`Date` arrives as its ISO text, and that spelling moved the same way.
Rows 6/7 on the base were identical to the card's readings, so PR
#20174's instant lift did not move them.
The family H1 asked for, all now equal to their `where` twin on every
cell (base in brackets): `$ne` 4 (6), implicit equality 2 (0), `$in`
members 3 (0), `$nin` members 3 (6), `$between` endpoints 4 (3), ISO
`$lt` 3 (5) on a `date`; bare days as both `$between` endpoints 2 (1),
epoch-ms string 3 (6), epoch-ms `$lte` 3 (0) and `$eq` 1 (0), offset
instant `$gte` 4 (3) and `$eq` 1 (0), zone-naive `T` `$eq` 1 (0), ISO
without ms `$eq` 1 (0), `$ne` 5 (6), `$nin` 5 (6) on a `datetime`; ISO
instant `$gt` 3 (0) and `$eq` 1 (0), short wall clock `$eq` 1 (0),
`$lte` 3 (2), `$in` 2 (0), `$between` 3 (2), `.000` spelling `$eq` 1
(0), bare day `$lte` 0 (6), `Date` implicit 1 (0) on a `time`; `Date`
`$in` 2 (0), `$nin` 4 (6), `$between` 4 (3) on a `date`; the ISO row
under `$or` 3 (1), `$not` 3 (5), `$and` 2 (0); two operators on one
`datetime` 2 (1) and a bare-day `$lte` beside an author `$lt` 4 (3); a
bare-day `$lte` of today on `created_at` 6 (0). `having`, both doors: 15
more shapes on `max(date)`, `min(date)`, `min(datetime)`, `max(time)`, a
`groupBy` of a `date` / `datetime` field and a `day` bucket, each moved
to the date/datetime/time rule's answer.
## H2 — do the two drivers' rules agree?
Measured before lifting: `driver-memory`'s `coerceTemporalValue` against
`SqlDriver.temporalFilterValue` (better-sqlite3, pg and mysql2 clients,
field maps seeded, no connection) on 71 shapes × 3 kinds (`Date`s valid,
invalid and out of range, numbers incl. NaN / Infinity / past the Date
range, epoch strings, bare days incl. impossible ones, zone-naive and
zone-explicit timestamps, RFC 2822, wall clocks in and out of range,
junk, placeholders, `null`, booleans, objects, nested and mixed arrays,
bigint): memory = sqlite = pg on all 213 cells; mysql differs on the 36
`datetime` cells only by its literal spelling. Base to head, all 852
cells are byte-identical, so the re-pointing moved nothing.
One divergence sits in the operator-sensitive layer, not in the lifted
function: a bare day as a `$lte` / `$between` max on a **`time`** field.
`driver-memory` widens it (`$lt '00:00:00'`, no row), `driver-sql` does
not (`$lte '00:00:00'`, keeps a midnight row). Neither driver's emitter
is touched here. This position follows the spec: ADR-0053 D-D's table
defines the whole-day reading for `datetime`, and `TEMPORAL_TIME_CASES`
says it must not reach a time column. Reported, not filed here.
## Tests
- New:
`packages/objectql/src/engine-aggregate-temporal-storage-rule.test.ts`
(84 — the card's rows, the family, unmoved positions, `having` on both
doors, the shared `TEMPORAL_CASES` + token spellings +
`TEMPORAL_TIME_CASES` through a per-aggregation filter and through
`having`, and the no-declaration default);
`packages/rest/src/aggregation-filter-temporal-storage-rule.test.ts`
(13, REST + SqlDriver, per-aggregation count equals the `where` twin,
empty and populated, and `having` row 8);
`packages/drivers/driver-memory/src/memory-temporal-storage-form.test.ts`
(59 — `coerceTemporalValue` is the core rule, and the card's `where`
twins on this driver);
`packages/drivers/driver-sql/src/sql-driver-temporal-storage-form.test.ts`
(10 — `temporalFilterValue` is the core rule on three clients);
`packages/core/src/utils/temporal-storage-form.test.ts` (52).
- Whole suites at `b48f417154` (after merging `main`, full package
build): objectql `317 files / 5712 passed`; core `54 / 1410`;
driver-memory `54 / 1328`; driver-sql `188 passed, 11 skipped / 3099
passed, 170 skipped`; REST aggregate files (5 incl. the new one) `109
passed, 1 skipped`; service-analytics `128 / 3017`.
- Live PostgreSQL 16 (private server, server zone Asia/Shanghai, process
`TZ=America/New_York`): driver-sql whole suite `196 files passed, 3
skipped / 3806 tests passed, 86 skipped`;
`sql-driver-temporal-conformance` 103 with only its MySQL cell skipped.
MySQL: NOT MEASURED, no server in this container.
- Skewed process zone (America/New_York, Asia/Kolkata, Pacific/Chatham):
the new objectql + `engine-aggregate-filter`, driver-memory new +
temporal conformance, core new — all green in each zone.
- Typecheck `core`, `driver-memory`, `driver-sql`, `objectql`, `rest`:
exit 0 each; every new test file is in a typecheck program
(`--listFiles`); objectql test-typecheck debt unchanged at 40 files /
234 errors.
- Lint (declared narrowing): `eslint --no-inline-config --format json`
on the 13 changed `.ts` files → 13 files, 0 errors / 0 warnings / 0
fatal; `--print-config` resolves a config for each; `eslint.config.mjs`
enables no type-aware linting, so no untouched file's verdict can move.
## Reverse verification
Fix committed first. `ablation-replace` (wrap mode, `EXIT/INT/TERM` trap
on the absolute path) replaced the one threading site in `matchesHaving`
so no column ever gets its kind: anchor 1 to 0, blob `77400359cc` to
`7a2f5b7f14`. objectql rebuilt; `ablation-dist-preflight --absent`
passed on all 14 built files. New objectql pins `58 failed / 26 passed
(84)` — the 26 are the controls, the unmoved positions, the #20148
`Date` lift on an undeclared column and the kit cases whose canonical
comparands compare correctly as text. REST pins `11 failed / 2 passed
(13)` — the two controls. The harness in that state reproduced the base
table 5024 / 5024. Restored by `git checkout HEAD --`: blob back to
`77400359cc` = HEAD, `git diff HEAD` empty; rebuilt; preflight marker
present in 4 built files and the tree clean; pins `84` and `13` passed.
## Gates
`dispatch-gates --commands --repo objectstack-ai/objectstack` at
`b48f417154`: 69 commands (the clue list's 56 plus
`check-adr-0087-registration`, `check-empty-changeset`,
`release-rehearsal-clone --self-test`, `check:engine-double-contract`,
`check:objectql-double-limit`, `check:objectui-changeset`,
`check:pm-changeset-deadline-census`, `check:query-options-erasure`,
`check:type-check-coverage`, `check:type-check-debt`,
`check:where-matcher`). 68 exit 0; `--ran` reconciles 69 of 69, 0 NOT
MEASURED. One is red on purpose:
- `check-empty-changeset --base origin/main` exits 1 on
`.changeset/20148-aggregation-filter-where-doors.md` — a **deliberate
correction**, not a collision, and it needs a person's confirmation
here. That pending note said "Two readings still differ from a `where`"
(a `Date` with a time of day on a `date` field, and a `Date` on a `time`
field). This PR closes both, so the sentence would ship false in the
same release. The one-clause correction reads "Two readings still
differed from a `where` in this change alone … (#20176 closes both in
the same release, reading every temporal comparand at this position by
the column's storage rule)". Nothing else in the note changed.
Also run: the three roster gates whose roster sits under `packages/`
(`check:authz-resolver`, `check:error-code-casing`,
`check:filter-alias-parity`) and `check:published-readme-exports`, exit
0 each; `check-issue-citations --base 560b724` (the merge base): 28
citations, all resolve; a control-byte scan of the changed files found
none.
## Deviations
- The claim scopes `engine.ts` to "the per-aggregation loop only". The
loop runs before filter tokens resolve, so a comparand normalised there
would miss every `{today}`-style bound, which resolves to a bare day
later. The normalisation therefore happens where the filter is evaluated
(`checkCondition`), and the `engine.ts` edit is the threading: one
declaration read beside the existing `having` class read, and the class
maps passed to both `applyHaving` calls and to
`applyInMemoryAggregation`. `in-memory-aggregation.ts` carries the
per-aggregation filter's class map, not `having`'s (which the engine
hands to `applyHaving` directly).
- The re-pointing in `driver-sql` replaces the bodies of
`canonicalUtcDatetime`, `toDateOnly` and `canonicalTimeOfDay` (which
`temporalFilterValue` reaches through `coerceFilterValue`) rather than
only `temporalFilterValue`, so the driver's write and read paths use the
same single rule. The H2 measurement is the byte-identity evidence.
- The REST `where` twin on `InMemoryDriver` is measured (scratch
harness, 5120 cells) but not committed as a pin: `packages/rest` has no
`driver-memory` dependency. That driver's `where` half is pinned in its
own package.
- `.changeset/20148-aggregation-filter-where-doors.md`: one-clause
deliberate correction, above.
## Acceptance notes
- `driver-mongodb`'s `mongodb-temporal.ts` keeps its own copy of the
rule, in BSON `Date` form for `datetime`. Its instants agree with the
core rule, but it is outside this claim and was not re-pointed.
- `core`'s `temporal-comparand.ts` predicates (`readsAsInstant`,
`readsAsCalendarDay`, `readsAsWallClock`) restate the rule's regexes as
yes/no questions for the temporal-comparand door. Deriving them from
`temporalStorageForm` would remove the last mirror in `core`; not done
here.
- `ObjectQL.resolveNowDefault` spells the `date` / `time` forms of a
`Date` by hand; it agrees with the rule for every `Date` it is handed.
- A per-aggregation filter and `having` on a column whose class the
declaration cannot tell (no field map, a formula) keep the previous
comparison, including PR #20174's instant reading of a `Date` bound.
- An epoch-ms number against a `date` field is not read by the `date`
rule on either driver, and the two drivers' `where` answer differently
(driver-memory 0 of 6, driver-sql 6 of 6). This position answers 0,
before and after; reported separately.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 9dacf61 commit e5cf27d
15 files changed
Lines changed: 1179 additions & 227 deletions
File tree
- .changeset
- packages
- core/src
- utils
- drivers
- driver-memory/src
- driver-sql/src
- objectql/src
- rest/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
97 | 97 | | |
98 | 98 | | |
99 | 99 | | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
100 | 107 | | |
101 | 108 | | |
102 | 109 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
45 | | - | |
46 | | - | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
90 | | - | |
| 90 | + | |
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
108 | | - | |
| 108 | + | |
109 | 109 | | |
110 | 110 | | |
111 | 111 | | |
112 | | - | |
| 112 | + | |
113 | 113 | | |
114 | 114 | | |
115 | 115 | | |
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
123 | | - | |
| 123 | + | |
124 | 124 | | |
125 | 125 | | |
126 | 126 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
0 commit comments