Skip to content

Commit 35dfb81

Browse files
fix(service-analytics): the ObjectQL face echoes a date bucket in the driver's own expression, so SQLite runs it (#21587)
Fixes #21441 Clause-②: yes (widening) ## What changes The ObjectQL face's echoed `sql` and the `POST /api/v1/analytics/sql` body now print a date-bucketed dimension in the bucket expression the driver itself groups by for its dialect. Before, `generateSql` printed `date_trunc('GRANULARITY', col)` on every dialect. SQLite refuses that, and PostgreSQL answers timestamps where the face answers `2026-01`. The route is the one the seat answered for Q1 (A), with Q2 = A from triage: the mechanism governs on every dialect. - **`driver-sql`**: one public member, `SqlDriver.dateBucketSql(objectName, field, granularity)`. It returns `knex.raw(sql, bindings).toQuery()` over the unchanged `buildDateBucketExpr(field, granularity, objectName)`, or `null` where that returns `null`. No change to what `buildDateBucketExpr` returns, and no spec member. - **`service-analytics`**: - `strategies/types.ts`: one optional context member, `dateBucketSql`, beside `sqlDialect`. - `analytics-service.ts`: one optional `AnalyticsServiceConfig.dateBucketSql` and one `baseCtx` pass-through line. - `plugin.ts`: wires the hook from `getDriverForObject`, as `sqlDialect` is wired (structural read, `typeof` guard, `undefined` on every tier that cannot answer). - `objectql-strategy.ts`: `dimExpr` prints the hook's answer, and keeps `date_trunc` where nothing answers. The "REPRESENTATIVE" docstring sentence is narrowed to exactly those cases. The false comment ("the SQL shape the driver's own bucketing implements") is corrected. - **`driver-turso`**: the comment that said `SqlDriver` emits `date_trunc` is corrected (comment only). `REMOTE_FACE_ANSWERS` gains one row, `dateBucketSql: 'inherited'`. Its `satisfies` pin over every key of `SqlDriver` fails the package's build until every public `SqlDriver` member is classified, so the ruled driver member forces this row. The row is not on the package's public surface: it is not exported from the index, and tsup drops it from `dist/`. There is no second bucketing table and no dialect branch in `service-analytics`. ## Where the echo keeps `date_trunc` The hook answers nothing, and the bucket stays representative, in four cases: 1. No hook is wired. 2. The driver has no bucket expression (a non-SQL driver). 3. The granularity is one the driver buckets in memory (`week` on SQLite: `buildDateBucketExpr` returns `null`). 4. The query has a non-UTC `timezone`. Case 4 is the strategy's own gate (`zone && zone !== 'UTC'`). It mirrors objectql's `tzRequiresInMemory` (ADR-0053 Phase 2, D2). A non-UTC zone makes the engine bucket in memory on that zone's calendar, which the driver's UTC expression does not describe. Measured: with `timezone: 'Asia/Shanghai'` the face answers `2026-01: 20, 2026-02: 8`, while the driver's UTC expression would answer `27, 1`. **Not gated: a measure `filter`.** The engine also buckets in memory when a measure carries a `filter` (`hasAggregationFilter`). It does so on the same UTC calendar the driver's expression is held to ("Must match `bucketDateValue()` exactly"), so there the driver expression answers the face's keys, and the echo uses it. Measured on both engines (pinned below). ## Measured **Premise, at `main` `0bddffd55b`.** Measured through the real `createDispatcherPlugin` mount (`POST /api/v1/analytics/query` and `/sql`), default composition, with `SqlDriver` on better-sqlite3 and on a live PostgreSQL 16.14 (private cluster, stopped and removed afterwards). Each query ran 0 raw statements and 1 engine aggregate, so these are ObjectQL-face answers. | cell | the driver ran | echo (= `/sql` body) | the echo, run | |:--|:--|:--|:--| | SQLite month / quarter (date and datetime column) | `strftime('%Y-%m', ...)` / `(strftime('%Y', ...) \|\| '-Q' \|\| ...)` | `date_trunc('month' / 'quarter', col)` | `no such function: date_trunc` | | SQLite week | `select *` (in-memory bucketing) | `date_trunc('week', col)` | `no such function: date_trunc` | | PG month / quarter / week | `to_char((col)::timestamptz AT TIME ZONE 'UTC', 'YYYY-MM' / 'YYYY"-Q"Q' / 'IYYY"-W"IW')` | `date_trunc(...)` | runs; keys `2026-01-01T00:00:00.000Z` where the face answers `2026-01` | | any engine, `timezone: 'Asia/Shanghai'` | `select *` (in-memory bucketing) | `date_trunc('month', col)` | SQLite refuses; PG answers UTC buckets | **After, at `a61c6d79a9`** (same harness, same mount): - SQLite month and quarter echo `strftime(...)` on both column types, and run with the face's row count. - PG month, quarter and week echo the driver's `to_char(...)` and run. - SQLite week, and the non-UTC zone on both engines, keep `date_trunc`. - The measure-filter dataset echoes the driver expression. The harness was a scratch copy of `runtime/src/analytics-query-window-validity.test.ts`, deleted after the run. **The MySQL arm is by code read only.** No MySQL server was available. The member renders the driver's own `date_format(convert_tz(??, @@session.time_zone, '+00:00'), ...)` arm through the same `toQuery`. **Turso remote face, measured with a scratch harness (deleted).** A `TursoDriver` in remote mode over a libSQL `file:` client answers `dateBucketSql` byte-identically to the local face, with no connection. libSQL runs it (`2026-01`, `2026-Q1`). Week answers `null` on both faces. Remote mode advertises an empty `queryDateGranularity`, so the engine buckets there in memory, on the UTC keys this expression answers. ## Pins `packages/services/service-analytics/src/__tests__/objectql-echo-date-bucket.test.ts`, default plugin composition. SQLite runs every time; live PostgreSQL runs behind `OS_TEST_POSTGRES_URL`, which no CI step sets for this package (a named skip there). Run at the final head `5032b8f3e9` with live PG 16.14: **16 passed (16)**, 8 of them live PG. - **Month and quarter** (SQLite), and **month, quarter and week** (PG), on a `date` and a `datetime` column: - the echo equals `generateSql` (the `/sql` body), with no params; - it selects and groups by an expression that is not `date_trunc` and that the driver's own aggregate statement contains; - run through the engine's raw-SQL bridge, it answers the face's rows. - **A measure `filter`** (the engine aggregates in memory): the echo carries the driver expression, and run with its params it answers the face's rows. - **FALLBACK, week on SQLite**: the driver grouped nothing, and the echo keeps `date_trunc('week', closed_on)`. - **FALLBACK, non-UTC `timezone`** (both engines): the face answers the zone's calendar, and the echo keeps `date_trunc('month', closed_at)`. - **FALLBACK, no hook**: an `ObjectQLStrategy` whose context names no hook echoes `date_trunc('month', closed_on)`. The dispatch's pin "on SQLite, week bucketed echoes run" does not hold under the ruled fallback. The SQLite driver buckets week in memory (`dateGranularityCapabilities.week` is false), so the hook answers nothing there and the echo keeps `date_trunc`, which SQLite refuses. It is pinned as a fallback instead. ## Ablations Each was predicted first, run through `scripts/ablation-replace.mjs` in wrap mode, and its restore was proven by the tool: blob equals the HEAD blob, and `git diff HEAD` is empty. The subject is `src`, imported relatively by the pin, so no `dist` leg applies. - **A. Hook wiring removed** (the plugin's `dateBucketSql,` config line replaced by a comment). - Predicted: red, 12 failed / 4 passed. - Observed: red, **12 failed / 4 passed**, every failure `expected 'date_trunc(...)' not to contain 'date_trunc'`. The 4 fallback pins stayed green. - Restore: blob `e1378f313e49` equals HEAD's. - **B. The non-UTC gate removed.** - Predicted: red, 2 failed / 14 passed. - Observed: red, **2 failed / 14 passed**: the two non-UTC pins, which got `strftime(...)` and `to_char(...)` where they expect `date_trunc`. - Restore: blob `3769d2062c91` equals HEAD's. ## Verification At final head `5032b8f3e9` unless noted: - `pnpm --filter @objectstack/service-analytics test` (no PG, as CI runs it): 176 files, **4160 passed**, 261 skipped, exit 0 (at `1e3cc1fc72`; the only later change is the changeset file). - The same suite with live PG 16.14 set: 4418 passed, **1 failed**. The failure is `read-scope-temporal-coercion.test.ts`'s premise check, "the server is not on UTC": this private server ran `Etc/UTC`. That is environmental, not this change. - `pnpm --filter @objectstack/service-analytics typecheck`: exit 0. `tsc --listFiles` reaches the new test file. - `pnpm --filter @objectstack/driver-sql test`: 216 files, **3611 passed**, 204 skipped, exit 0. `driver-sql typecheck`: exit 0. - `pnpm --filter @objectstack/driver-turso test`: 88 files, **2366 passed**, 33 skipped, exit 0. `driver-turso typecheck`: exit 0. - Gates: `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands` derives 67 commands. All 67 ran at `5032b8f3e9` and exited 0. `--ran` reconciliation: "67 derived, 67 run, 0 NOT-MEASURED, 0 UNRUN". `check:dual-build-cjs-loads` first answered `PREREQUISITE NOT MET` (no `dist`) and was re-run after a root build. - Lint, a declared narrowing: 1. **Population**, read from eslint's own config: of the 8 changed paths, the 7 `.ts` files are linted, and the changeset `.md` is ignored ("no matching configuration"). 2. **Count**, from `--format json`: 7 files linted, 0 errors, 0 warnings, at `5032b8f3e9`. 3. **Invariance**: `eslint.config.mjs` never enables type-aware linting (no `parserOptions.project`), so this diff cannot move an untouched file's verdict. The full `pnpm lint` is CI's. ## Acceptance notes - **Comment drift left in place (excluded):** the `objectql` `engine.ts` comment above `tzRequiresInMemory` still calls native driver bucketing `date_trunc`. Open PR #21545 holds that file. Carrier: whoever next edits that comment. - **Comment drift in a governed surface (noted, not filed):** `skills/objectstack-ui/rules/dashboards.md` ("Engine support") says Postgres buckets with `date_trunc`. `SqlDriver` emits `to_char(... AT TIME ZONE 'UTC', ...)`. `skills/**` is Tier H, so it is not touched here. Carrier: none. - **SQLite week stays unrunnable:** `date_trunc('week', ...)`, by the ruled fallback. Closing it needs the driver to bucket week in SQL on SQLite (a `driver-sql` capability decision), which this card does not make. - #21485 (the PG and MySQL `date`-column shift under a non-UTC server) is not addressed here. Because the echo now renders the driver's expression, its fix reaches the echo with no second edit. - Merged `origin/main` once (`cc645f2385`, docs-only, disjoint). A later `main` commit (`a7ab047cf6`, `packages/rest` only) is not merged; the queue rebuilds on it. --- _Generated by [Claude Code](https://claude.ai/code/session_01DiCSbmJrkzNhuEAier4VoJ)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent d2e687f commit 35dfb81

11 files changed

Lines changed: 444 additions & 17 deletions

File tree

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
"@objectstack/service-analytics": minor
3+
"@objectstack/driver-sql": minor
4+
"@objectstack/driver-turso": patch
5+
---
6+
7+
fix(service-analytics): the ObjectQL face echoes a date-bucketed dimension in the bucket expression the driver itself groups by, so SQLite runs the statement it prints
8+
9+
Clause-②: yes (widening)
10+
11+
**Before**, the ObjectQL strategy printed every date-bucketed dimension as `date_trunc('<granularity>', col)` in the `sql` it echoes and in the `POST /analytics/sql` body, on every dialect. The native strategy declines a granularity, so every bucketed query lands on this face. Measured through `POST /api/v1/analytics/query` and `POST /api/v1/analytics/sql` in the default composition: the rows were right. On SQLite the echo failed with `no such function: date_trunc` (month, quarter and week). On PostgreSQL 16.14 it ran but answered `2026-01-01T00:00:00.000Z` where the face answers `2026-01`. The driver groups by `strftime('%Y-%m', …)` on SQLite and `to_char((…)::timestamptz AT TIME ZONE 'UTC', 'YYYY-MM')` on PostgreSQL.
12+
13+
**Now** the echo prints the driver's own expression, so it runs on that dialect and answers the face's bucket keys.
14+
15+
- **`@objectstack/driver-sql`**: `SqlDriver.dateBucketSql(objectName, field, granularity)` returns the expression `aggregate` groups by, rendered as SQL text: the existing `buildDateBucketExpr`, unchanged, with each identifier quoted by the dialect. It returns `null` for a granularity the dialect buckets in memory (`week` on SQLite). The MySQL arm (`date_format(convert_tz(…))`) is checked by code read only, because no MySQL server was available.
16+
- **`@objectstack/service-analytics`**: the new optional `AnalyticsServiceConfig.dateBucketSql` hook carries the expression to the ObjectQL strategy. `AnalyticsServicePlugin` wires it from the driver that serves the object, as it wires `sqlDialect`.
17+
- **`@objectstack/driver-turso`**: a comment that said `SqlDriver` buckets with `date_trunc` now names the SQLite `strftime` expression it emits. The inherited `dateBucketSql` answers on the remote face too: it renders the same SQLite expression with no connection, and libSQL runs it.
18+
19+
**Unchanged.** The rows every face answers. The echo keeps `date_trunc(…)` where nothing answers: a host that wires no hook, a driver with no bucket expression (memory, MongoDB), a granularity the driver buckets in memory, and a query with a non-UTC `timezone`, which the engine buckets in memory on that zone's calendar.

‎packages/client/src/envelope-caller-census.test.ts‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -487,6 +487,12 @@ const LEDGER: readonly LedgerRow[] = [
487487
method: 'analytics.query', receiver: 'service', count: 2, verdict: 'NOT_SDK',
488488
why: 'a MemoryAnalyticsService bound to `analytics`, called directly (no HTTP, no dispatcher envelope) to assert the cube face refuses a non-boolean $exists and answers find()\'s rows for true / false',
489489
},
490+
// ── [#21441] the ObjectQL face's date-bucket echo pin: producer reads only ──
491+
{
492+
file: 'packages/services/service-analytics/src/__tests__/objectql-echo-date-bucket.test.ts',
493+
method: 'analytics.query', receiver: 'service', count: 1, verdict: 'NOT_SDK',
494+
why: 'the real AnalyticsService that AnalyticsServicePlugin registers over a live engine, called directly (no HTTP, no dispatcher envelope) to read the ObjectQL face\'s rows and the echoed `sql` it runs against them',
495+
},
490496
{
491497
file: 'packages/client/src/analytics-automation-json-erasure.test.ts',
492498
method: 'analytics.meta', receiver: 'sdk', count: 2, verdict: 'PAYLOAD_DEPENDENT',
@@ -689,11 +695,16 @@ describe('#13079 §2 — positive controls on the matcher itself', () => {
689695
// nested-relation pin, and driver-memory's cube service
690696
// (`MemoryAnalyticsService`) in its own refusal suite. None of the
691697
// receivers is the client, and every file is pinned.
692-
expect(service.length, literalNote()).toBe(8);
698+
// [#21441] A fourth: `service-analytics`' date-bucket echo pin calls
699+
// the real AnalyticsService that `AnalyticsServicePlugin` registers
700+
// over a live engine, to read the ObjectQL face's rows beside its
701+
// echoed `sql`. Its receiver is that service, not the client.
702+
expect(service.length, literalNote()).toBe(9);
693703
expect([...new Set(service.map((s) => s.file))].sort()).toEqual([
694704
'packages/client/src/analytics-automation-json-erasure.test.ts',
695705
'packages/drivers/driver-memory/src/memory-exists-non-boolean-refusal.test.ts',
696706
'packages/rest/src/analytics-nested-relation-filter.test.ts',
707+
'packages/services/service-analytics/src/__tests__/objectql-echo-date-bucket.test.ts',
697708
]);
698709
expect(service.every((s) => s.method === 'analytics.query')).toBe(true);
699710
});
@@ -740,10 +751,11 @@ describe('#13079 §3 — every call site is classified', () => {
740751
expect(production, literalNote()).toEqual([]);
741752
});
742753

743-
it('records the split: 18 payload pins, 10 result-insensitive, 8 not-SDK', () => {
754+
it('records the split: 18 payload pins, 10 result-insensitive, 9 not-SDK', () => {
744755
expect(verdictTotal('PAYLOAD_DEPENDENT')).toBe(18);
745756
expect(verdictTotal('RESULT_INSENSITIVE')).toBe(10);
746-
expect(verdictTotal('NOT_SDK')).toBe(8);
757+
// [#21441] 8 -> 9: the date-bucket echo pin's producer call (§2).
758+
expect(verdictTotal('NOT_SDK')).toBe(9);
747759
// The three above are LEDGER sums and cannot move on a census reading;
748760
// this one is census-derived, so it carries the note. [#13874]
749761
expect(sdkSites.length, literalNote()).toBe(28);

‎packages/drivers/driver-sql/src/sql-driver.ts‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
* Supports PostgreSQL, MySQL, SQLite, and other SQL databases.
88
*/
99

10-
import type { DriverOptions, FilterCondition, SchemaMode } from '@objectstack/spec/data';
10+
import type { DateGranularityValue, DriverOptions, FilterCondition, SchemaMode } from '@objectstack/spec/data';
1111
// The ONE introspection contract (ADR-0015 / `ISchemaDiffService`). This
1212
// driver's introspection types are DERIVED from these rather than
1313
// re-declared next to them — see the `Introspection Types` region below.
@@ -6166,6 +6166,36 @@ export class SqlDriver implements IDataDriver {
61666166
return null;
61676167
}
61686168

6169+
/**
6170+
* [#21441] The date-bucket expression this dialect groups `field` by at
6171+
* `granularity` (the one {@link aggregate} runs), rendered as SQL text, or
6172+
* `null` where {@link buildDateBucketExpr} has none: a granularity this
6173+
* dialect buckets in memory (`week` on SQLite, see
6174+
* {@link dateGranularityCapabilities}), or a client this driver does not
6175+
* model.
6176+
*
6177+
* It is for callers that PRINT the statement an aggregate stands for rather
6178+
* than run it. `service-analytics`' ObjectQL face echoes a date-bucketed
6179+
* query as SQL (`/analytics/sql`). While it spelled the bucket itself it
6180+
* printed `date_trunc(…)` on every dialect: SQLite refuses that, and
6181+
* PostgreSQL answers a timestamp where this driver answers `2026-01`.
6182+
*
6183+
* Reading the expression from here keeps this driver the one source of its
6184+
* bucketing. The text is `buildDateBucketExpr`'s, unchanged, with each `??`
6185+
* rendered by knex (`Raw.toQuery`) through this dialect's own identifier
6186+
* quoting. The expression binds identifiers only, so the text carries no
6187+
* value placeholder. `objectName` goes in as the coercion key, as
6188+
* `aggregate` passes it, so a SQLite `Field.datetime` column that may still
6189+
* hold pre-canonical values renders the repair the GROUP BY runs (#3773).
6190+
*
6191+
* Read structurally by its caller, like {@link dialectName}. It is not a
6192+
* member of the `IDataDriver` contract.
6193+
*/
6194+
public dateBucketSql(objectName: string, field: string, granularity: DateGranularityValue): string | null {
6195+
const bucket = this.buildDateBucketExpr(field, granularity, objectName);
6196+
return bucket ? this.knex.raw(bucket.sql, bucket.bindings).toQuery() : null;
6197+
}
6198+
61696199
/**
61706200
* Schema ownership mode (ADR-0015). When not `'managed'`, all
61716201
* schema-mutating DDL is rejected by {@link assertSchemaMutable}. The

‎packages/drivers/driver-turso/src/turso-driver.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -940,6 +940,11 @@ export const REMOTE_FACE_ANSWERS = {
940940
// Pure functions of the registries the remote arms read themselves.
941941
temporalFilterValue: 'inherited',
942942
temporalFilterColumnSql: 'inherited',
943+
// The SQLite bucket expression, rendered by Knex's compiler, which needs no
944+
// connection; libSQL runs it. The remote `aggregate` buckets nothing (its
945+
// `queryDateGranularity` is empty), so the engine buckets in memory, on the
946+
// UTC calendar whose keys this expression answers.
947+
dateBucketSql: 'inherited',
943948
} as const satisfies Record<keyof SqlDriver, RemoteFaceAnswer>;
944949

945950
// ── Remote operation timeout ─────────────────────────────────────────────────
@@ -1609,7 +1614,8 @@ export class TursoDriver extends SqlDriver {
16091614
batchSchemaSync: true,
16101615

16111616
// Remote transport does NOT do native date bucketing. SqlDriver's
1612-
// `aggregate` — which emits `date_trunc`/`strftime` for structured
1617+
// `aggregate` — which emits its dialect's bucket expression (`strftime`
1618+
// here; see `SqlDriver.buildDateBucketExpr`) for structured
16131619
// `{ field, dateGranularity }` groupBy items — is only reached in
16141620
// local/replica mode; remote mode delegates `aggregate` to
16151621
// `RemoteTransport.aggregate`, which accepts only string group-by

0 commit comments

Comments
 (0)