Repository navigation
Commit fc0db22
fix(driver-sql): sum / avg accumulate in double on PostgreSQL and MySQL, as on SQLite and the rows path (#20486)
Fixes #20387
Clause-②: no
## What changed
`SqlDriver.aggregate` (`packages/drivers/driver-sql/src/sql-driver.ts`)
now makes PostgreSQL and MySQL accumulate `sum` / `avg` in double. That
is the arithmetic SQLite and the engine's rows path (`objectql`'s
`in-memory-aggregation.ts`) already use. This is route (a) of triage
5865067775, under the one-double policy that PR #20372's changeset
stated for the answer's type.
- `AGGREGATE_ACCUMULATION` is a `Record` over the spec's
`AggregationFunction`, a sibling of `AGGREGATE_ANSWER_KIND`. Its entries
are `avg: 'double'`, `sum: 'double-over-fractional'`, and `count` /
`count_distinct` / `min` / `max`: `'as-stored'`. A function added to the
enum without an entry fails `tsc`.
- `fractionalNumericFields` is a registry filled beside `numericFields`
in both fills and aliased for shards. It holds the declared columns
whose type stores fractions: `numericColumnFor(type).kind === 'exact'`
(`number`, `currency`, `percent`, `slider`, `progress`, `summary`) and
the driver's `float` alias.
- In `aggregate()`, on PostgreSQL and MySQL only, two cases take a
double operand: `avg` over a declared numeric or boolean column, and
`sum` over a fractional column. The operand is `cast(cast(x as text) as
double precision)` on PostgreSQL and `cast(cast(x as char) as double)`
on MySQL. The column stays one `??` binding.
- A `sum` over an integer-valued column (`rating`, the `integer` / `int`
aliases, a boolean) keeps the database's exact total. A column with no
numeric or boolean declaration keeps the database's own arithmetic.
SQLite is untouched.
**Why the operand is the column's text, not a plain cast.** The text is
what the SQL client hands `find()`, so it is what the rows path adds.
- On the exact-decimal columns the two are identical. Both servers turn
a decimal into a double through its text. Measured on 20,000 random
`numeric(65,30)` / `DECIMAL(65,30)` values per server: 0 differ.
- On a binary32 column they are not identical. A table created before
the exact-decimal columns (landed in `9cdffbe36`) still has `real`
(PostgreSQL) / `FLOAT(8,2)` (MySQL) under a `number` declaration. There
a plain cast adds the widened binary value, `0.30000000447034836` for
`0.1 + 0.2`, where `find()` reads `0.1` and `0.2`.
- Ablation 2 below pins this difference.
## Route picked by measurement: (a)
**H1: reproduced at base `75b216924`.** Measured through
`engine.aggregate` and `POST /api/v1/data/:object/query` on SQLite, a
private live PostgreSQL 16.13 (`Asia/Shanghai`) and a private live MySQL
8.0.46 (`+08:00`). The rows path was forced with a filtered sibling
aggregation. The two doors agree cell for cell.
| `having` (object with `number` w, `rating` st, `boolean` flag) |
SQLite native | SQLite rows | PG native | PG rows | MySQL native | MySQL
rows |
|:--|:--|:--|:--|:--|:--|:--|
| `s $eq 0.3`, `s $in [0.3]` (sum w of 0.1, 0.2) | none | none |
**kept** | none | **kept** | none |
| `a $eq 0.15`, `a $in [0.15]` (avg w) | none | none | **kept** | none |
**kept** | none |
| `s $eq 0.1 + 0.2` | kept | kept | **none** | kept | **none** | kept |
| `sa $eq 5 / 3` (avg st of 1, 2, 2) | kept | kept | kept | kept |
**none** (`1.6667`) | kept |
| `fa $eq 1 / 3` (avg flag of 1, 0, 0) | kept | kept | kept | kept |
**none** (`0.3333`) | kept |
Base values: PG and MySQL native `sum` = `0.3`, `avg` = `0.15`; every
other face `0.30000000000000004` / `0.15000000000000002`.
**At head** (driver-sql source identical from `07f81ea19` to
`4caf9e6ef`), same doors:
- `s $eq 0.3`, `s $in [0.3]`, `a $eq 0.15` and `a $in [0.15]` keep no
group on any face.
- `s $eq 0.1 + 0.2` and `a $eq (0.1 + 0.2) / 2` keep the group on all
six.
- `sa $eq 5 / 3` and `fa $eq 1 / 3` keep it on all six.
- PG and MySQL native answer `0.30000000000000004` /
`0.15000000000000002`.
**H2: confirmed, with the order question answered.**
- `sum(float8)` on PostgreSQL and `SUM(DOUBLE)` on MySQL add in scan
order, one value after another, as the rows path's `reduce` does. Raw
SQL over 0.1, 0.2, 0.3 answered `0.6000000000000001`, and `avg` answered
`0.20000000000000004`, on both. The JS left fold gives the same.
- SQLite 3.53.4, as bundled by better-sqlite3, answered `0.6` /
`0.19999999999999998`. It uses compensated (Kahan-Babuska-Neumaier)
summation, which SQLite has done since 3.43.
- With two addends, no order and no compensation scheme can move the
last place. The pin `0.1 + 0.2` is therefore deterministic across all
four faces. For three or more fractions it is not deterministic on
SQLite's native face: see the residual below.
- `count` / `min` / `max` are untouched (pinned).
- An integer column's `sum` stays exact (pinned). A `bigint` column
holding 2^53 + 1 and 1 answers `9007199254740994` on all three cells,
where adding doubles would give `9007199254740992`.
**H3: (b) does not hold, on SQLite, for the pin itself.** SQLite's
native `sum` over a REAL column adds the stored doubles,
`0.30000000000000004`. An exact rows path (`0.3`) would disagree with it
for exactly `0.1 + 0.2`, and changing SQLite's native face is outside
route (b). The scale half of H3 also holds: in `examples/` at
`4caf9e6ef`, 2 of the 70 fractional-family declarations carry a `scale`
(app-todo `estimated_hours` and `actual_hours`).
### In-place fix, declared: `avg` over an integer-valued column
Triage's route (a) names "a non-integer column". `avg` over an integer
column is the same defect class: a native decimal quotient against the
rows path's double. It sits in the same function and is closed by the
same operand. No other claim holds the file, and it runs in the same
gate family. So `avg` is `'double'` whatever the column holds. Evidence:
- MySQL rounds a `DECIMAL` average to `div_precision_increment` (4)
places: `1.6667` and `0.3333` above.
- PostgreSQL's `numeric` average rounds to 16 places before the
presenter rounds again. Over sums 1 to 3000 and counts 3 to 13 (27,962
non-integral pairs), 10 differ from JS division. Example: `11 / 9`
answered `1.2222222222222222`, JS `1.2222222222222223`.
- Pinned by the `nine` and `three` groups of the new test.
### Residual, stated and not fixed: SQLite's compensated sum
For three or more fractional addends, SQLite's native face can still
differ from every other face in the last place. `0.1 + 0.2 + 0.3`
answers `0.6` on SQLite native and `0.6000000000000001` everywhere else,
so `having { s: { $eq: 0.6 } }` keeps the group on SQLite native only,
at head.
- This was already true between SQLite's own two paths at base.
- Neither route reaches it. SQLite's `sum` cannot be made to add naively
in SQL, and PostgreSQL / MySQL cannot add with compensation in SQL.
- It is reported to the seat as a separate finding.
## Tests
- New:
`packages/drivers/driver-sql/src/sql-driver-20387-aggregate-double-accumulation.test.ts`.
It has 5 cases for each cell of the live matrix (`declareDialectCell`),
so the PostgreSQL and MySQL cells run in `Temporal Conformance (live PG
+ MySQL)`, which runs the whole driver-sql suite with both live URLs and
`OS_EXPECT_LIVE_DIALECT_MATRIX=1`:
- **The pin.** `sum` / `avg` over `number` and `currency` columns
holding 0.1 and 0.2 equal the rows path's arithmetic over `find()`'s own
rows, and the literals `0.30000000000000004` / `0.15000000000000002`. So
`having` `$eq 0.3` / `$in [0.3]` / `$eq 0.15` keep no group and `$eq 0.1
+ 0.2` keeps it: `having-filter.ts` compares the value through
`@objectstack/formula`'s `looseEq`, which is strict `===` for numbers
(seat edit after review 5875498653).
- **Binary32.** A `number` column retyped to `real` / `FLOAT(8,2)` adds
what `find()` reads.
- **Integer average.** `avg` over `rating` and `boolean` columns is the
double quotient (5 / 3, 11 / 9, 1 / 3).
- **Integer sum.** `sum` over an integer column keeps the exact total
(the 2^53 + 2 case).
- **Unchanged functions.** `count` / `min` / `max` are unchanged.
- Result: 15 passed (5 cases × SQLite, live PostgreSQL, live MySQL),
every cell exercised (verbose reporter).
- **Ablations.** The fix was committed first, and each mutation went
through `scripts/ablation-replace.mjs` in wrap mode. The test imports
`./sql-driver.js` (source), so no `dist/` leg exists. Each run was the
new file on all three cells:
1. The operand was disabled (`false &&`). Anchor 1 → 0, blob
`63e5dac455` → `5e33daf1c6`. **6 failed | 9 passed:** the PG and MySQL
pin (`expected 0.3 to be 0.30000000000000004`), binary32 (`0.3`) and
integer average (PG `nine avg(rating): expected 1.222222222222222…`,
MySQL `three avg(rating): expected 1.6667`). Every SQLite case, the
integer sum and count/min/max stayed green.
2. A plain cast replaced the text operand. Blob → `4929128349`. **2
failed | 13 passed:** the binary32 case on PG and MySQL only (`expected
0.30000000447034836 to be 0.30000000000000004`). The pin stayed green,
so the plain cast equals the text cast on exact decimals.
3. `sum: 'double'` dropped the integer exception. Blob → `958360a033`.
**2 failed | 13 passed:** the integer sum on PG and MySQL (`expected
9007199254740992 to be 9007199254740994`).
- All three failed in the predicted direction, on the predicted cells.
Each restore reported blob == HEAD (`63e5dac455`) and an empty `git diff
HEAD`, and `git status --porcelain` was empty afterwards.
- **Whole `@objectstack/driver-sql` suite** at `4caf9e6ef`, with live
PostgreSQL and MySQL, `TZ=America/New_York` and
`OS_EXPECT_LIVE_DIALECT_MATRIX=1`: 207 files passed, 4720 passed | 1
skipped. The reporter confirmed that all 3 dialects were exercised; the
1 skip is for a reason other than a missing backend.
- `packages/rest/src/rest-aggregate-numeric-having.test.ts`, the pins
from PR #20372, ran with both live URLs against this driver: 36 passed
(12 cases × 3 cells).
- **Typecheck** at `4caf9e6ef`: `@objectstack/driver-sql`, and its two
subclasses `@objectstack/driver-turso` and
`@objectstack/driver-sqlite-wasm`, all exit 0. The new test file is in
driver-sql's `tsc` program (`--listFiles`).
## Gates
Measured at head `4caf9e6ef`. That head is a true merge of `origin/main`
(`8e0285918`), after a full workspace build (`turbo run build`, 72 of 72
tasks).
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` derived 63 commands. All 63 ran, each exit
code captured before any pipe, and all 63 exited 0. `--ran`
reconciliation: 63 derived, 63 run, 0 NOT-MEASURED, 0 UNRUN.
- `origin/main` moved during the run, so the three `--base` gates were
also run with `--base 8e02859`, each exiting 0.
`check-changeset-no-major` has no PR payload locally; its level axis
reads the PR in CI.
- `check:driver-conformance` before and after: 50 covered, 0 DEBT, 0
exempt; dialect axis 8 suites, 0 in the DIALECT ledger. The ledger did
not move.
- **Lint, a proved narrowing.** `eslint --no-inline-config --format
json` over the two touched TypeScript files: 2 files, 0 errors, 0
warnings. `eslint --print-config` resolves both files with
`parserOptions` `ecmaVersion` / `sourceType` only, with no `project` and
no type-aware rules. So this diff cannot move the verdict on an
untouched file. The repo-wide `pnpm lint` is CI's.
- NOT MEASURED locally, owned by CI:
- the workspace type-check lanes, the `Test Core` shards, `Dogfood`,
`Build Core`, and `Temporal Conformance` as CI spells it;
- the five CI-variable families dispatch-gates names:
`check-issue-citations --census`, three `check-shard-attestation
--emit`, and `check-test-completeness`.
## Acceptance notes
- **Legacy binary32 columns change answer.** On a table created before
the exact-decimal columns, the native `sum` / `avg` over a `real` /
`FLOAT` column used to be computed as follows. PostgreSQL's `sum(real)`
accumulated in float4 (0.1, 0.2, 1234567.9 answered `1.2345681e+06`).
MySQL's `SUM(FLOAT(8,2))` answered a double shown to 2 decimals
(`123457.20` for 0.1, 0.2, 123456.9). Both now answer the rows path's
double, the sum of what `find()` reads: `1234568.2` on the PostgreSQL
example, where a plain cast would have answered `1234568.1750000045`.
- **MySQL floor.** `CAST(… AS DOUBLE)` needs MySQL 8.0.17 or later. CI's
`mysql:8.0` image is newer than that. An older server refuses `sum` /
`avg` over a declared numeric column with a syntax error, which is loud,
not a wrong number.
- **Analytics face, read and not measured.** service-analytics'
`NativeSQLStrategy` (`AGGREGATE_SQL` in `native-sql-strategy.ts`) emits
a raw `SUM(col)` / `AVG(col)`. `AnalyticsServicePlugin` auto-bridges it
when the data engine exposes `execute()`. By reading, a cube measure on
PostgreSQL / MySQL still adds exact decimals there. No door was
measured, so there is no `reach:`. Carrier: none.
- **Where the `having` pin lives.** The `having` pin at the engine and
REST doors across dialects is this PR's local measurement (above). In CI
the value pins sit in driver-sql's matrix, and those values decide
`having`. The REST test's PostgreSQL / MySQL cells are still provisioned
by no CI job, as PR #20372 recorded.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01N8TPEsoJxPsdSdNKGnNGEN)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 3062e50 commit fc0db22
3 files changed
Lines changed: 401 additions & 1 deletion
File tree
- .changeset
- packages/drivers/driver-sql/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
Lines changed: 216 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
0 commit comments