Skip to content

feat(server): read the Analytics Engine mirror behind a breakdown endpoint - #39

Merged
dcondrey merged 1 commit into
mainfrom
feat/ae-read-breakdown
Aug 4, 2026
Merged

feat(server): read the Analytics Engine mirror behind a breakdown endpoint#39
dcondrey merged 1 commit into
mainfrom
feat/ae-read-breakdown

Conversation

@dcondrey

@dcondrey dcondrey commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

The AE write path shipped in #38 with nothing reading it. Every accepted event has been mirrored into the columnar store since then, and the dataset has been write-only ever since — so the half that actually buys anything was still missing. This is it: GET /api/stats/breakdown, the first read that queries Analytics Engine.

Nothing existing was re-sourced. Every figure on /api/stats is exact today and stays exact and D1-only; quietly moving it onto a store that samples would trade that for scale nobody asked for. What the new endpoint adds is the columns D1 has always stored and no endpoint ever surfaced — city, timezone, the three UTM columns, form_factor, currency, hostname — grouped over the range with the ordinary path/referrer/country/device/channel filters composed on top, from whichever store the deployment has.

AE SQL documents no escape sequence for a string literal, so this does not escape one. That is the substantive finding here and it shapes the whole module. The SQL reference documents literals as '…' and documents nothing about a quote or a backslash inside one, so there is no escaping rule to implement correctly — only a vendor behaviour to guess at, in a query built by concatenation because AE has no bound parameters. aeLiteral therefore refuses any value containing a quote, a backslash, or a control character, and the caller falls back to D1, which answers the same query exactly through a bound parameter. A path with an apostrophe in it is served by D1 and nobody notices; the alternative is a query whose meaning depends on something unverified. The dimension→blobN mapping is the other half of that boundary and resolves through blobColumn against BLOB_SCHEMA itself, so the only reachable columns are the twenty the write path fills and the layout keeps exactly one definition.

CF_ACCOUNT_ID is validated to 32 lowercase hex before it reaches the URL. It is interpolated into the API path of a request that carries CF_API_TOKEN in an Authorization header, so an id containing a /, an @, or a .. segment does not merely 404 — it retargets the request and hands the token to whatever host the typo names. Same class as #37, checked at the same point: before the value can carry a credential.

Every aggregate is sampling-corrected, and the one that cannot be is labelled. AE samples under load, so a bare count() reports surviving rows as though they were the traffic; counts are SUM(_sample_interval) and pageviews are SUM(_sample_interval * double3). visitors is a distinct count, and no weight recovers identities that were dropped — it is a lower bound whenever sampling kicked in. Rather than bury that, the response carries source and sampled, and BreakdownResponse states plainly that d1 is always exact and a sampled visitors is a floor. A caller that needs exact figures now has a way to know it did not get them.

Every group clears a k-anonymity floor of three distinct visitors, not three events. The existing breakdowns floor on event count, which is right for the dimensions they cover; this endpoint reaches city and utm_campaign, where three pageviews by one person is one person, and that is exactly the shape /api/stats/journeys already floors on visitors to avoid. Sampling only tightens it, since a suppressed group under-counts in the safe direction. visitor_hash is mirrored and is deliberately absent from the dimension allowlist — it is the one column that identifies a session rather than describing it, and grouping by it would return one row per person.

Two places where the stores would otherwise disagree, both resolved by declining rather than by approximating. AE has no NULL and keeps an absent dimension as '' while D1 keeps it as NULL, so country= matches nothing in D1 and every country-less row in AE; an empty filter value falls back to D1. And a filter value the literal guard rejects declines the whole read rather than dropping that one term, because a dropped term returns unfiltered rows under a filtered label — the failure /api/stats/distribution already refuses to ship. D1's NULL is folded to '' on the way out so both sources label an absent value identically.

What the risk is not: no new identifier, no new stored column, no schema change, no migration. The endpoint reads columns that were already being written and already being stored. It is additive — no existing route, query, or response shape changed — and a deployment without CF_ACCOUNT_ID/CF_API_TOKEN, without the binding, or with RAW_RETENTION_DAYS under 90 (where the write path already declines, so there is nothing to read) gets D1 for every request and identical answers. The fallback is not a degraded mode; it is the same events, scanned instead of sampled.

One thing this cannot verify from here: no local runtime executes AE SQL, so the dialect assertions — toUInt32(timestamp) in WHERE, alias references in GROUP BY/HAVING/ORDER BY, max(_sample_interval) — are asserted against the emitted query text, not against a live parser. Each is drawn from the current SQL reference. If one is wrong the API rejects the query, queryAe logs ae_query_rejected with the status only (a rejected analytics query echoes the query text, and a site's paths do not belong in this deployment's logs) and returns null, and the read falls back to D1. A production call returning source: "analytics_engine" is what closes it.

Left out on purpose: the MCP top_dimension tool still covers only its original enum, and no dashboard panel consumes the endpoint yet.

The new assertions were mutation-checked — disabling the literal guard fails 3 tests, weakening the account-id pattern fails 1, dropping the empty-value decline fails 1, removing the k-anonymity floor fails 10, and replacing SUM(_sample_interval) with count() fails 1, each failing exactly what covers it.

pnpm lint, pnpm typecheck, pnpm test green: 1499 tests across 170 files, up from 1463/169.

@dcondrey
dcondrey merged commit 279e650 into main Aug 4, 2026
6 checks passed
@dcondrey
dcondrey deleted the feat/ae-read-breakdown branch August 4, 2026 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant