feat(server): read the Analytics Engine mirror behind a breakdown endpoint - #39
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The AE write path shipped in #38 with nothing reading it. Every accepted event has been mirrored into the columnar store since then, and the dataset has been write-only ever since — so the half that actually buys anything was still missing. This is it:
GET /api/stats/breakdown, the first read that queries Analytics Engine.Nothing existing was re-sourced. Every figure on
/api/statsis exact today and stays exact and D1-only; quietly moving it onto a store that samples would trade that for scale nobody asked for. What the new endpoint adds is the columns D1 has always stored and no endpoint ever surfaced —city,timezone, the three UTM columns,form_factor,currency,hostname— grouped over the range with the ordinarypath/referrer/country/device/channelfilters composed on top, from whichever store the deployment has.AE SQL documents no escape sequence for a string literal, so this does not escape one. That is the substantive finding here and it shapes the whole module. The SQL reference documents literals as
'…'and documents nothing about a quote or a backslash inside one, so there is no escaping rule to implement correctly — only a vendor behaviour to guess at, in a query built by concatenation because AE has no bound parameters.aeLiteraltherefore refuses any value containing a quote, a backslash, or a control character, and the caller falls back to D1, which answers the same query exactly through a bound parameter. A path with an apostrophe in it is served by D1 and nobody notices; the alternative is a query whose meaning depends on something unverified. The dimension→blobNmapping is the other half of that boundary and resolves throughblobColumnagainstBLOB_SCHEMAitself, so the only reachable columns are the twenty the write path fills and the layout keeps exactly one definition.CF_ACCOUNT_IDis validated to 32 lowercase hex before it reaches the URL. It is interpolated into the API path of a request that carriesCF_API_TOKENin anAuthorizationheader, so an id containing a/, an@, or a..segment does not merely 404 — it retargets the request and hands the token to whatever host the typo names. Same class as #37, checked at the same point: before the value can carry a credential.Every aggregate is sampling-corrected, and the one that cannot be is labelled. AE samples under load, so a bare
count()reports surviving rows as though they were the traffic; counts areSUM(_sample_interval)and pageviews areSUM(_sample_interval * double3).visitorsis a distinct count, and no weight recovers identities that were dropped — it is a lower bound whenever sampling kicked in. Rather than bury that, the response carriessourceandsampled, andBreakdownResponsestates plainly thatd1is always exact and a sampledvisitorsis a floor. A caller that needs exact figures now has a way to know it did not get them.Every group clears a k-anonymity floor of three distinct visitors, not three events. The existing breakdowns floor on event count, which is right for the dimensions they cover; this endpoint reaches
cityandutm_campaign, where three pageviews by one person is one person, and that is exactly the shape/api/stats/journeysalready floors on visitors to avoid. Sampling only tightens it, since a suppressed group under-counts in the safe direction.visitor_hashis mirrored and is deliberately absent from the dimension allowlist — it is the one column that identifies a session rather than describing it, and grouping by it would return one row per person.Two places where the stores would otherwise disagree, both resolved by declining rather than by approximating. AE has no NULL and keeps an absent dimension as
''while D1 keeps it as NULL, socountry=matches nothing in D1 and every country-less row in AE; an empty filter value falls back to D1. And a filter value the literal guard rejects declines the whole read rather than dropping that one term, because a dropped term returns unfiltered rows under a filtered label — the failure/api/stats/distributionalready refuses to ship. D1's NULL is folded to''on the way out so both sources label an absent value identically.What the risk is not: no new identifier, no new stored column, no schema change, no migration. The endpoint reads columns that were already being written and already being stored. It is additive — no existing route, query, or response shape changed — and a deployment without
CF_ACCOUNT_ID/CF_API_TOKEN, without the binding, or withRAW_RETENTION_DAYSunder 90 (where the write path already declines, so there is nothing to read) gets D1 for every request and identical answers. The fallback is not a degraded mode; it is the same events, scanned instead of sampled.One thing this cannot verify from here: no local runtime executes AE SQL, so the dialect assertions —
toUInt32(timestamp)inWHERE, alias references inGROUP BY/HAVING/ORDER BY,max(_sample_interval)— are asserted against the emitted query text, not against a live parser. Each is drawn from the current SQL reference. If one is wrong the API rejects the query,queryAelogsae_query_rejectedwith the status only (a rejected analytics query echoes the query text, and a site's paths do not belong in this deployment's logs) and returns null, and the read falls back to D1. A production call returningsource: "analytics_engine"is what closes it.Left out on purpose: the MCP
top_dimensiontool still covers only its original enum, and no dashboard panel consumes the endpoint yet.The new assertions were mutation-checked — disabling the literal guard fails 3 tests, weakening the account-id pattern fails 1, dropping the empty-value decline fails 1, removing the k-anonymity floor fails 10, and replacing
SUM(_sample_interval)withcount()fails 1, each failing exactly what covers it.pnpm lint,pnpm typecheck,pnpm testgreen: 1499 tests across 170 files, up from 1463/169.