Add experiments_column_type "json" for top-level JSON experiments maps#542
Open
danielkberry wants to merge 1 commit into
Open
Add experiments_column_type "json" for top-level JSON experiments maps#542danielkberry wants to merge 1 commit into
danielkberry wants to merge 1 commit into
Conversation
The "events_stream" experiments_column_type reads the experiment/branch from each event's own event_extra JSON. Those keys are only populated on nimbus event rows (enrollment/exposure/etc.), so the day-0 "ignore pre-enrollment data" guard drops *all* enrollment-day metric events for events_stream data sources (generic events like contile_impression carry no $.experiment in event_extra). For first-run experiments — where most activity happens on the enrollment day — this badly undercounts exposure/engagement metrics. However, *.events_stream tables also carry the standard client-level `experiments` map (a top-level BigQuery JSON column) on every row. This adds a new "json" type that gates day-0 rows on that map via JSON_VALUE(ds.experiments, '$."<slug>".branch'), mirroring the "glean"/"native" semantics but for a JSON (rather than repeated-struct) column. Data sources on events_stream can switch to "json" so enrollment-day metric events are attributed correctly. - experiments_column_expr and get_sanity_metrics gain a "json" branch - registered in EXPERIMENT_COLUMN_TYPES and documented on DataSource - tests for the day-0 guard and sanity metrics; extended the constructor test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
danielkberry
force-pushed
the
add-json-experiments-column-type
branch
from
July 10, 2026 17:07
9ab1dc2 to
f31c7d3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
experiments_column_type = "events_stream"day-0 guard attributes events using each event's ownevent_extraJSON ($.experiment/$.branch). Those keys are only present on nimbus event rows (enrollment/exposure/etc.) — generic metric events (e.g.contile_impression) carry no$.experiment. So theexperiments_column_expr"ignore pre-enrollment day-0 data" guard drops all enrollment-day metric events forevents_streamdata sources.For first-run / new-user experiments, where most activity happens on the enrollment day, this badly undercounts exposure and engagement. Observed example: a Fenix sponsored-tiles experiment's control exposure read 16.2% via the events_stream outcome vs 56.4% ground truth (~3.5×), entirely because day-0 impressions were dropped — even though those rows are correctly branch-annotated in the client-level
experimentsmap (event_extrajust doesn't carry the experiment/branch on non-nimbus events).Fix
*.events_streamtables also carry the standard client-levelexperimentsmap as a top-level BigQueryJSONcolumn on every row. This adds a newexperiments_column_type = "json"that gates day-0 rows on that map:mirroring the
"glean"/"native"semantics but for a JSON (rather than repeated-struct) column. No existing type fits:"glean"usesmozfun.map.get_key(ds.ping_info.experiments, ...), which expects a repeatedSTRUCTunderping_info, whereas events_stream'sexperimentsis a top-levelJSONcolumn.Changes:
experiments_column_exprandget_sanity_metricsgain a"json"branchEXPERIMENT_COLUMN_TYPES, documented onDataSourceAll 199 tests pass; ruff lint + format clean.
Follow-up
A companion metric-hub change will switch the events_stream data sources (fenix
sponsored_tiles_events_stream,events_stream_awesomebar; desktopprotections_popup_events,site_breakage_events) from"events_stream"to"json", and affected experiments will need re-analysis.