Skip to content

Add experiments_column_type "json" for top-level JSON experiments maps#542

Open
danielkberry wants to merge 1 commit into
mainfrom
add-json-experiments-column-type
Open

Add experiments_column_type "json" for top-level JSON experiments maps#542
danielkberry wants to merge 1 commit into
mainfrom
add-json-experiments-column-type

Conversation

@danielkberry

Copy link
Copy Markdown
Contributor

Problem

The experiments_column_type = "events_stream" day-0 guard attributes events using each event's own event_extra JSON ($.experiment / $.branch). Those keys are only present on nimbus event rows (enrollment/exposure/etc.) — generic metric events (e.g. contile_impression) carry no $.experiment. So the experiments_column_expr "ignore pre-enrollment day-0 data" guard drops all enrollment-day metric events for events_stream data sources.

For first-run / new-user experiments, where most activity happens on the enrollment day, this badly undercounts exposure and engagement. Observed example: a Fenix sponsored-tiles experiment's control exposure read 16.2% via the events_stream outcome vs 56.4% ground truth (~3.5×), entirely because day-0 impressions were dropped — even though those rows are correctly branch-annotated in the client-level experiments map (event_extra just doesn't carry the experiment/branch on non-nimbus events).

Fix

*.events_stream tables also carry the standard client-level experiments map as a top-level BigQuery JSON column on every row. This adds a new experiments_column_type = "json" that gates day-0 rows on that map:

OR JSON_VALUE(ds.experiments, '$."<slug>".branch') IS NOT NULL

mirroring the "glean" / "native" semantics but for a JSON (rather than repeated-struct) column. No existing type fits: "glean" uses mozfun.map.get_key(ds.ping_info.experiments, ...), which expects a repeated STRUCT under ping_info, whereas events_stream's experiments is a top-level JSON column.

Changes:

  • experiments_column_expr and get_sanity_metrics gain a "json" branch
  • registered in EXPERIMENT_COLUMN_TYPES, documented on DataSource
  • tests for the day-0 guard + sanity metrics; extended the constructor parametrization

All 199 tests pass; ruff lint + format clean.

Follow-up

A companion metric-hub change will switch the events_stream data sources (fenix sponsored_tiles_events_stream, events_stream_awesomebar; desktop protections_popup_events, site_breakage_events) from "events_stream" to "json", and affected experiments will need re-analysis.

The "events_stream" experiments_column_type reads the experiment/branch from
each event's own event_extra JSON. Those keys are only populated on nimbus
event rows (enrollment/exposure/etc.), so the day-0 "ignore pre-enrollment
data" guard drops *all* enrollment-day metric events for events_stream data
sources (generic events like contile_impression carry no $.experiment in
event_extra). For first-run experiments — where most activity happens on the
enrollment day — this badly undercounts exposure/engagement metrics.

However, *.events_stream tables also carry the standard client-level
`experiments` map (a top-level BigQuery JSON column) on every row. This adds a
new "json" type that gates day-0 rows on that map via
JSON_VALUE(ds.experiments, '$."<slug>".branch'), mirroring the "glean"/"native"
semantics but for a JSON (rather than repeated-struct) column. Data sources on
events_stream can switch to "json" so enrollment-day metric events are
attributed correctly.

- experiments_column_expr and get_sanity_metrics gain a "json" branch
- registered in EXPERIMENT_COLUMN_TYPES and documented on DataSource
- tests for the day-0 guard and sanity metrics; extended the constructor test

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant