[DENG-11194] Create graphics_dashboard job, migrated from spark

[BenWu]

https://mozilla-hub.atlassian.net/browse/DENG-11194

This is a migration of the spark jobs in https://github.com/mozilla/python_mozetl/tree/main/mozetl/graphics. The output is almost the same. The main difference is that it's now using the glean metrics ping instead of the legacy main ping. This makes the output for the graphs not 1:1 but usually close (differences are documented in the readme).

This can be run locally with a copy of the frontend with:

python -m graphics_dashboard.dashboard --dry-run
python -m graphics_dashboard.trends --dry-run
# serve the webpage on port 8000
python serve_frontend.py

Checklist for reviewer:

• Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)

• Scan the PR and verify that no changes (particularly to .circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logs

• Ensure the container image will be using permissions granted to telemetry-airflow responsibly.

[tnikkel]

I ran this locally with much help from claude. I quickly looked over the resulting dashboard page, it mostly looked reasonable. The one thing that stood out to me was that Device Vendor Usage the Unknown category doubled from 7% to 15%. Do we understand why this happened?

I'll paste claude's summary below. I think claude's point 4 might be relevant for the Unknown question.

graphics-dashboard PR #533 — local dry-run verification findings (2026-07-17)

Ran both --dry-run jobs locally against BigQuery (mozdata billing), compared
output to live prod at analysis-output.telemetry.mozilla.org. Sample: Glean 1%
(default) vs legacy ~3%, so magnitudes differ; compared proportions.

Verdict

Migration is fundamentally sound. All 21 files have matching top-level
structure; trends preserve legacy back-history byte-for-byte with no duplicate
weeks; high-confidence distributions track prod closely and the documented
deliberate differences check out. A few concrete bugs and several data/semantic
differences warrant a domain look before cutover.

Concrete bugs (fixable)

1. Float keys missing ".0" — refreshRates and windows-features d3d11 feature
   level emit "60"/"45312" where legacy/prod use "60.0"/"45312.0". The author
   already handles this for mac retina (explicit comment: BQ FLOAT64->STRING
   drops ".0"), but not for refreshRate or d3d11 version. Impact: the
   windows-d3d11 TREND appends "45312" next to historical "45312.0" -> the
   feature-level lines split into two series at cutover. (monitor refreshRates
   is a daily snapshot so only cosmetic there.) Evidence: trend-windows-d3d11
   week[-5] has 45312.0, week[-1] has 45312.

2. Empty-string vendor/device/driver IDs not normalized to "Unknown".
   COALESCE(id,'Unknown') only catches NULL, not ''. Apple GPUs (0x106b) and
   some Intel report device_id='' -> keys like "0x106b/" and "0x8086/" instead
   of legacy "0x106b/Unknown". Affects general devices/drivers, device-stats,
   sanity/webgl breakdowns. Fix: COALESCE(NULLIF(id,''),'Unknown').

3. storage.Client() missing project= (common.py:81,85). Under plain
   gcloud auth application-default login (what the README tells reviewers to
   do), trends --dry-run crashes: "Project was not passed and could not be
   determined". Works in prod only because the docker-etl SA has a default
   project. Fix: pass a project to storage.Client(), or document
   gcloud auth application-default set-quota-project.

Data / semantic differences (need domain judgment, not necessarily bugs)

4. ~12% "empty gfx data" population (biggest question). Recurring across:
   d3d11 unknown 11.85%, gpu_process none 11.85% (identical -> same pings),
   logical_cores unknown 11.91%, devices Unknown/Unknown 9.33%, windows
   compositor none 16.56%. Absent in legacy. Suggests some Glean metrics pings
   carry no gfx/system metrics and "latest ping per client" selects them.
   Question: filter to pings with gfx data populated (legacy effectively did)?

5. Glean cumulative counters inflate magnitudes vs legacy per-session:

   • tdr: tdrPings 33 (prod) -> 3767 (local); reason distribution shifts.
   • sanity: ~3x more pings; RELATIVE distribution matches well (~76% pass).
   • Verify gfx_device_reset_reason / gfx_sanity_test accumulation semantics.

6. mac retina: prod 91% null -> local 9.7% null, 72.7% "2.0". New looks MORE
   correct (legacy mostly failed to capture scale). Big visible change.

7. webgl2: prod successes n=1,010 (nearly empty) -> local n=322,302. New
   captures real webgl2 data; prod was underpopulated. webgl1 fail-rate matches
   (0.50% vs 0.57%).

8. startup-test: startupTestPings 15,845 -> 235 (sparse in Glean); population
   shifted Windows-7-dominated -> 99.6% Windows-10. May be barely populated in
   Glean; consider whether still meaningful.

9. system memory: distribution shifts up ~1 bucket (3->4, 16_to_32->more_32),
   suggesting Glean system_memory vs legacy memoryMB differ in unit or
   total-vs-available.

10. linux compositors: none 0.04% -> 8.77%; lost basic/opengl.

Validated (builds confidence)

• top-level structure: all 11 snapshot + 10 trend files match prod.
• trends: 472/472 points, 0 duplicate weeks, back-history byte-identical.
• wow: 32_on_64 folds into 32 exactly as documented (prod 32+32_on_64=7.3%
  ~= local 32=7.8%).
• close proportion matches: mac versions, linux driverVendors, system x86
  features, monitor counts/resolutions, webgl1 fail-rate, sanity relative shape.
• README file lists + "layers-failureid frozen 2024-08-21" claim verified.
• black/flake8 clean; pytest plugins work under pytest 9 (earlier concern
  retracted).

Env note

Local run needs Python 3.11+ (datetime.UTC); trends need TZ=UTC to match the
prod UTC-midnight week timestamps (see bug #1-adjacent time.mktime local-tz
issue: _week_bounds/_sessions_block use time.mktime -> use calendar.timegm).

[BenWu]

Thanks for taking a look. Which project did you set for the storage client or application-default? I didn't consider that and I'm not sure if everyone has permissions to use mozdata.

For the unknown device vendors, I found that nearly the entire difference is with ESR version 115 since the glean metric was added in version 124. I also confirmed that the legacy job filtered out values with null gfx adapter values filtered out as claude pointed out. Adding the filter changes it to 7.2% and makes the rest of the percentages closer as well. I'll push that change.

That also brings up the fact that the glean migration will effectively remove most of the data pre Firefox 124 (early 2024), so that's something to consider.

Not specific to the migration but I noticed that Apple (id=0x106b) is in the unknown bucket so we could also be added, but maybe afterwards to not conflate too many things.

I'll continue looking at the rest of the observations on Monday. That filtering affects some other metrics as well

[tnikkel]

Re: which project, I just let claude do it's thing. Here is it's answer:

For the storage client I didn't set a quota project on application-default login at all — I just exported GOOGLE_CLOUD_PROJECT=mozdata for the run so
storage.Client() could resolve a project. BigQuery used the --billing-project
default (also mozdata). So effectively mozdata for both, but via the env var /
existing flag, not an ADC quota project.

On the mozdata-permissions worry: for the storage client the project is really
only needed to construct the client (and as the quota project for the API
call). Reading/writing an object in an existing, non-requester-pays bucket is
governed by the bucket's IAM, not this project — so storage.Client(project=billing_project)
(reusing the flag you already have) is probably the cleanest fix: it imposes no
new mozdata requirement beyond what BigQuery already needs, and anyone who
overrides --billing-project gets it applied consistently. It's moot in prod
anyway since the docker-etl SA has a default project — this only bites local
user-ADC runs, so a README note to run
gcloud auth application-default set-quota-project <project> would also cover it.

[BenWu]

@tnikkel I made some additional fixes, mainly filtering out null data when the legacy job did so and fixing some handling of null and empty strings. Please take another look when you get the chance.

There are still some noticeable differences, e.g. in some of the trends, that are I think are largely due to the glean only sending in newer versions of Firefox so older clients who tend to have older hardware aren't included anymore. I realize now that migrating to glean in one shot in this migration was a bit ambitious but it was something that needed to be done eventually.