Golden fixtures in shared/test-data no longer match the JS reference (705/705)

[zl190]

Summary

The golden fixtures under shared/test-data/ no longer match what flint-js produces today. I hit this while working on a Waterfall fix (#79) and went looking for how far it reached, since it decides whether a JS-side change can be mirrored into flint-py.

Measured on dev @ c73d597, over the 705 fixture directories that have both input.json and expected.json:

Comparison	Identical	Differing
current flint-js == stored expected.json	0	705
current flint-py == current flint-js	0	705
current flint-py == stored expected.json	525	180

The third row is what pytest reports today (180 failures out of the 705 fixture tests), so it doubles as a calibration check on my comparator — it reproduces your number exactly.

Where the difference sits

Counted as "how many of the 705 fixtures differ at this path":

Fixtures	Path
705	config.legend
705	_transform
591	config.axisX/axisY.titleFontSize
524	_pivot
208 / 188	_options[].step / _options[].label
168 / 163	config.axisX/axisY.labelFontSize
148 / 133	config.view.continuousHeight / continuousWidth
133	config.facet.spacing

Every one is in the shared assembler / config layer. None is in a chart template. The per-chart Python ports look fine; what has moved is the layer underneath them.

What seems to have happened

packages/flint-py/tests/FULL_GALLERY_REPORT.md records 658/658 byte-for-byte parity with 0 mismatches, and its Methodology section names the tool that produced the fixtures:

The extractor at tests/frontend/unit/lib/flint-chart/flint_py_extract.test.ts walks GALLERY_TREE … Each case is written to flint-py/tests/fixtures/<slug>/ as input.json + expected.json

Two things about that, as the repo stands now:

1. The extractor is not in this repository. git log --all -- "*flint_py_extract*" returns nothing, and the path it is described at does not correspond to the current tree. It looks like it stayed behind when the project was restructured into a monorepo (53d50be, 2026-06-11).
2. tools/run_full_eval.py still points at the pre-rename location. packages/flint-py/tools/run_full_eval.py:23 has FIXTURES = ROOT / "tests" / "fixtures", but the corpus moved to shared/test-data/ in 1d1ee39 ("rename agents→agent-skills, test-fixtures→test-data") on the same day. That directory no longer exists, so the eval tool reports FIXTURE_MISSING.

Timeline from git log: the last commit that regenerated fixtures is f32e75f (2026-06-18, "port 7 Vega-Lite templates to JS parity + regen fixtures"); the corpus was last touched at all in dd09c9b (2026-06-26). Since then packages/flint-js/src has had 41 commits.

Why it stayed quiet

The half that broke is the generating side, not the checking side. pytest kept running and kept 525 fixtures green — against a corpus frozen in June. A stale golden file is the easiest kind to match, so the green signal held while the reference drifted away underneath it. Nothing in the suite fails when the corpus goes out of date, only when Python diverges from it.

One idea, if it is useful

meta.json already carries provenance (slug, generator, library, status) but not the JS revision the golden was recorded from. Adding something like sourceCommit and asserting that all fixtures share one value, and that it is not more than N commits behind HEAD, would make "the corpus is stale" a condition the suite can fail on — rather than something that has to be noticed.

Reproducing

1. In packages/flint-js, run a throwaway vitest that walks every shared/test-data/*/input.json and calls assembleVegaLite(fx.input), dumping the results to one JSON file. (705 inputs, 0 throw.)
2. In packages/flint-py, load each fixture's expected.json, compute assemble_vegalite(input), and compare all three with the canonicalisation from tests/test_fixtures.py (_canon: sorted keys, floats rounded to 9dp).
3. The calibration to check the comparator: Python == stored should come out at 705 − 180 = 525.

Happy to send the two scripts, or to open a PR that adds them under packages/flint-py/tools/ if that is somewhere you would want them.

I also have a Waterfall parity patch for flint-py ready (the totals inference, the connector layer, the value labels, and the title fix — its template output is byte-identical to current flint-js), but it is not much use until the corpus can be regenerated, since on its own it turns the 8 currently-green Waterfall fixtures red. Glad to send it whenever it is useful, or to leave it.

[zl190]

Three notes, meant to make this cheaper to act on.

1. The eval tool runs again

#81 repoints tools/run_full_eval.py at shared/test-data/ (three lines). With it applied the tool completes and reports:

PASS: 476   MISMATCH: 180   PY_ERROR: 0   JS_ERROR: 1   FIXTURE_MISSING: 0

MISMATCH: 180 is the same number pytest -q reports, so the two instruments agree.

On the two counts in this thread: PASS: 476 is over the 657 entries in manifest.json, while the 525 identical figure in the issue above is over all 705 fixture directories. The 48 directories outside the manifest plus the one Sunburst JS_ERROR account for the difference (476 + 1 + 48 = 525).

2. The 180 mismatches group into about six causes

Grouped by where Python and the stored golden differ. A fixture usually differs at more than one path, so the rows sum to more than 180:

Fixtures	Path
75	encoding.color.scale.domain
55	_options[].applicable
50	_warnings[].message
32 / 12	encoding.x.scale.domain[] / encoding.y.scale.domain[]
27	data.values[].*
12	layer[]

By chart type, they concentrate hard:

Fixtures	Chart type
55	Area Chart
51	Stacked Bar Chart
21	Bar Chart
12	Grouped Bar Chart
10	Lollipop Chart

Area + Stacked Bar + Bar + Grouped Bar is 139 of the 180.

3. One ordering trap worth flagging

These 180 are Python measured against the stored goldens, and those goldens predate current flint-js. So closing them would align Python to a June snapshot rather than to the compiler as it stands — real work spent chasing a target that has already moved.

Regenerating everything first has the opposite problem: since Python currently matches none of current JS, a full regeneration would take the suite from 476 green to 0.

A middle path that avoids both, if that fits how you would rather sequence it: go one assembler feature at a time, and regenerate only the goldens that feature touches.

• port feature X into flint-py
• regenerate just the goldens whose output X changes
• that batch goes green; everything else is untouched

Each step is checkable by byte-equality against the JS reference, and the suite never goes broadly red.

Offers

I have two things ready if they are useful, and neither is a PR yet because both depend on how you want to sequence the above:

• The three scripts behind these numbers (JS-side sweep, three-way comparator, the mismatch grouping). Happy to send them as a PR under packages/flint-py/tools/, or just paste them here.
• A flint-py Waterfall parity patch: the totals inference, the connector layer, the value labels, and the title fix from fix(vegalite): Waterfall axis titles: display names, no __wf_lead #79. Its template output is byte-identical to current flint-js, but on its own it turns the 8 currently-green Waterfall goldens red, so it wants the regeneration question settled first.

Also glad to leave all of it alone if flint-py parity is not where you want attention right now — the survey stands on its own as a record either way.