powerbi-semantic-diff

A pull request gate for Power BI semantic models that classifies every change by blast radius and blocks the merge: it read a 2000 measure model, built the full dependency graph and classified every change in 586 ms.

What this solves

• A renamed or deleted column breaks reports nobody knows about. The tool builds the full column, measure and report page dependency DAG from the model file and reports the downstream set before the merge. Deleting Sales[Discount] from the sample model is reported as breaking 11 measures and 5 report pages, 10 of those 11 measures only through the NetAmount calculated column, which a text diff of the measure never touches.
• A one line DAX edit silently changes the number. The DAX is parsed into an expression tree and compared as a feature set, not as text. Removing ALL(Customer) is reported as "this measure is now slicer sensitive and its denominator will change with every page filter", while a comment plus reformat on Total Sales is classified SAFE with the reason "the semantic fingerprint is identical". Across the sample pull request the tool separates 13 changes into 6 breaking, 4 risky and 3 safe.
• Semantic models have no unit tests. Golden values for the 14 of 19 sample measures that fall inside the translatable DAX subset are computed on a deterministically seeded DuckDB copy of the model and committed to git. The regression pull request produces 7 golden value failures: 5 measures move beyond tolerance, 1 was deleted and 1 lost its verified value because the column it aggregates is gone. The gate exits 1 and names each one.

Executive summary

A finance director opens the Monday revenue page and the number is 12 percent off. Nobody changed the data. Somebody flipped one relationship to bidirectional cross filtering last Wednesday, and a second fact table started double counting through a shared dimension. The elapsed time between the merge and the question is usually days, and the work that follows is a room of people bisecting a model file by hand. For a team of six analysts, a change like this typically burns a day of investigation across three or four people and puts a hold on every decision that used the affected page. Representative scenario, not a measured incident: at a loaded analyst cost near 90 US dollars an hour, a single day of that is roughly 2,500 US dollars, and it repeats every time somebody edits the model without a check.

This repository is the check. It parses the two text formats Power BI already publishes, TMSL (model.bim) and TMDL, into a typed object graph covering tables, columns with data types, measures, relationships with cardinality and cross filter direction, hierarchies, RLS roles with their filter expressions, and perspectives. Every DAX expression goes through a hand written lexer and recursive descent parser into an expression tree, and the differ compares the extracted feature sets: aggregation targets, iterator targets, filter context modifiers, time intelligence, literals and a canonical form. Each change lands in one of three tiers with a stated reason and a blast radius drawn from the dependency DAG. Separately, a defined subset of DAX is translated into SQL, run against a seeded DuckDB copy of the model tables, and compared to a committed snapshot so the gate also fails on unexplained numeric movement. Python 3.11, DuckDB, sqlglot for dialect checks, jinja2 for the HTML report. No Power BI install, no XMLA endpoint, no cloud credentials.

Measured on this repository, on a 2 vCPU / 7.8 GB container running Python 3.11.15 and DuckDB 1.5.5: parsing a 2000 measure model takes 6.7 ms, building its 2060 node and 5393 edge dependency graph takes 200 ms, and classifying every change between two versions takes 586 ms at p50 and 639 ms at p95. Golden value evaluation of all 2000 measures against 50,000 seeded fact rows takes 3.9 seconds. The test suite is 157 tests at 93 percent statement and branch coverage. Raw numbers are in benchmark/results/results.json.

Architecture

flowchart LR
  subgraph inputs["Inputs, all plain text in git"]
    OLD["old model<br/>model.bim or .tmdl"]
    NEW["new model<br/>model.bim or .tmdl"]
    BIND["report_bindings.json<br/>page to measure map"]
    SNAP["golden/&lt;model&gt;.json<br/>committed baseline"]
  end

  OLD --> P["TMSL / TMDL parser"]
  NEW --> P
  P -->|"ModelParseError:<br/>exit 2, nothing else runs"| FAIL2["bad input"]

  P --> DAXP["DAX lexer and<br/>recursive descent parser"]
  DAXP -->|"DaxSyntaxError:<br/>measure marked unparsed,<br/>change downgraded to RISKY"| DIFF
  DAXP --> FEAT["feature extraction<br/>aggregations, filter context,<br/>literals, canonical form"]

  FEAT --> DIFF["differ<br/>BREAKING / RISKY / SAFE"]
  P --> GRAPH["dependency DAG<br/>columns, measures, roles, pages"]
  BIND --> GRAPH
  GRAPH -->|"cycle found:<br/>BREAKING, refresh would fail"| DIFF
  GRAPH --> DIFF

  P --> SQL["DAX to SQL translator<br/>defined subset only"]
  SQL -->|"UnsupportedDax:<br/>recorded as UNSUPPORTED,<br/>never given a value"| GOLD
  SQL --> SEED["seeded DuckDB copy<br/>deterministic generator"]
  SEED --> GOLD["golden value compare"]
  SNAP --> GOLD

  DIFF --> OUT["HTML report + JSON diff + Mermaid graph"]
  GOLD --> OUT
  OUT --> GATE{"--fail-on breaking"}
  GATE -->|"breaking change or<br/>moved golden value"| E1["exit 1, merge blocked"]
  GATE -->|clean| E0["exit 0"]

Loading

Tech stack

Technology	Role here	Why chosen for this problem
Python 3.11	The whole tool: parsers, DAX front end, differ, CLI	The gate has to run on a stock ubuntu-latest runner after one pip install. A .NET based DAX parser would need a runtime in the CI image or a Windows runner, and a gate that is awkward to install gets skipped.
DuckDB 1.5.5	Executes the translated measure SQL over the seeded model tables	Golden values need a real SQL engine with correct aggregate and NULL semantics, in process, with no server. 50,000 fact rows across five tables seed in 0.26 s and 2000 measures evaluate in 3.9 s, in a CI container with 2 vCPU.
sqlglot 30.14	Parses the generated SQL and transpiles it to Snowflake and T-SQL in tests	The same measure logic often has to be reproduced in a Snowflake or Azure SQL model. Asserting the generated SQL survives a dialect round trip proves the translation is not relying on DuckDB specific syntax. Nothing connects to Snowflake.
pandas / numpy	Deterministic synthetic data generation for the seeded tables	Column values are generated per column from a stable derived seed, so adding a column never reshuffles the others and a committed golden value stays valid. numpy's default_rng gives that reproducibility across machines.
jinja2	Renders the self contained HTML diff report	The report is attached to a CI run and opened offline, so it must be one file with no asset server. jinja2 with autoescape also stops a DAX expression containing angle brackets from breaking the page.
rich	Terminal rendering of the diff, verification table and graph stats	The primary consumer is a CI log and a developer's terminal. rich degrades to plain text automatically when stdout is not a tty, so the same code produces readable CI logs and readable local output.
matplotlib	Benchmark chart and the blast radius diagram in docs/screenshots/	Already a dependency of the analyst toolchain, renders headless with the Agg backend, no browser needed for the chart path.
playwright (chromium)	Screenshots the HTML report and the terminal capture for the README	Rendering the actual artifact is the only honest way to show what the report looks like. Dev dependency only, never needed to run the gate.

Quickstart

Prerequisites: Python 3.11 or newer, git. Nothing else. No Power BI, no Azure, no Snowflake, no Docker unless you want it.

git clone https://github.com/Sandeep0430/powerbi-semantic-diff.git
cd powerbi-semantic-diff
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# 1. What is in the model?
semantic-diff parse models/retail_sales/v1.bim

# 2. The risky pull request. Exits 1 and explains why.
semantic-diff diff models/retail_sales/v1.bim models/retail_sales/v2.bim --fail-on breaking
echo "exit code: $?"

# 3. The safe release. Exits 0.
semantic-diff diff models/retail_sales/v1.bim models/retail_sales/v3.bim --fail-on breaking
echo "exit code: $?"

# 4. Blast radius of deleting one column, plus a Mermaid artifact.
semantic-diff graph models/retail_sales/v1.bim \
  --focus 'column:Sales[Discount]' \
  --impact 'column:Sales[Discount]' \
  --out artifacts/blast-radius.mmd

# 5. Golden values: verify the regression pull request against the committed snapshot.
semantic-diff verify models/retail_sales/v2.bim --baseline golden/retail_sales.json
echo "exit code: $?"

# 6. The HTML report, diff plus golden values in one file.
semantic-diff report models/retail_sales/v1.bim models/retail_sales/v2.bim \
  --baseline golden/retail_sales.json --out artifacts/diff-report.html

# 7. Tests and benchmark.
pytest tests/ --cov=src/semantic_diff --cov-report=term
python benchmark/run_benchmark.py

TMDL works the same way: semantic-diff diff models/hr_headcount/v1.tmdl models/hr_headcount/v2.tmdl.

Docker, if you prefer: docker compose run --rm gate.

GitHub Action usage

Drop this into .github/workflows/semantic-model.yml in the repository that holds your model.bim or TMDL folder:

name: semantic model gate

on:
  pull_request:
    paths:
      - "model/**"

jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install the gate
        run: pip install git+https://github.com/Sandeep0430/powerbi-semantic-diff.git

      - name: Check out the base version of the model
        run: |
          git show "origin/${{ github.base_ref }}:model/model.bim" > /tmp/base.bim

      - name: Diff the semantic model
        env:
          SEMDIFF_RUN_ID: ${{ github.run_id }}
        run: |
          semantic-diff report /tmp/base.bim model/model.bim \
            --bindings model/report_bindings.json \
            --baseline golden/model.json \
            --fail-on breaking \
            --out artifacts/diff-report.html

      - name: Publish the report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: semantic-diff-report
          path: artifacts/diff-report.html

Use --fail-on risky once the team is comfortable, or --fail-on none to run the gate in report only mode while you build confidence.

Screenshots

1. The HTML diff report

The report rendered by jinja2 and screenshotted with playwright chromium at 1440x900. Real output of semantic-diff report models/retail_sales/v1.bim models/retail_sales/v2.bim. Each card carries the tier, the rule that fired, the before and after DAX and the blast radius. The Gross Margin Pct card shows the semantic explanation the DAX tree produces: DIVIDE() was replaced by the / operator, so a zero denominator now errors instead of returning blank.

2. Stage latency against model size

Real output of python benchmark/run_benchmark.py, plotted from benchmark/results/results.json. Both axes are logarithmic. Diff time is the number a pull request waits on; golden value evaluation is the slow stage because it executes one SQL statement per translatable measure.

3. The CI gate failing, and the test suite

Real terminal output, captured to docs/screenshots/ci-gate-terminal.txt and rendered to PNG. Top: semantic-diff diff naming the breaking changes and exiting 1. Middle: semantic-diff verify on the TMDL model, failing on moved golden values and exiting 1. Bottom: pytest --cov, the full suite passing at 93 percent coverage.

Bonus: the dependency graph as an image

The same DAG the gate walks, drawn from semantic-diff graph. Ten of these eleven measures reach Sales[Discount] only through the NetAmount calculated column; Discount Amount is the one that names it directly. The CLI also emits Mermaid (--out artifacts/blast-radius.mmd) for pasting into a pull request.

Performance under load

Methodology: benchmark/run_benchmark.py generates synthetic TMSL models at 50, 500 and 2000 measures with five dimensions and a mutated second version (10 percent of measures edited, 2 percent deleted, one relationship flipped to bidirectional, one column removed), then times each stage 7 times after one untimed warm up and reports the percentiles. Golden values run against 50,000 seeded fact rows. Container: 2 vCPU, 7.8 GB, Python 3.11.15, DuckDB 1.5.5, Linux 6.18.5, no other load controlled for.

Measures	Graph nodes	Graph edges	Parse p50	Graph build p50	Diff p50	Diff p95	Golden eval p50	Snapshot compare p50
50	110	193	0.21 ms	2.94 ms	7.10 ms	7.70 ms	193 ms	0.05 ms
500	560	1,393	0.73 ms	35.04 ms	137.43 ms	158.89 ms	1,229 ms	0.46 ms
2,000	2,060	5,393	6.67 ms	200.21 ms	585.65 ms	639.21 ms	3,943 ms	2.33 ms

Seeding the DuckDB copy is paid once per fact row count, not once per measure: 0.22 s, 0.33 s and 0.26 s at the three scales for the same 50,000 fact rows.

Where it degrades and why: golden value evaluation is linear in the number of measures because each translatable measure is executed as its own SQL statement, and at 2000 measures that is 3.9 seconds of round trips inside DuckDB. Batching independent scalar aggregations into a single SELECT per base table would collapse most of that, and is the first optimisation to reach for if a real model crosses roughly 5000 measures. Diff time grows faster than linearly for the same reason as the graph build: every changed measure is parsed twice and its blast radius is a fresh reverse traversal.

Architecture Decision Records

• ADR 0001: Parse DAX into an expression tree instead of text diffing measures. Why a hand written lexer and recursive descent parser beat a text diff, a regex, or shelling out to a .NET DAX parser, plus an explicit list of what the tree does not handle.
• ADR 0002: Keep golden values in a JSON file in git, not in a database of historical runs. The boring choice: a committed JSON snapshot is reviewable, credential free and revertible, at the cost of having no history.

Severity tiers, and why each rule sits where it does

BREAKING: something stops working, or a security boundary moves. The gate fails.

Rule	Why breaking
Measure or table removed	Every visual, dependent measure and paginated report binding it renders an error, and Power BI does not warn the author who deleted it.
Column removed while something depends on it	The dependent measure, relationship or visual fails to resolve. The dependency DAG decides "depends on it", so a calculated column in the chain still counts.
Relationship removed	Filters stop propagating along that path. This one is worse than an error: the visual keeps rendering and quietly shows the unfiltered total.
Cardinality changed to many to many	The engine switches to a limited relationship. The blank row disappears, unmatched fact rows stop being counted, and totals no longer equal the sum of their parts.
Relationship deactivated	Every measure that did not explicitly wrap USERELATIONSHIP now aggregates without that filter path.
RLS role removed, or a table permission removed from a role	Members either lose access entirely, or see every row the deleted filter used to hide.
RLS filter expression changed	Row visibility moved. The tool cannot prove the new predicate is narrower than the old one, and guessing wrong is a data disclosure, so this is always breaking even when the edit is a tightening.
ALL, ALLEXCEPT or REMOVEFILTERS removed from a measure	The measure becomes slicer sensitive. Denominators in share and percentage of total measures change with every page filter.
DIVIDE() replaced by the / operator	A zero or blank denominator returns an error instead of blank, which surfaces as a broken visual rather than an empty one.
USERELATIONSHIP removed	The measure silently switches back to the default relationship path.
Hierarchy removed	Visuals that drill through it lose their field binding.
Circular dependency introduced	Refresh fails outright.
Security filtering behaviour changed on a relationship	Directly changes which rows a secured user can see.

RISKY: the numbers can move without anybody asking. Needs a human to confirm intent.

Rule	Why risky rather than breaking
Cross filter direction changed to both	The classic silent failure: a row level security filter on one table leaks into a table that was never meant to be secured, and two fact tables sharing a dimension start double counting. Nothing errors, so it cannot be breaking by the definition above, but it is the change most worth a second reviewer.
Data type narrowed (for example double to int64)	Values that do not fit truncate or error at refresh. Promoted to BREAKING when the column is a relationship key, because a type mismatch breaks the join itself.
Aggregation or iterator changed	SUM becoming SUMX over a different table changes the grain the measure evaluates at.
A literal inside an expression changed	A threshold or filter constant moved, so the returned number moves with it. This rule exists because every other feature is identical for > 5 becoming > 10, and without it that edit would be reported as cosmetic.
ALL or ALLEXCEPT added	The measure now ignores filters on that scope.
CALCULATE context transition added or removed	Row context stops being converted to filter context, which changes iterator results.
Time intelligence function swapped	The comparison period the measure returns has moved.
Relationship added	A new propagation path can change existing visuals and can create an ambiguous chain the engine resolves in a way nobody chose.
Hierarchy levels changed	Drill paths change and saved bookmarks can land on a missing level.
Perspective removed	No value changes, but an app or Excel connection pinned to that perspective falls back to the full field list.
Measure moved to a different home table	Perspectives, display folders and fully qualified references change.
Table, column or measure hidden	Existing visuals keep working; report authors lose the ability to rebuild them.
A measure expression that will not parse	Reported as unreviewed rather than assumed safe.

SAFE: nothing a consumer can observe as a number. Descriptions, format strings, display folders, new columns, new tables, new measures nothing depends on yet, new roles with no members, perspective membership, and expression edits whose canonical form is unchanged (reformatting, comments, variable renames, reordered operands of a commutative operator).

Supported DAX subset for golden values

This is the part to read before trusting a number, and the part where being wrong is dangerous. Golden values are evaluated at the grand total with no external filter context. Inside that envelope, these are translated to SQL exactly:

Category	Functions and forms
Scalar aggregations	SUM, AVERAGE, MIN, MAX, COUNT, COUNTA, COUNTROWS, DISTINCTCOUNT
Iterators	SUMX, AVERAGEX, MINX, MAXX, COUNTX over a base table with a row level body
Arithmetic	+ - * / ^, unary minus, parentheses, numeric and string literals
Safe division	DIVIDE(numerator, denominator [, alternate])
Scalar helpers	ABS, ROUND, INT, SQRT, BLANK()
Filters	CALCULATE(<expr>, Table[Column] <op> <literal>) where the filtered table is the aggregated table or one active many to one hop away
Composition	References to other measures, inlined recursively with a cycle guard

Everything else is reported as UNSUPPORTED with the reason and is never given a value: ALL, ALLEXCEPT, ALLSELECTED, REMOVEFILTERS, KEEPFILTERS, USERELATIONSHIP, CROSSFILTER, FILTER, RELATED, RELATEDTABLE, VALUES, VAR/RETURN, IF, SWITCH, RANKX, TOPN, and every time intelligence function (TOTALYTD, SAMEPERIODLASTYEAR, DATEADD, PARALLELPERIOD and the rest).

ALL deserves a specific note. At the grand total CALCULATE(SUM(x), ALL(t)) returns the same number as SUM(x), so translating it would produce a golden value that looks correct and that would keep looking correct after somebody deleted the ALL. That is exactly the regression this repository exists to catch, so ALL stays unsupported and the structural differ, not the snapshot, is what flags it.

On the shipped models the coverage is 14 of 19 measures for Retail Sales, 9 of 10 for Finance GL and 5 of 6 for HR Headcount. tests/test_sqlgen.py asserts that TOTALYTD, SAMEPERIODLASTYEAR, USERELATIONSHIP and ALL are reported as unsupported rather than silently given a value, because silently passing is the failure mode that would make this whole component worse than useless.

Two further honesty notes. SUM, COUNT, COUNTROWS, DISTINCTCOUNT, SUMX and COUNTX are wrapped in COALESCE(..., 0) to match DAX treating BLANK as zero in arithmetic; AVERAGE, MIN and MAX are not, so an empty set stays NULL as it does in DAX. And the underlying data is synthetic, generated from the model definition, so a golden value proves the expression did not change meaning, not that production numbers are correct.

Intentionally out of scope

• Calculation groups and calculation items. They rewrite measures at query time through SELECTEDMEASURE(), which the expression tree cannot resolve without an engine. Add support when a model in scope actually ships one; the trigger is the first calculationGroup node appearing in a parsed file, which currently parses as an ordinary table with no measures.
• Power Query / M expressions in partitions. A partition's source query can change what rows land in the table, which changes every number, and the tool does not read M at all. The honest mitigation today is that the golden values run on generated data, so they would not catch it either. Add an M diff when a team starts editing partition queries in the same pull request as the model.
• Incremental refresh policies and aggregation tables. Parsed as ordinary partitions. Worth adding when a model uses aggregations, because a change to an aggregation table silently changes query performance rather than results.
• Reading .pbix directly. The report page bindings ship as a small JSON file extracted from the report layout. Unzipping a .pbix and parsing Report/Layout is straightforward and is the right next step once the gate is running on a real repository; the trigger is the first time somebody forgets to update report_bindings.json.
• Running real DAX. Would need an Analysis Services instance. The whole design assumes that is unavailable in CI, which is why the SQL subset exists and why its limits are documented above rather than hidden.
• Auto fixing. The tool reports and blocks. It never edits a model file.

Security and compliance

• No secrets are needed or read. The tool runs on local files and an in process DuckDB database. There is no credential to leak, and .env.example contains only paths, tolerances and log settings.
• Nothing sensitive is logged. Log lines carry object names (measure, column, table, role) and counts. Measure values appear only in the golden snapshot and the report you asked it to write. RLS filter expressions are echoed in the diff, because reviewing them is the point, so treat the HTML report as an internal artifact.
• Least privilege at runtime. The container runs as a non root user (uid 10001) that can read the model directory and write only to /app/artifacts. The gate needs no database, no network and no cloud role.
• Data never leaves the machine. Golden values are computed on synthetic data generated from the model definition, not on a production extract, so the committed snapshot contains no customer data.
• Supply chain. Runtime dependencies are duckdb, pandas, numpy, sqlglot, jinja2, rich, pyyaml and matplotlib. The DAX front end has no dependencies at all, which keeps the part that reads untrusted expression text small and auditable.
• Untrusted input. Model files are treated as untrusted text: the parsers raise typed errors instead of evaluating anything, and jinja2 autoescaping is on so a DAX expression containing HTML cannot inject into the report.

Failure modes

Failure	Detection	Behaviour	Recovery
Model file is not valid JSON or TMDL	ModelParseError raised by the parser at load time	CLI prints model error: <file> is not valid JSON: <position> and exits 2, distinct from the exit 1 the gate uses for a real breaking change	Fix the file; a truncated model.bim is usually a bad export or a merge conflict marker
A measure expression will not parse	DaxSyntaxError captured by the analyser, which returns parsed=False instead of raising	The change is reported as RISKY with "falling back to text comparison, treat this change as unreviewed", so it can never be silently classified SAFE	Fix the DAX, or open an issue with the expression if the parser is at fault
A measure leaves the translatable SQL subset	UnsupportedDax raised by the translator and recorded on the snapshot entry	Measure is recorded as UNSUPPORTED with the offending function name and never given a value. If it previously had one, the comparison status is COVERAGE_LOST and the gate fails	Either accept the loss and re-snapshot, or rewrite the measure inside the subset
A measure evaluates to Infinity or NaN (divide by zero)	math.isfinite check in compute_snapshot	Recorded as blank with an explicit reason, which the comparator treats as a value movement, so the regression is still caught and the snapshot file stays valid JSON	Fix the denominator, usually by restoring DIVIDE()
Committed snapshot was written by an older format version	Version field checked in read_snapshot	Raises with "regenerate it with 'semantic-diff snapshot'", CLI exits 2 rather than comparing incompatible files	Re-run semantic-diff snapshot and commit the result
Circular measure dependency introduced	Iterative Tarjan pass over the dependency DAG, and a separate cycle guard inside the SQL translator	Reported as a BREAKING change naming the chain; the translator refuses the measure instead of recursing until the stack overflows	Break the cycle
Model grows past the point where the graph build is slow	Benchmark at three scales, committed	No automatic behaviour: the tool degrades in latency, not correctness. Diff at 2000 measures is 586 ms p50	Batch the golden value SQL per base table, see the performance section
Report bindings file missing	load_report_bindings returns an empty page list	The diff still runs; blast radius reports measures but no pages, and column removals with no other dependent drop from BREAKING to RISKY with the reason stated in the change detail	Extract report_bindings.json from the report layout

Hardest problem solved

The bug that cost the most was not in the DAX parser. It was one line in the golden value writer, and it only appeared because of a test written to be adversarial rather than convenient.

The regression sample model (models/retail_sales/v2.bim) does two things at once that a real pull request often does: it deletes Sales[Discount], and it replaces DIVIDE([Gross Margin], [Total Sales], 0) with ([Gross Margin]) / ([Total Sales]). Deleting the column breaks the NetAmount calculated column, so Total Sales collapses to zero, so the new / operator divides a large negative number by zero. DuckDB returns -inf, and json.dumps cheerfully wrote "value": -Infinity into the snapshot file. Every local check passed, because Python's own json.loads accepts Infinity, -Infinity and NaN as an extension. The file is not valid JSON. node, jq, Go and every strict parser reject it. The failure would have surfaced in CI as an unparseable artifact, which reads as "the tool is broken" and gets the gate disabled, rather than as "your pull request broke revenue", which is what actually happened.

I only found it because I wrote the round trip test with json.loads(..., parse_constant=...) raising on any non standard constant instead of a plain json.loads. The plain version passes. The fix, in 342c4ed, records a non finite result as blank with an explicit reason ("evaluated to the non finite value -inf ... the usual cause is a / operator dividing by zero"), which the comparator already treats as a value movement, so the regression is still caught and the file stays parseable by anything. The lesson I took from it is that a serialisation format's permissive reader is not a validator: if an artifact is going to be read by something other than the program that wrote it, the test has to read it the way that other thing will.

A second, smaller one is worth recording because it came from the same habit. A test that built a model containing M1 = [M1] + 1 found that find_cycles reported no cycle: the graph builder was dropping any edge whose endpoints were equal, so the self loop never reached the Tarjan pass. Fixed in ca85966.

Future work

• Batch the golden value SQL. One SELECT per translatable measure is what makes the 2000 measure case take 3.9 seconds. Independent scalar aggregations over the same base table can be collapsed into a single statement, which should cut that by an order of magnitude. This is the first thing to do if a real model crosses roughly 5000 measures.
• Read .pbix report layouts directly instead of a hand maintained report_bindings.json, so the page level blast radius cannot go stale.
• Widen the SQL subset carefully, and only where it stays honest. FILTER with a simple predicate and RELATED across one hop are both translatable. ALL is not, and should stay out for the reason stated above.
• Post the report as a pull request comment rather than an uploaded artifact, with the Mermaid blast radius inline, so reviewers see the impact without downloading anything.
• The first metric to watch after deploying is the ratio of BREAKING findings that get overridden with --fail-on none or by re-snapshotting without a code change. If that climbs above roughly one in five, a tier is miscalibrated and the gate is training people to ignore it, which is worse than having no gate. The second metric is the count of measures reported UNSUPPORTED: if it grows as the model grows, golden value coverage is quietly eroding and the structural diff is carrying the whole load.