Snapshot testing for LLM outputs — golden files with exact, structural, and fuzzy matching. Zero runtime dependencies.
A model upgrade changes outputs silently; vardi turns that into a failing test. Record an approved ("golden") output once, and every later run compares fresh output against it — including the run right after you bump the model version.
vardi never calls a model API itself. Your code produces the output; vardi stores and compares it. That keeps it provider-neutral and dependency-free (standard library only).
A varði is a stone cairn marking a route — you notice when the path stops matching the markers.
pip install git+https://github.com/HenrikVarmer/vardiAs a pytest fixture (the plugin registers itself on install):
def test_ticket_summary(vardi):
output = my_llm_pipeline("summarize", ticket_text) # your code, any provider
vardi.match(output, "ticket-summary")Record the first golden (and re-approve after intentional changes):
pytest --vardi-updateGolden files land in vardi/ at the repo root as readable JSON — review them
in the PR like any other fixture. On later runs the test fails with a diff if
the output drifts.
Three comparison modes:
# exact: byte-for-byte (default) — for deterministic pipelines
vardi.match(output, "sql-query")
# structure: JSON keys and types must match, values may differ —
# for "the model must return this schema" contracts
vardi.match(output, "extraction", mode="structure")
# fuzzy: token-F1 against the golden must clear a threshold —
# for prose that may be reworded but not rewritten
vardi.match(output, "summary", mode="fuzzy", threshold=0.8)Outside pytest, the same operations are plain functions:
import vardi
vardi.record(output, "summary", mode="fuzzy")
result = vardi.check(new_output, "summary")
print(result.passed, result.score, result.detail)| Function | Description |
|---|---|
vardi.record(output, name, mode="exact", directory="vardi") |
Store output (str or JSON value) as the golden for name; returns the file path. |
vardi.check(output, name, mode=None, threshold=0.8, directory="vardi") |
Compare against the golden; mode=None uses the recorded mode. Returns CheckResult. |
vardi.golden_path(name, directory="vardi") |
Path of the golden file for name. |
CheckResult |
name, mode, passed, score (token F1 in fuzzy mode, else 0/1), detail (diff or explanation). Truthy when passed. |
fixture vardi.match(output, name, mode=None, threshold=0.8) |
pytest assertion; records instead when --vardi-update is set. |
--vardi-update |
pytest flag: write fresh outputs as the new goldens instead of comparing. |
MissingGoldenError |
Raised by check when no golden exists for name. |
vardi is snapshot testing, not evaluation. It tells you that output changed,
not whether it got better or worse — after an intentional model upgrade you
review the diffs and re-approve with --vardi-update, exactly as with
image-snapshot tools.
Know what each mode can and cannot catch:
- exact is brittle by design. Any sampling nondeterminism fails it, so use it only where you expect determinism (temperature 0, constrained output).
- structure validates shape, never content: an output with the right keys and types passes even if every value is wrong. It answers "did the contract hold", nothing more.
- fuzzy compares token multisets (lowercase alphanumeric tokens, F1 score). Rewording with the same vocabulary scores high — but so can a sentence that reuses the words while inverting the meaning ("approve" vs "do not approve" share most tokens). A low score reliably means drift; a high score does not prove equivalence. There are no embeddings and no LLM judge here; if you need semantic comparison, use a semantic tool and accept its dependencies.
Timestamps in golden files record when a golden was approved; they are metadata and never compared.
MIT