Skip to content

feat: an export verb that flattens the parse into cited tables - #63

Merged
ChelseaKR merged 1 commit into
mainfrom
feat/export-cited-tables
Sep 6, 2026
Merged

feat: an export verb that flattens the parse into cited tables#63
ChelseaKR merged 1 commit into
mainfrom
feat/export-cited-tables

Conversation

@ChelseaKR

Copy link
Copy Markdown
Owner

What was missing

parse emits one nested JSON document per schedule. The shape downstream
tooling actually consumes is one row per charge with the citation beside it.

Reshaping is where this project's guarantee is easiest to lose. A spreadsheet
of tariff prices with the citations left behind looks more actionable than
the JSON and is worth less, so the design question here is not "how do I write
a CSV" but "what must survive the flattening".

What changed

ca-tariff-parse export <parsed.json> --table charges writes a flat table.
--format jsonl writes the same rows as objects. --all DIR writes every
table. Tables are charges, tou_windows, holidays, proration,
conditions, cross_references and applicability.

Every value travels with its citation. Each cited field is immediately
followed by a <field>.locator column, and document_id, document_sha256
and parser_version sit on every row — so a row lifted out of its file still
names the bytes it came from. --snippets adds <field>.snippet, off by
default because a snippet carries the document's own text (ADR 0003).

Columns are derived from schemas/parsed-schedule-v1.schema.json, not
listed in the export. A field added to the model and the schema gets a column
without anyone remembering; a field added to only one of them fails loudly.

A null is an empty cell. Never 0, never n/a, never something a
spreadsheet will total. A charge that states no season states no season.

Nothing is dropped in the reshape. Each table's row count is checked
against the record count of the parse; a mismatch raises rather than writing a
short table, because a lost row is the one failure a reader of the table
cannot see.

Nothing is computed. No annualised price, no hours-per-window. A derived
number sitting in a table of cited ones is indistinguishable from them.

An empty table is a file with a header. smud-ssr prices nothing, so its
charges table is one header line. A missing file would read as "not
exported".

notes and unparsed are deliberately not tables — notes are the
document's prose rather than records with fields, and an unparsed section
reports where the parser stopped rather than a value it read, so it has no
citation to flatten. That exclusion is checked against the schema: if
unparsedSection ever gains a cited field, the export refuses to run rather
than quietly hiding it. A stale exclusion naming a collection the schema no
longer declares fails the same way.

Determinism. Rows sort by their first citation's own page, sheet,
section and line — not by the rendered locator text, which would put p.10
before p.2 — with the whole row as the final tiebreak. Two exports of one parse
are byte identical.

CSV formula neutralisation with one exception. Cells starting =, +,
@, tab or CR are prefixed with an apostrophe. A leading minus is neutralised
only when the cell is not a number, because a credit is printed as -0.05
and prefixing it would change what a reader sees.

export reads a full parse and a watch baseline alike; the projection removes
only the verbatim prose, which no table carries, and a test asserts both
produce byte-identical tables.

How it was verified

ruff check, ruff format --check, mypy (strict, 27 files) and pytest all
green; coverage 94.37% against an 85% floor. 82 new tests. No golden file or
baseline changed.

Negative controls, each read back out of the file before the gate ran, then
restored from a pre-edit byte copy (diff clean) and re-run green:

sabotage gate
drop the .locator columns red — column-order test
render an absent cited field as 0 / "n/a" red — 2 tests
silently drop credit records from the reshape red — 4 tests incl. row-count
sort by the rendered locator text instead of its parts red — the p.10/p.2 fixture
disable formula neutralisation red — 4 parametrised cases
neutralise negative numbers too red — 3 tests incl. the negative-price case

Two notes on how those tests were written. The page-ordering test is a
synthetic fixture, not a committed document: no pinned document reaches
page 10, so a test written against real data would have passed whatever the
sort key did. And the exclusion guards are exercised by handing _derive_tables
a doctored schema, so they are gates that can actually fail rather than
assertions about the current file.

Closes #49

Prepared with AI assistance; reviewed before submission.

`parse` emits a tree; the shape downstream tooling consumes is one row per
charge. `export` writes that as CSV or JSONL, one table per record type, with
`--all` for every table at once.

The reshape is where the citation is easiest to lose, so every cited field is
followed by its locator column and every row names the document digest and
parser version it came from. A null stays an empty cell, never 0 or n/a. Each
table's row count is checked against the record count, so a reshape cannot
silently write a short table. Nothing is computed.

Columns are derived from the published schema rather than listed here, so they
cannot drift from the model, and the exclusion of `unparsed` is checked against
that schema: if it ever gains a cited field the export refuses rather than
hiding it.

Rows sort on the citation's own page, sheet, section and line rather than the
rendered locator, so page 10 follows page 9. CSV cells a spreadsheet would
evaluate are neutralised, except a leading minus on a number, because a credit
is printed as -0.05.
@ChelseaKR
ChelseaKR merged commit 0aa6275 into main Sep 6, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an export verb that flattens charges, windows and holidays into cited tables

1 participant