feat: an export verb that flattens the parse into cited tables - #63
Merged
Conversation
`parse` emits a tree; the shape downstream tooling consumes is one row per charge. `export` writes that as CSV or JSONL, one table per record type, with `--all` for every table at once. The reshape is where the citation is easiest to lose, so every cited field is followed by its locator column and every row names the document digest and parser version it came from. A null stays an empty cell, never 0 or n/a. Each table's row count is checked against the record count, so a reshape cannot silently write a short table. Nothing is computed. Columns are derived from the published schema rather than listed here, so they cannot drift from the model, and the exclusion of `unparsed` is checked against that schema: if it ever gains a cited field the export refuses rather than hiding it. Rows sort on the citation's own page, sheet, section and line rather than the rendered locator, so page 10 follows page 9. CSV cells a spreadsheet would evaluate are neutralised, except a leading minus on a number, because a credit is printed as -0.05.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was missing
parseemits one nested JSON document per schedule. The shape downstreamtooling actually consumes is one row per charge with the citation beside it.
Reshaping is where this project's guarantee is easiest to lose. A spreadsheet
of tariff prices with the citations left behind looks more actionable than
the JSON and is worth less, so the design question here is not "how do I write
a CSV" but "what must survive the flattening".
What changed
ca-tariff-parse export <parsed.json> --table chargeswrites a flat table.--format jsonlwrites the same rows as objects.--all DIRwrites everytable. Tables are
charges,tou_windows,holidays,proration,conditions,cross_referencesandapplicability.Every value travels with its citation. Each cited field is immediately
followed by a
<field>.locatorcolumn, anddocument_id,document_sha256and
parser_versionsit on every row — so a row lifted out of its file stillnames the bytes it came from.
--snippetsadds<field>.snippet, off bydefault because a snippet carries the document's own text (ADR 0003).
Columns are derived from
schemas/parsed-schedule-v1.schema.json, notlisted in the export. A field added to the model and the schema gets a column
without anyone remembering; a field added to only one of them fails loudly.
A null is an empty cell. Never
0, nevern/a, never something aspreadsheet will total. A charge that states no season states no season.
Nothing is dropped in the reshape. Each table's row count is checked
against the record count of the parse; a mismatch raises rather than writing a
short table, because a lost row is the one failure a reader of the table
cannot see.
Nothing is computed. No annualised price, no hours-per-window. A derived
number sitting in a table of cited ones is indistinguishable from them.
An empty table is a file with a header.
smud-ssrprices nothing, so itschargestable is one header line. A missing file would read as "notexported".
notesandunparsedare deliberately not tables — notes are thedocument's prose rather than records with fields, and an unparsed section
reports where the parser stopped rather than a value it read, so it has no
citation to flatten. That exclusion is checked against the schema: if
unparsedSectionever gains a cited field, the export refuses to run ratherthan quietly hiding it. A stale exclusion naming a collection the schema no
longer declares fails the same way.
Determinism. Rows sort by their first citation's own
page,sheet,sectionandline— not by the rendered locator text, which would put p.10before p.2 — with the whole row as the final tiebreak. Two exports of one parse
are byte identical.
CSV formula neutralisation with one exception. Cells starting
=,+,@, tab or CR are prefixed with an apostrophe. A leading minus is neutralisedonly when the cell is not a number, because a credit is printed as
-0.05and prefixing it would change what a reader sees.
exportreads a full parse and a watch baseline alike; the projection removesonly the verbatim prose, which no table carries, and a test asserts both
produce byte-identical tables.
How it was verified
ruff check,ruff format --check,mypy(strict, 27 files) andpytestallgreen; coverage 94.37% against an 85% floor. 82 new tests. No golden file or
baseline changed.
Negative controls, each read back out of the file before the gate ran, then
restored from a pre-edit byte copy (
diffclean) and re-run green:.locatorcolumns0/"n/a"creditrecords from the reshapeTwo notes on how those tests were written. The page-ordering test is a
synthetic fixture, not a committed document: no pinned document reaches
page 10, so a test written against real data would have passed whatever the
sort key did. And the exclusion guards are exercised by handing
_derive_tablesa doctored schema, so they are gates that can actually fail rather than
assertions about the current file.
Closes #49
Prepared with AI assistance; reviewed before submission.