diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a0612d..4a737b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,69 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this project a ## [Unreleased] +### Fixed, five fields published a zero nobody had ruled on, and the registry could not say so + +- **A reviewed decision and an unexamined field published the same thing.** `NOOFCARSONPROPERTY`, + `NOOUTBUILDINGSDAMAGED`, `NOOUTBUILDINGSNOTDAMAGED`, `NUMBEROFUNITPERSTRUCTURE` and + `ASSESSEDIMPROVEDVALUE` are each `FieldSpec(..., numeric=True)` with no declared + vocabulary, so each carried `Basis.NONE` and published `marker_basis: "none"`. So does a + field nobody has ever opened. `docs/MARKERS.md` section 7 named the problem in its own + last sentence: *"the gate that reads this file has no way to tell a reviewed decision + from an unexamined one."* Four of the five record a zero in over 94% of the cells they + record anything in, and on the measurement reading `NOOFCARSONPROPERTY` says 357 of + 56,188 inspections found a car. + +- **`FieldSpec.zero_reading` is the second axis.** `ZeroReading` has four members and the + default is `UNREVIEWED`: `measurement` (a zero is a finding), `undecidable` (reviewed, + and the published documentation does not settle it, so the zero is left present and must + not be read either way), `marker` (declared, so the zeros are counted as + recorded-as-unknown and the field publishes none), and `unreviewed` (nobody has ruled). + It is a second axis rather than more `Basis` members because `Basis` answers *where a + declared vocabulary came from* and this answers *what a zero is*; a field can be in any + combination of the two, and folding them is the same conflation one column over. + +- **The readings are the ones section 7 already recorded**, transcribed into the registry + rather than invented: three counts are `measurement`, the two contested fields are + `undecidable`, and `YEARBUILT` is `marker`. Section 7 now states the word beside each + row, and `tests/test_schema.py` holds the document and the registry to each other in + both directions by an exact token. + +- **Two numbers on every surface, derived from the rows they describe.** Each artifact + carries a `recorded_zero_review` block: fields measured as numbers, how many carry a + reviewed reading, which ones do not (named, not counted away), and the same pair + restricted to the fields actually publishing a zero. `dins.html` and `perimeters.html` + state it above each field table, and every field's row carries its own reading beside + its zero count. Measured on this retrieval: **6 of 8** DINS fields measured as numbers + carry a reading and **5 of 5** of those publishing a zero do; on the FRAP side **0 of + 1**, and `GIS_ACRES` publishes no zero. `LATITUDE` and `LONGITUDE` are the DINS pair + with no reading, and they publish none either. + +- **The gate.** A field publishing a recorded zero while declaring `unreviewed` fails the + build, with a message naming the two files to change. The older gate asked only whether + a field was *named* somewhere in `docs/MARKERS.md`, which a field can be while the + document says nothing about its zeros. Both are kept: they catch different things, and + the new one is exercised by a positive control built from the five fields exactly as + they stood when issue #83 was filed. + +- **Two gates that would have passed over the change being made.** Both were found by + sabotaging the tree and watching nothing go red. Withdrawing `NOOFCARSONPROPERTY`'s + ruling in `src/perimeter/schema.py`, so the field publishes 55,831 zeros while declaring + that nobody ruled on them, left all 534 tests in `tests/test_schema.py` green: every gate + there reads the *committed* artifact, and the only thing that rebuilds it — `make + site-check` — needs `data/raw/` and cannot run in CI. The published artifact's spine + comparison, which already holds `marker_basis` between a fixture build and the committed + copy, now holds `recorded_zero_reading` beside it. And counting every examinable field as + reviewed, so the block publishes `8 of 8`, left all 1,158 tests green, because every gate + on that block recomputes it from the same payload it was written into; + `zero_review` now has two tests that read its own answer over a field set whose two + numbers are known to differ. + +- **`artifact_schema_version` is now 2.** `recorded_zero_review` is a new required + top-level key and `recorded_zero_reading` a new optional per-field one, so a consumer + validating against version 1 — whose schema closes with `additionalProperties: false` — + would reject the new artifact. That is the stated rule for a bump, and it is stated here + and in `src/perimeter/artifacts.py` so neither can be edited alone. + ### Fixed, the secret scan read one of `main`'s eighty-one commits and reported success - **`secret-scan` was `gitleaks/gitleaks-action`, which picks its scan range from the diff --git a/docs/MARKERS.md b/docs/MARKERS.md index 138e9da..c8843f4 100644 --- a/docs/MARKERS.md +++ b/docs/MARKERS.md @@ -275,11 +275,28 @@ as well: the count in the sentence above, the set of rows, and every number in t | Field | Recorded zeros | Reading | Declared? | |---|---|---|---| -| `NOOUTBUILDINGSDAMAGED` | 59,628 | a count of no outbuildings damaged | no, a zero is the measurement | -| `NOOUTBUILDINGSNOTDAMAGED` | 58,888 | a count of no undamaged outbuildings | no, a zero is the measurement | -| `NOOFCARSONPROPERTY` | 55,831 | a count of no cars | no, a zero is the measurement | -| `NUMBEROFUNITPERSTRUCTURE` | 58,411 | contested; see below | no, and the reason is recorded | -| `ASSESSEDIMPROVEDVALUE` | 6,613 | contested; see below | no, and the reason is recorded | +| `NOOUTBUILDINGSDAMAGED` | 59,628 | a count of no outbuildings damaged | `measurement` : no marker declared; a zero is the finding | +| `NOOUTBUILDINGSNOTDAMAGED` | 58,888 | a count of no undamaged outbuildings | `measurement` : no marker declared; a zero is the finding | +| `NOOFCARSONPROPERTY` | 55,831 | a count of no cars | `measurement` : no marker declared; a zero is the finding | +| `NUMBEROFUNITPERSTRUCTURE` | 58,411 | contested; see below | `undecidable` : reviewed, no marker declared, and the reason is recorded | +| `ASSESSEDIMPROVEDVALUE` | 6,613 | contested; see below | `undecidable` : reviewed, no marker declared, and the reason is recorded | + +The word in the last column is the one the registry carries in +`FieldSpec.zero_reading` and the artifacts publish as `recorded_zero_reading`, and +`tests/test_schema.py` holds the two to each other in both directions. Before it existed +the registry had no way to say any of this: a field whose zeros a reviewer had read the +form for and left as measurements, and a field nobody had ever opened, both carried +`Basis.NONE` and published `marker_basis: "none"`. That is the last sentence of this +section, and issue #83 is it being fixed. + +**What a reader should conclude from each word.** + +| Word | What it says | What a reader should do with a `0` | +|---|---|---| +| `measurement` | Somebody read the inspection form and the zeros' distribution, and a recorded `0` is a finding | Use it as a count of none | +| `undecidable` | Somebody read them and the published documentation does not settle it. Both readings are live and the file cannot separate them, so the zero is **left present** : declaring it a marker would move tens of thousands of records on a reading nothing supports | Do not use it as a finding, and do not use it as an absence. `recorded_zero_values` is published so either reading can be applied | +| `marker` | Declared: `0` stands in for a value that was not recorded, so the zeros are counted as recorded-as-unknown and this field publishes none | There is no `0` to read; it is already counted as an absence | +| `unreviewed` | **Nobody has ruled.** A recorded `0` would be published as a value and might be an absence | Nothing : and nothing can be, which is why the build fails if a field in this state ever publishes one | `YEARBUILT` was the sixth row of that table, and it is the reason the checking above exists. Declaring `0` a marker for it moved all 12,148 of its zeros out of @@ -391,5 +408,20 @@ from scratch. and a count of zero is what an inspector writes when there were none. Their positive values run from 1 upward in the shape a real count has. Nothing is declared and nothing moves; they are listed here because "we looked and a zero is a measurement" is a finding a -reader is entitled to, and because the gate that reads this file has no way to tell a -reviewed decision from an unexamined one. +reader is entitled to. + +`NOOFCARSONPROPERTY` is the hardest of the three and is recorded as such: 55,831 zeros +against 357 positives is 99.4% of everything the field records, which on the measurement +reading means 357 of the 56,188 inspections that wrote anything at all found a car. It is +left as a measurement because that is what the distribution of the positives supports and +because no published documentation offers a placeholder convention to read instead : not +because the share is comfortable. **This is the second entry to revisit**, after +`ASSESSEDIMPROVEDVALUE`, and the reading is now carried in the registry rather than only +here, so changing it is a one-line change with a gate on both sides of it. + +The last sentence of this section used to read *"and because the gate that reads this file +has no way to tell a reviewed decision from an unexamined one."* It now has one: +`FieldSpec.zero_reading`, published per field as `recorded_zero_reading`, summarised in +each artifact's `recorded_zero_review` block as two numbers, and stated on the page above +each field table. A field measured as a number that publishes a recorded zero while +carrying `unreviewed` fails the build. diff --git a/site/data/datapackage.json b/site/data/datapackage.json index 55e6e49..00beb7e 100644 --- a/site/data/datapackage.json +++ b/site/data/datapackage.json @@ -1,5 +1,5 @@ { - "artifactSchemaVersion": 1, + "artifactSchemaVersion": 2, "description": "Counts of cells in three named states across CAL FIRE's historical fire perimeter record and its damage inspection record. Unofficial. Both publishers state in their own words that these files are incomplete; this measures how, in counts, with no estimation and no share over an empty denominator.", "fieldStateMeaning": "The three counts are in the order published as `field_state_order`: present is a recorded value, explicit_unknown is a value the publisher records to mean it could not be determined, and not_recorded is an empty cell. A recorded unknown is neither present nor missing, and folding it into either one is the error this project exists to refuse.", "fieldStateOrder": [ diff --git a/site/data/dins-coverage.json b/site/data/dins-coverage.json index 229e968..b91ff11 100644 --- a/site/data/dins-coverage.json +++ b/site/data/dins-coverage.json @@ -6,7 +6,7 @@ "inaccessible": 591, "total": 132522 }, - "artifact_schema_version": 1, + "artifact_schema_version": 2, "completeness_by_access": [ { "assessed_present": 131931, @@ -53497,6 +53497,7 @@ "outside_published_domain_values_listed": null, "present": 61929, "present_tenths_pct": 467, + "recorded_zero_reading": "undecidable", "recorded_zero_values": 58411, "total": 132522 }, @@ -53513,6 +53514,7 @@ "outside_published_domain_values_listed": null, "present": 61939, "present_tenths_pct": 467, + "recorded_zero_reading": "measurement", "recorded_zero_values": 59628, "total": 132522 }, @@ -53529,6 +53531,7 @@ "outside_published_domain_values_listed": null, "present": 61958, "present_tenths_pct": 468, + "recorded_zero_reading": "measurement", "recorded_zero_values": 58888, "total": 132522 }, @@ -53545,6 +53548,7 @@ "outside_published_domain_values_listed": null, "present": 56188, "present_tenths_pct": 424, + "recorded_zero_reading": "measurement", "recorded_zero_values": 55831, "total": 132522 }, @@ -53604,6 +53608,7 @@ "outside_published_domain_values_listed": null, "present": 89943, "present_tenths_pct": 679, + "recorded_zero_reading": "marker", "recorded_zero_values": 0, "total": 132522 }, @@ -53620,6 +53625,7 @@ "outside_published_domain_values_listed": null, "present": 126482, "present_tenths_pct": 954, + "recorded_zero_reading": "undecidable", "recorded_zero_values": 6613, "total": 132522 }, @@ -53686,6 +53692,7 @@ "outside_published_domain_values_listed": null, "present": 132522, "present_tenths_pct": 1000, + "recorded_zero_reading": "unreviewed", "recorded_zero_values": 0, "total": 132522 }, @@ -53702,6 +53709,7 @@ "outside_published_domain_values_listed": null, "present": 132522, "present_tenths_pct": 1000, + "recorded_zero_reading": "unreviewed", "recorded_zero_values": 0, "total": 132522 } @@ -146015,6 +146023,16 @@ "field": "UTILITYMISCSTRUCTUREDISTANCE", "note": "How often each declared spelling of this field's Not Applicable value was written, per incident-start year. Both spellings are counted as the publisher's finding rather than one being read as missing data; docs/MARKERS.md section 2 sets out the evidence and the confidence." }, + "recorded_zero_review": { + "fields_measured_as_numbers": 8, + "fields_publishing_a_recorded_zero": 5, + "fields_publishing_a_recorded_zero_with_a_reviewed_reading": 5, + "fields_with_a_reviewed_zero_reading": 6, + "fields_without_one": [ + "LATITUDE", + "LONGITUDE" + ] + }, "records": 132522, "records_not_attributable_to_an_incident": 0, "source": { diff --git a/site/data/perimeters-coverage.json b/site/data/perimeters-coverage.json index 7204512..b2b1aee 100644 --- a/site/data/perimeters-coverage.json +++ b/site/data/perimeters-coverage.json @@ -169,7 +169,7 @@ 300 ] }, - "artifact_schema_version": 1, + "artifact_schema_version": 2, "duplicate_signals": [ { "description": "Records sharing one IRWIN ID. IRWIN is the federal incident identifier, so a repeat means two perimeter records point at one federal incident.", @@ -427,6 +427,7 @@ "outside_published_domain_values_listed": null, "present": 23334, "present_tenths_pct": 1000, + "recorded_zero_reading": "unreviewed", "recorded_zero_values": 0, "total": 23334 }, @@ -494,6 +495,15 @@ } ], "measurement": "Completeness of the FRAP historical fire perimeter record", + "recorded_zero_review": { + "fields_measured_as_numbers": 1, + "fields_publishing_a_recorded_zero": 0, + "fields_publishing_a_recorded_zero_with_a_reviewed_reading": 0, + "fields_with_a_reviewed_zero_reading": 0, + "fields_without_one": [ + "GIS_ACRES" + ] + }, "records": 23334, "records_without_year": 77, "source": { diff --git a/site/data/schema/dins-coverage.schema.json b/site/data/schema/dins-coverage.schema.json index 8b1b6e9..01f4f74 100644 --- a/site/data/schema/dins-coverage.schema.json +++ b/site/data/schema/dins-coverage.schema.json @@ -44,7 +44,7 @@ "type": "object" }, "artifact_schema_version": { - "const": 1, + "const": 2, "description": "The version of this contract. Bumped when a consumer validating against the previous schema would reject the new artifact or would read an existing key as meaning something else.", "type": "integer" }, @@ -597,6 +597,16 @@ "null" ] }, + "recorded_zero_reading": { + "description": "What a recorded zero in this field MEANS, as a reviewer has ruled. `measurement`: a zero is a finding. `undecidable`: reviewed, and the published documentation does not settle it, so the zero is left present and must not be read as a finding OR as an absence. `marker`: declared as a marker, so this field's zeros are counted in `explicit_unknown` and `recorded_zero_values` is 0. `unreviewed`: NOBODY HAS RULED -- a zero here may be an absence, and the build fails if such a field ever publishes one. Read beside `recorded_zero_values`, never instead of it, and see docs/MARKERS.md section 7. Absent for a field that is not numeric.", + "enum": [ + "unreviewed", + "measurement", + "undecidable", + "marker" + ], + "type": "string" + }, "recorded_zero_values": { "description": "Cells recording a literal zero, for a field measured as a number. A zero is a judgment call (ADR 0006) and is published separately rather than folded into either presence or absence. Absent for a field that is not numeric.", "type": "integer" @@ -794,6 +804,43 @@ ], "type": "object" }, + "recorded_zero_review": { + "additionalProperties": false, + "description": "Coverage of the zero question over this artifact's fields. Every figure is derived from the fields beside it; none is stated anywhere.", + "properties": { + "fields_measured_as_numbers": { + "description": "The denominator: every field that can carry a zero, and so every field this project either has or has not ruled on.", + "type": "integer" + }, + "fields_publishing_a_recorded_zero": { + "description": "Of the fields measured as numbers, how many actually carry a zero in this retrieval.", + "type": "integer" + }, + "fields_publishing_a_recorded_zero_with_a_reviewed_reading": { + "description": "Of those, how many carry a reading. Smaller than the figure above means a zero nobody has ruled on is being published as a value, which is the condition `tests/test_schema.py` fails the build on.", + "type": "integer" + }, + "fields_with_a_reviewed_zero_reading": { + "description": "Of those, how many carry a `recorded_zero_reading` other than `unreviewed`. `undecidable` counts as reviewed: a review that reached no verdict is still a review, and it is a finding rather than a gap.", + "type": "integer" + }, + "fields_without_one": { + "description": "The rest, named rather than counted away. A zero appearing in any of these fails the build until somebody rules on it.", + "items": { + "type": "string" + }, + "type": "array" + } + }, + "required": [ + "fields_measured_as_numbers", + "fields_publishing_a_recorded_zero", + "fields_publishing_a_recorded_zero_with_a_reviewed_reading", + "fields_with_a_reviewed_zero_reading", + "fields_without_one" + ], + "type": "object" + }, "records": { "description": "Records measured.", "type": "integer" @@ -957,6 +1004,7 @@ "is_fixture", "measurement", "not_applicable_spellings_by_year", + "recorded_zero_review", "records", "records_not_attributable_to_an_incident", "source" diff --git a/site/data/schema/perimeters-coverage.schema.json b/site/data/schema/perimeters-coverage.schema.json index 654571d..040d35b 100644 --- a/site/data/schema/perimeters-coverage.schema.json +++ b/site/data/schema/perimeters-coverage.schema.json @@ -72,7 +72,7 @@ "type": "object" }, "artifact_schema_version": { - "const": 1, + "const": 2, "description": "The version of this contract. Bumped when a consumer validating against the previous schema would reject the new artifact or would read an existing key as meaning something else.", "type": "integer" }, @@ -205,6 +205,16 @@ "null" ] }, + "recorded_zero_reading": { + "description": "What a recorded zero in this field MEANS, as a reviewer has ruled. `measurement`: a zero is a finding. `undecidable`: reviewed, and the published documentation does not settle it, so the zero is left present and must not be read as a finding OR as an absence. `marker`: declared as a marker, so this field's zeros are counted in `explicit_unknown` and `recorded_zero_values` is 0. `unreviewed`: NOBODY HAS RULED -- a zero here may be an absence, and the build fails if such a field ever publishes one. Read beside `recorded_zero_values`, never instead of it, and see docs/MARKERS.md section 7. Absent for a field that is not numeric.", + "enum": [ + "unreviewed", + "measurement", + "undecidable", + "marker" + ], + "type": "string" + }, "recorded_zero_values": { "description": "Cells recording a literal zero, for a field measured as a number. A zero is a judgment call (ADR 0006) and is published separately rather than folded into either presence or absence. Absent for a field that is not numeric.", "type": "integer" @@ -302,6 +312,43 @@ "description": "What this artifact measures, in one sentence.", "type": "string" }, + "recorded_zero_review": { + "additionalProperties": false, + "description": "Coverage of the zero question over this artifact's fields. Every figure is derived from the fields beside it; none is stated anywhere.", + "properties": { + "fields_measured_as_numbers": { + "description": "The denominator: every field that can carry a zero, and so every field this project either has or has not ruled on.", + "type": "integer" + }, + "fields_publishing_a_recorded_zero": { + "description": "Of the fields measured as numbers, how many actually carry a zero in this retrieval.", + "type": "integer" + }, + "fields_publishing_a_recorded_zero_with_a_reviewed_reading": { + "description": "Of those, how many carry a reading. Smaller than the figure above means a zero nobody has ruled on is being published as a value, which is the condition `tests/test_schema.py` fails the build on.", + "type": "integer" + }, + "fields_with_a_reviewed_zero_reading": { + "description": "Of those, how many carry a `recorded_zero_reading` other than `unreviewed`. `undecidable` counts as reviewed: a review that reached no verdict is still a review, and it is a finding rather than a gap.", + "type": "integer" + }, + "fields_without_one": { + "description": "The rest, named rather than counted away. A zero appearing in any of these fails the build until somebody rules on it.", + "items": { + "type": "string" + }, + "type": "array" + } + }, + "required": [ + "fields_measured_as_numbers", + "fields_publishing_a_recorded_zero", + "fields_publishing_a_recorded_zero_with_a_reviewed_reading", + "fields_with_a_reviewed_zero_reading", + "fields_without_one" + ], + "type": "object" + }, "records": { "description": "Records measured.", "type": "integer" @@ -510,6 +557,7 @@ "latest_year", "marker_counterfactuals", "measurement", + "recorded_zero_review", "records", "records_without_year", "source", diff --git a/site/dins.html b/site/dins.html index edccb58..eba81fa 100644 --- a/site/dins.html +++ b/site/dins.html @@ -212,7 +212,8 @@

Three states, field by field

missing data. A recorded unknown is a published code or a marker word standing where a value would go. An empty cell is what CAL FIRE describes as an attribute that could not be determined.

-
Damage inspection fields, each counted in three states
FieldRecorded valueRecorded as unknownEmpty cellValue presentSplit
Damage
Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
DAMAGE
132,52200100.0%
Structure type
156 values outside the published domain across 4 distinct spellings
STRUCTURETYPE
132,52200100.0%
Structure category
7 values outside the published domain across 1 distinct spelling
STRUCTURECATEGORY
132,52200100.0%
Incident name
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
INCIDENTNAME
132,52200100.0%
Incident number
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
INCIDENTNUM
132,52200100.0%
Incident start date
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
INCIDENTSTARTDATE
132,52200100.0%
Hazard type
HAZARDTYPE
132,52200100.0%
County
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
COUNTY
132,492030100.0%
CAL FIRE unit
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
CALFIREUNIT
132,52200100.0%
Community
markers: - (43), n/a (50), na (640), none (11), unknown (28). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
COMMUNITY
54,50277277,24841.1%
City
markers: n/a (731), na (26,824), none (1), unknown (9). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
CITY
100,03327,5654,92475.5%
Roof construction
markers: unknown (18,535). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
14,203 values outside the published domain across 5 distinct spellings
ROOFCONSTRUCTION
109,25718,5354,73082.4%
Eaves
markers: not applicable (893), unknown (51,471). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
2 values outside the published domain across 1 distinct spelling
EAVES
74,94452,3645,21456.6%
Vent screen
markers: unknown (40,139). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
5,740 values outside the published domain across 5 distinct spellings
VENTSCREEN
87,08140,1395,30265.7%
Exterior siding
markers: unknown (17,540). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
37,576 values outside the published domain across 4 distinct spellings
EXTERIORSIDING
109,97017,5405,01283.0%
Window pane
markers: unknown (33,469). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
3 values outside the published domain across 3 distinct spellings
WINDOWPANE
93,97333,4695,08070.9%
Deck or porch on grade
markers: unknown (15,777). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
7 values outside the published domain across 1 distinct spelling
DECKPORCHONGRADE
95,15315,77721,59271.8%
Deck or porch elevated
markers: unknown (18,750). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
17 values outside the published domain across 1 distinct spelling
DECKPORCHELEVATED
92,17918,75021,59369.6%
Patio cover or carport
markers: unknown (20,235). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
PATIOCOVERCARPORT
90,68920,23521,59868.4%
Fence attached to structure
markers: unknown (15,133). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
FENCEATTACHEDTOSTRUCTURE
95,79515,13321,59472.3%
Propane tank distance
markers: unknown (6,646). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
PROPANETANKDISTANCE
48,5476,64677,32936.6%
Utility or misc structure distance
Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
6,787 values outside the published domain across 3 distinct spellings
UTILITYMISCSTRUCTUREDISTANCE
37,783094,73928.5%
Where fire started on structure
markers: not applicable (23), unknown (8,716). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
157 values outside the published domain across 7 distinct spellings
WHEREFIRESTARTEDONSTRUCTURE
3,6618,739120,1222.8%
What the fire started from
markers: not applicable (18), unknown (5,810). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
5 values outside the published domain across 4 distinct spellings
WHATDIDFIRESTARTFROM
4,8485,828121,8463.7%
Defensive actions taken
markers: unknown (27,707). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
691 values outside the published domain across 4 distinct spellings
DEFENSIVEACTIONS
8,46227,70796,3536.4%
Units in structure
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
NUMBEROFUNITPERSTRUCTURE
61,929070,59346.7%
Damaged outbuildings
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
NOOUTBUILDINGSDAMAGED
61,939070,58346.7%
Undamaged outbuildings
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
NOOUTBUILDINGSNOTDAMAGED
61,958070,56446.8%
Damaged or destroyed cars
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
NOOFCARSONPROPERTY
56,188076,33442.4%
APN (parcel)
markers: none (42), unknown (1). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
APN
131,6154386499.3%
Site address (parcel)
markers: - (3), no address available (513), none (56), null null unknown ca 00000 (160). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
SITEADDRESS
124,1387327,65293.7%
Year built (parcel)
markers: 0 (12,148). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
YEARBUILT
89,94312,14830,43167.9%
Assessed improved value (parcel)
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
ASSESSEDIMPROVEDVALUE
126,48206,04095.4%
Street name (field determined)
markers: n/a (18), na (1,204), none (12), unknown (295). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
STREETNAME
126,6601,5294,33395.6%
Street type (field determined)
markers: - (7), n/a (1), unk (1). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
1,516 values outside the published domain across 8 distinct spellings
STREETTYPE
125,80896,70594.9%
Latitude
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
LATITUDE
132,52200100.0%
Longitude
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
LONGITUDE
132,52200100.0%

There are three things a field can say about a published coded-value domain, and every row above says exactly one of them. It carries values the domain does not describe, which are counted and named under the field. It has a published domain and holds nothing outside it, and carries no note. Or the layer publishes no domain for that field at all, which the row says in words, because nothing there was compared against anything and a zero would have read as a finding that the file and the domain agree.

+

A zero is a judgment call, and this says who made it. 6 of the 8 fields measured as numbers carry a reviewed reading of what a recorded zero means; 2 do not. None of them is: LATITUDE, LONGITUDE, which publish no recorded zero in this retrieval. A zero appearing in any of them stops the build until somebody rules on it. Of the 5 that publish a recorded zero today, 5 carry a reading. Each field's own reading is on its row below, and the evidence is in docs/MARKERS.md section 7.

+
Damage inspection fields, each counted in three states
FieldRecorded valueRecorded as unknownEmpty cellValue presentSplit
Damage
Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
DAMAGE
132,52200100.0%
Structure type
156 values outside the published domain across 4 distinct spellings
STRUCTURETYPE
132,52200100.0%
Structure category
7 values outside the published domain across 1 distinct spelling
STRUCTURECATEGORY
132,52200100.0%
Incident name
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
INCIDENTNAME
132,52200100.0%
Incident number
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
INCIDENTNUM
132,52200100.0%
Incident start date
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
INCIDENTSTARTDATE
132,52200100.0%
Hazard type
HAZARDTYPE
132,52200100.0%
County
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
COUNTY
132,492030100.0%
CAL FIRE unit
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
CALFIREUNIT
132,52200100.0%
Community
markers: - (43), n/a (50), na (640), none (11), unknown (28). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
COMMUNITY
54,50277277,24841.1%
City
markers: n/a (731), na (26,824), none (1), unknown (9). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
CITY
100,03327,5654,92475.5%
Roof construction
markers: unknown (18,535). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
14,203 values outside the published domain across 5 distinct spellings
ROOFCONSTRUCTION
109,25718,5354,73082.4%
Eaves
markers: not applicable (893), unknown (51,471). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
2 values outside the published domain across 1 distinct spelling
EAVES
74,94452,3645,21456.6%
Vent screen
markers: unknown (40,139). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
5,740 values outside the published domain across 5 distinct spellings
VENTSCREEN
87,08140,1395,30265.7%
Exterior siding
markers: unknown (17,540). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
37,576 values outside the published domain across 4 distinct spellings
EXTERIORSIDING
109,97017,5405,01283.0%
Window pane
markers: unknown (33,469). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
3 values outside the published domain across 3 distinct spellings
WINDOWPANE
93,97333,4695,08070.9%
Deck or porch on grade
markers: unknown (15,777). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
7 values outside the published domain across 1 distinct spelling
DECKPORCHONGRADE
95,15315,77721,59271.8%
Deck or porch elevated
markers: unknown (18,750). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
17 values outside the published domain across 1 distinct spelling
DECKPORCHELEVATED
92,17918,75021,59369.6%
Patio cover or carport
markers: unknown (20,235). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
PATIOCOVERCARPORT
90,68920,23521,59868.4%
Fence attached to structure
markers: unknown (15,133). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
FENCEATTACHEDTOSTRUCTURE
95,79515,13321,59472.3%
Propane tank distance
markers: unknown (6,646). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
PROPANETANKDISTANCE
48,5476,64677,32936.6%
Utility or misc structure distance
Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
6,787 values outside the published domain across 3 distinct spellings
UTILITYMISCSTRUCTUREDISTANCE
37,783094,73928.5%
Where fire started on structure
markers: not applicable (23), unknown (8,716). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
157 values outside the published domain across 7 distinct spellings
WHEREFIRESTARTEDONSTRUCTURE
3,6618,739120,1222.8%
What the fire started from
markers: not applicable (18), unknown (5,810). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
5 values outside the published domain across 4 distinct spellings
WHATDIDFIRESTARTFROM
4,8485,828121,8463.7%
Defensive actions taken
markers: unknown (27,707). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
691 values outside the published domain across 4 distinct spellings
DEFENSIVEACTIONS
8,46227,70796,3536.4%
Units in structure
58,411 recorded zeros; reviewed, and undecidable from the published documentation: both readings are live and the file cannot separate them, so the zeros are left as recorded values. Do not read a zero here as a finding, and do not read it as an absence
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
NUMBEROFUNITPERSTRUCTURE
61,929070,59346.7%
Damaged outbuildings
59,628 recorded zeros; reviewed: a zero here is a measurement. An inspector writing 0 observed none, and the zeros stay counted as recorded values
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
NOOUTBUILDINGSDAMAGED
61,939070,58346.7%
Undamaged outbuildings
58,888 recorded zeros; reviewed: a zero here is a measurement. An inspector writing 0 observed none, and the zeros stay counted as recorded values
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
NOOUTBUILDINGSNOTDAMAGED
61,958070,56446.8%
Damaged or destroyed cars
55,831 recorded zeros; reviewed: a zero here is a measurement. An inspector writing 0 observed none, and the zeros stay counted as recorded values
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
NOOFCARSONPROPERTY
56,188076,33442.4%
APN (parcel)
markers: none (42), unknown (1). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
APN
131,6154386499.3%
Site address (parcel)
markers: - (3), no address available (513), none (56), null null unknown ca 00000 (160). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
SITEADDRESS
124,1387327,65293.7%
Year built (parcel)
markers: 0 (12,148). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
0 recorded zeros; reviewed and declared a marker: the zeros are counted as recorded-as-unknown above, not as values, so this field publishes none
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
YEARBUILT
89,94312,14830,43167.9%
Assessed improved value (parcel)
6,613 recorded zeros; reviewed, and undecidable from the published documentation: both readings are live and the file cannot separate them, so the zeros are left as recorded values. Do not read a zero here as a finding, and do not read it as an absence
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
ASSESSEDIMPROVEDVALUE
126,48206,04095.4%
Street name (field determined)
markers: n/a (18), na (1,204), none (12), unknown (295). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
STREETNAME
126,6601,5294,33395.6%
Street type (field determined)
markers: - (7), n/a (1), unk (1). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
1,516 values outside the published domain across 8 distinct spellings
STREETTYPE
125,80896,70594.9%
Latitude
0 recorded zeros; NOT REVIEWED. Nobody has read the inspection form for this field, so a zero here is published as a recorded value and may be an absence
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
LATITUDE
132,52200100.0%
Longitude
0 recorded zeros; NOT REVIEWED. Nobody has read the inspection form for this field, so a zero here is published as a recorded value and may be an absence
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
LONGITUDE
132,52200100.0%

There are three things a field can say about a published coded-value domain, and every row above says exactly one of them. It carries values the domain does not describe, which are counted and named under the field. It has a published domain and holds nothing outside it, and carries no note. Or the layer publishes no domain for that field at all, which the row says in words, because nothing there was compared against anything and a zero would have read as a finding that the file and the domain agree.

Each field carries the basis its markers rest on. A published basis means the value appears in the layer's own coded-value domain, which anyone can re-read at the endpoint listed under Provenance below. An inferred basis means diff --git a/site/perimeters.html b/site/perimeters.html index 23aa401..bcd5379 100644 --- a/site/perimeters.html +++ b/site/perimeters.html @@ -197,7 +197,8 @@

Three states, field by field

and a large share of them carry the published code for Unknown / Unidentified. Collection method behaves the same way, which is what FRAP's release note about editing null collection methods to Unknown describes.

-
Perimeter fields, each counted in three states
FieldRecorded valueRecorded as unknownEmpty cellValue presentSplit
Year
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
YEAR_
23,25707799.7%
IRWIN ID
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
IRWINID
3,633019,70115.6%
Alarm date
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
ALARM_DATE
17,95005,38476.9%
Containment date
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
CONT_DATE
10,716012,61845.9%
Fire name
markers: n/a (72), none (1), unknown (12). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
FIRE_NAME
16,666856,58371.4%
Local incident number
markers: 00000000 (12,469). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
INC_NUM
9,91012,46995542.5%
Fire number (historical use)
markers: 00000000 (3,620), <null> (3). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
FIRE_NUM
13,5213,6236,19057.9%
Cause
markers: 14 (10,514). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
CAUSE
12,82010,514054.9%
Collection method
markers: 8 (15,081). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
C_METHOD
8,25315,081035.4%
Management objective
OBJECTIVE
23,063027198.8%
Reporting agency
AGENCY
23,28604899.8%
Unit
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
UNIT_ID
23,27406099.7%
State
STATE
23,33400100.0%
GIS calculated acres
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
GIS_ACRES
23,33400100.0%
Complex name
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
COMPLEX_NAME
618022,7162.6%
Complex ID
markers: 00000000 (1). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
COMPLEX_ID
583122,7502.5%
Decade
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
DECADES
18,06405,27077.4%

There are three things a field can say about a published coded-value domain, and every row above says exactly one of them. It carries values the domain does not describe, which are counted and named under the field. It has a published domain and holds nothing outside it, and carries no note. Or the layer publishes no domain for that field at all, which the row says in words, because nothing there was compared against anything and a zero would have read as a finding that the file and the domain agree.

+

A zero is a judgment call, and this says who made it. 0 of the 1 fields measured as numbers carry a reviewed reading of what a recorded zero means; 1 do not. None of them is: GIS_ACRES, which publish no recorded zero in this retrieval. A zero appearing in any of them stops the build until somebody rules on it. Of the 0 that publish a recorded zero today, 0 carry a reading. Each field's own reading is on its row below, and the evidence is in docs/MARKERS.md section 7.

+
Perimeter fields, each counted in three states
FieldRecorded valueRecorded as unknownEmpty cellValue presentSplit
Year
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
YEAR_
23,25707799.7%
IRWIN ID
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
IRWINID
3,633019,70115.6%
Alarm date
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
ALARM_DATE
17,95005,38476.9%
Containment date
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
CONT_DATE
10,716012,61845.9%
Fire name
markers: n/a (72), none (1), unknown (12). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
FIRE_NAME
16,666856,58371.4%
Local incident number
markers: 00000000 (12,469). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
INC_NUM
9,91012,46995542.5%
Fire number (historical use)
markers: 00000000 (3,620), <null> (3). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
FIRE_NUM
13,5213,6236,19057.9%
Cause
markers: 14 (10,514). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
CAUSE
12,82010,514054.9%
Collection method
markers: 8 (15,081). Basis: published. Every value this field declares is a code in the layer's own coded-value domain, or a value the publisher's documentation states
C_METHOD
8,25315,081035.4%
Management objective
OBJECTIVE
23,063027198.8%
Reporting agency
AGENCY
23,28604899.8%
Unit
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
UNIT_ID
23,27406099.7%
State
STATE
23,33400100.0%
GIS calculated acres
0 recorded zeros; NOT REVIEWED. Nobody has read the inspection form for this field, so a zero here is published as a recorded value and may be an absence
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
GIS_ACRES
23,33400100.0%
Complex name
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
COMPLEX_NAME
618022,7162.6%
Complex ID
markers: 00000000 (1). Basis: inferred. At least one value this field declares is documented nowhere, and was read off the acquired file by this project. See docs/MARKERS.md
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
COMPLEX_ID
583122,7502.5%
Decade
The layer publishes no coded-value domain for this field, so no value here is counted as outside one
DECADES
18,06405,27077.4%

There are three things a field can say about a published coded-value domain, and every row above says exactly one of them. It carries values the domain does not describe, which are counted and named under the field. It has a published domain and holds nothing outside it, and carries no note. Or the layer publishes no domain for that field at all, which the row says in words, because nothing there was compared against anything and a zero would have read as a finding that the file and the domain agree.

Markers listed under a field name are the exact text this project reviewed for that field before counting it as a marker rather than as a value. A field number of all zeros, for instance, is counted apart from recorded numbers, because reading diff --git a/src/perimeter/artifacts.py b/src/perimeter/artifacts.py index 17765db..ffacbaf 100644 --- a/src/perimeter/artifacts.py +++ b/src/perimeter/artifacts.py @@ -22,8 +22,10 @@ DinsReport, FieldCoverage, PerimeterReport, + ZeroReview, dins_report, perimeter_report, + zero_review, ) from perimeter.dins import NOT_APPLICABLE_FIELD, AccessSplit, load_inspections from perimeter.perimeters import ( @@ -36,7 +38,7 @@ FIELD_STATE_ORDER = ("present", "explicit_unknown", "not_recorded") """The order of the three counts in every compact per-incident field triple.""" -ARTIFACT_SCHEMA_VERSION = 1 +ARTIFACT_SCHEMA_VERSION = 2 """The version of the published artifact contract, carried in every artifact. Bump it when a consumer validating against the previous schema in ``site/data/schema/`` @@ -118,11 +120,30 @@ def _field_json(field: FieldCoverage) -> dict[str, Any]: } if field.zero_values is not None: payload["recorded_zero_values"] = field.zero_values + if field.zero_reading is not None: + # Published beside the count and never instead of it. The count says how many + # zeros there are; this says whether anybody has ruled on what one MEANS, which + # `marker_basis: "none"` could not distinguish from "reviewed and left as a + # measurement" (issue #83). + payload["recorded_zero_reading"] = field.zero_reading.value if field.note: payload["note"] = field.note return payload +def _zero_review_json(review: ZeroReview) -> dict[str, Any]: + """The two numbers, and the unreviewed set named rather than counted away.""" + return { + "fields_measured_as_numbers": review.examinable, + "fields_with_a_reviewed_zero_reading": review.reviewed, + "fields_without_one": list(review.unreviewed_fields), + "fields_publishing_a_recorded_zero": review.publishing_zeros, + "fields_publishing_a_recorded_zero_with_a_reviewed_reading": ( + review.publishing_zeros_reviewed + ), + } + + def _signal_json(signal: DuplicateSignal) -> dict[str, Any]: return { "key": signal.key, @@ -185,6 +206,7 @@ def perimeters_payload(report: PerimeterReport, *, is_fixture: bool) -> dict[str "records_without_year": report.records_without_year, "irwin_id_present": report.irwin_present, "fields": [_field_json(field) for field in report.fields], + "recorded_zero_review": _zero_review_json(zero_review(report.fields)), "irwin_id_present_tenths_pct": present_tenths_of_percent( report.irwin_present, report.records ), @@ -241,6 +263,7 @@ def dins_payload(report: DinsReport, *, is_fixture: bool) -> dict[str, Any]: for value, count in report.damage.items() }, "fields": [_field_json(field) for field in report.fields], + "recorded_zero_review": _zero_review_json(zero_review(report.fields)), "field_state_order": list(FIELD_STATE_ORDER), # Three cuts of the same records. Each is a partition of the file, so summing a # cut per field and per state returns the file totals; nothing is estimated and diff --git a/src/perimeter/coverage.py b/src/perimeter/coverage.py index c6d8120..7a77fef 100644 --- a/src/perimeter/coverage.py +++ b/src/perimeter/coverage.py @@ -38,7 +38,7 @@ year_cohorts, ) from perimeter.records import Record -from perimeter.schema import DINS_FIELDS, FRAP_FIELDS, Basis, FieldSpec +from perimeter.schema import DINS_FIELDS, FRAP_FIELDS, Basis, FieldSpec, ZeroReading OUTSIDE_DOMAIN_VALUE_CAP = 12 """How many out-of-domain values to name per field. @@ -75,6 +75,15 @@ class FieldCoverage: outside_domain_values: dict[str, int] | None outside_domain_values_listed: int | None zero_values: int | None + zero_reading: ZeroReading | None + """What a recorded ``0`` in this field means, as a reviewer has ruled (ADR-0006, + issue #83), or ``None`` for a field that is not measured as a number. + + ``None`` together with ``zero_values`` for the same reason the four out-of-domain + measures are ``None`` together: a field with no zeros to rule on was never ruled on, + and publishing ``unreviewed`` for it would put every free-text column into the + denominator of a review that does not apply to it. + """ @property def total(self) -> int: @@ -132,6 +141,73 @@ def field_coverage(records: Sequence[Record], spec: FieldSpec) -> FieldCoverage: outside_domain_values=listed if has_domain else None, outside_domain_values_listed=len(listed) if has_domain else None, zero_values=zeros if spec.numeric else None, + zero_reading=spec.zero_reading if spec.numeric else None, + ) + + +@dataclass(frozen=True) +class ZeroReview: + """How much of the zero question has been answered, and how much has not. + + Two numbers, always, because one of them alone is the defect. "Five fields publish + recorded zeros and all five are reviewed" is a reassuring sentence that says nothing + about the fields measured as numbers that hold no zeros *today* -- and a zero arriving + in one of those tomorrow is exactly the case that used to be published as a + measurement nobody had ruled on. So: ``reviewed`` of ``examinable``, and the + unreviewed set named rather than counted away. + """ + + examinable: int + """Fields measured as numbers. Every one of them can have a zero, so every one of + them is a question this project either has or has not answered.""" + + reviewed: int + """Of those, how many carry a reading somebody made. See + :attr:`FieldSpec.zeros_are_reviewed`.""" + + unreviewed_fields: tuple[str, ...] + """The names of the rest, in registry order. Named rather than counted, because the + count alone cannot be acted on and this list is the work item.""" + + publishing_zeros: int + """Of the examinable fields, how many actually publish a recorded zero today.""" + + publishing_zeros_reviewed: int + """Of those, how many carry a reading. A build where this is smaller than + ``publishing_zeros`` is publishing a zero nobody has ruled on, and + ``tests/test_schema.py`` fails it.""" + + @property + def unreviewed(self) -> int: + return len(self.unreviewed_fields) + + +def zero_review(fields: Sequence[FieldCoverage]) -> ZeroReview: + """Count the zero question over one artifact's fields, from the fields themselves. + + Derived here rather than stated anywhere, so the pages and the artifact cannot carry a + figure that has fallen behind the registry. Every number a page prints about this + comes from this function via the artifact. + """ + numeric = [field for field in fields if field.zero_reading is not None] + reviewed = [ + field for field in numeric if field.zero_reading is not ZeroReading.UNREVIEWED + ] + with_zeros = [field for field in numeric if field.zero_values] + return ZeroReview( + examinable=len(numeric), + reviewed=len(reviewed), + unreviewed_fields=tuple( + field.name + for field in numeric + if field.zero_reading is ZeroReading.UNREVIEWED + ), + publishing_zeros=len(with_zeros), + publishing_zeros_reviewed=sum( + 1 + for field in with_zeros + if field.zero_reading is not ZeroReading.UNREVIEWED + ), ) diff --git a/src/perimeter/render.py b/src/perimeter/render.py index 3675a44..d5431f3 100644 --- a/src/perimeter/render.py +++ b/src/perimeter/render.py @@ -26,8 +26,9 @@ FieldCoverage, PerimeterReport, SpellingCohort, + zero_review, ) -from perimeter.schema import Basis +from perimeter.schema import Basis, ZeroReading from perimeter.sources import DINS, FRAP, Source # Where these pages are served. The path segment is load-bearing: this is one of six project @@ -64,6 +65,31 @@ "state agency." ) +ZERO_READING_NOTE: dict[ZeroReading, str] = { + ZeroReading.MEASUREMENT: ( + "reviewed: a zero here is a measurement. An inspector writing 0 observed none, " + "and the zeros stay counted as recorded values" + ), + ZeroReading.UNDECIDABLE: ( + "reviewed, and undecidable from the published documentation: both readings are " + "live and the file cannot separate them, so the zeros are left as recorded " + "values. Do not read a zero here as a finding, and do not read it as an absence" + ), + ZeroReading.MARKER: ( + "reviewed and declared a marker: the zeros are counted as recorded-as-unknown " + "above, not as values, so this field publishes none" + ), + ZeroReading.UNREVIEWED: ( + "NOT REVIEWED. Nobody has read the inspection form for this field, so a zero " + "here is published as a recorded value and may be an absence" + ), +} +"""What a recorded zero in a field means, for a reader. See docs/MARKERS.md section 7. + +`UNREVIEWED` carries a sentence like the other three rather than rendering nothing, +because rendering nothing is precisely what made an unexamined field and a reviewed one +look identical on this page (issue #83).""" + BASIS_NOTE: dict[Basis, str] = { Basis.PUBLISHED: ( "published. Every value this field declares is a code in the layer's own " @@ -454,6 +480,40 @@ def scroll_region(caption_id: str, *, tall: bool = False) -> str: return f'

' +def zero_review_note(fields: Sequence[FieldCoverage]) -> str: + """The two numbers: how many of the fields that can carry a zero have been ruled on. + + One number would be the defect. "Every field publishing a zero has been reviewed" is + true of this build and says nothing about the fields measured as numbers that hold no + zero today -- and a zero arriving in one of those is exactly the case that used to be + published as a measurement nobody had ruled on. So the denominator is every field that + CAN carry one, the unreviewed ones are named, and the sentence says what happens if a + zero turns up in one. + + Both figures are computed here from the fields the table below draws, so the page + cannot state a coverage it does not have. + """ + review = zero_review(fields) + named = ( + " None of them is: " + + ", ".join(f"{esc(name)}" for name in review.unreviewed_fields) + + ", which publish no recorded zero in this retrieval. A zero appearing in any of " + "them stops the build until somebody rules on it." + if review.unreviewed_fields + else " There is none left unreviewed." + ) + return ( + f'

A zero is a judgment call, and this says who made ' + f"it. {num(review.reviewed)} of the {num(review.examinable)} fields " + f"measured as numbers carry a reviewed reading of what a recorded zero means; " + f"{num(review.unreviewed)} do not.{named} Of the " + f"{num(review.publishing_zeros)} that publish a recorded zero today, " + f"{num(review.publishing_zeros_reviewed)} carry a reading. Each field's own " + f"reading is on its row below, and the evidence is in " + f"docs/MARKERS.md section 7.

" + ) + + def field_table( fields: Sequence[FieldCoverage], *, caption_id: str, caption: str ) -> str: @@ -474,6 +534,18 @@ def field_table( marker_note = ( f'
Basis: {esc(BASIS_NOTE[field.basis])}
' ) + # A zero is a judgment call of the same kind as a marker word (ADR-0006), and + # until this line existed the page published the judgment's RESULT -- the count of + # recorded values -- with no way to see whether anybody had made it. The count + # reached the artifact and never the page. + zeros = "" + if field.zero_reading is not None: + reading = ZERO_READING_NOTE[field.zero_reading] + recorded = field.zero_values or 0 + zeros = ( + f'
{num(recorded)} recorded zero' + f"{'' if recorded == 1 else 's'}; {esc(reading)}
" + ) count, distinct = field.outside_domain, field.outside_domain_distinct outside = "" if count is None or distinct is None: @@ -494,7 +566,7 @@ def field_table( ) rows.append( "" - f'{esc(field.label)}{marker_note}{outside}' + f'{esc(field.label)}{marker_note}{zeros}{outside}' f'
{esc(field.name)}
' f'{num(field.present)}' f'{num(field.explicit_unknown)}' @@ -701,6 +773,7 @@ def perimeters_page(report: PerimeterReport, *, is_fixture: bool) -> str: and a large share of them carry the published code for Unknown / Unidentified. Collection method behaves the same way, which is what FRAP's release note about editing null collection methods to Unknown describes.

+{zero_review_note(report.fields)} { field_table( report.fields, @@ -1082,6 +1155,7 @@ def dins_page(report: DinsReport, *, is_fixture: bool) -> str: missing data. A recorded unknown is a published code or a marker word standing where a value would go. An empty cell is what CAL FIRE describes as an attribute that could not be determined.

+{zero_review_note(report.fields)} { field_table( report.fields, diff --git a/src/perimeter/schema.py b/src/perimeter/schema.py index e3bd6ca..d1cc44b 100644 --- a/src/perimeter/schema.py +++ b/src/perimeter/schema.py @@ -67,6 +67,67 @@ class Basis(Enum): """At least one declared value was read off the acquired file's own distribution.""" +class ZeroReading(Enum): + """What a recorded ``0`` in a field measured as a number means — or that nobody has said. + + ADR-0006: a zero is a judgment call of exactly the same kind as a marker word. Until + this existed the code could not tell the two ends of that call apart. A field whose + zeros a reviewer had read the inspection form for and left as measurements, and a + field nobody had ever looked at, both carried :attr:`Basis.NONE` and published + ``marker_basis: "none"``, which is the defect this project is about committed by the + registry that describes it: *a zero nobody has ruled on, published as a measurement.* + ``docs/MARKERS.md`` section 7 said so in its own last sentence — "the gate that reads + this file has no way to tell a reviewed decision from an unexamined one." + + Deliberately a second axis rather than more :class:`Basis` members. ``Basis`` answers + *where a declared vocabulary came from*; this answers *what a zero is*, and a field can + have any combination of the two. Folding them would be the same conflation one column + over. + + The default is :attr:`UNREVIEWED` and it must stay the default: a field added tomorrow + has not been reviewed, and the honest state is the one a reader is warned about rather + than the one they are reassured by. + """ + + UNREVIEWED = "unreviewed" + """Nobody has ruled on this field's zeros. + + A recorded ``0`` is published as a recorded value **and may be an absence**. A reader + must not take it as a measurement. `tests/test_schema.py` fails the build if a field in + this state ever publishes a recorded zero, so the state is a promise that there are + none rather than a shrug about the ones there are. + """ + + MEASUREMENT = "measurement" + """Reviewed: a recorded ``0`` here is a finding. + + An inspector writing ``0`` damaged outbuildings observed no damaged outbuildings. The + zero is a value, it stays counted as present, and ``docs/MARKERS.md`` section 7 carries + the reading and the distribution it rests on. + """ + + UNDECIDABLE = "undecidable" + """Reviewed, and the published documentation does not settle it. + + Both readings are live and the file cannot separate them, so the zero is **left + present** — declaring it a marker would move tens of thousands of records on a reading + nothing supports. A reader must not treat a ``0`` here as a measurement **or** as an + absence; ``recorded_zero_values`` is published so either reading can be applied, and + section 7 records what would settle it. + + This is the third answer issue #83 asks for by name: "undecidable from the published + documentation, which is itself a finding worth a note." + """ + + MARKER = "marker" + """Reviewed and declared: ``0`` is this field's marker for a value that was not recorded. + + The zeros leave ``recorded_zero_values`` and are counted as recorded-as-unknown, so the + zero gate stops applying to the field exactly as ADR-0006 says. ``YEARBUILT`` is the + one instance: no structure standing in a California wildfire was built in year 0. + """ + + @dataclass(frozen=True) class FieldSpec: """One measured column and the vocabulary this project has reviewed for it.""" @@ -115,8 +176,30 @@ class FieldSpec: on the page, because the weakest declaration sets what a reader can rely on. """ + zero_reading: ZeroReading = ZeroReading.UNREVIEWED + """What a recorded ``0`` in this field means, as a reviewer has ruled. See + :class:`ZeroReading`. + + Meaningful only where ``numeric`` is true; a field that is not measured as a number has + no zeros to rule on and must leave this at :attr:`ZeroReading.UNREVIEWED`. Held to + ``docs/MARKERS.md`` section 7 in both directions by ``tests/test_schema.py``, so the + reading in the registry and the reading in the audit cannot drift apart. + """ + note: str = "" + @property + def zeros_are_reviewed(self) -> bool: + """True when somebody has ruled on what a ``0`` in this field means. + + The two-number denominator's numerator. Deliberately phrased as "somebody has + ruled" and not "the zeros are values": :attr:`ZeroReading.UNDECIDABLE` is a review + that reached no verdict, and a review that reached no verdict is still a review -- + it is the state issue #83 asks for, and collapsing it into either neighbour would + lose the finding. + """ + return self.numeric and self.zero_reading is not ZeroReading.UNREVIEWED + @property def declares_vocabulary(self) -> bool: """True when this field makes any judgment call that needs a basis.""" @@ -759,10 +842,30 @@ def require_columns( } ), ), - FieldSpec("NUMBEROFUNITPERSTRUCTURE", "Units in structure", numeric=True), - FieldSpec("NOOUTBUILDINGSDAMAGED", "Damaged outbuildings", numeric=True), - FieldSpec("NOOUTBUILDINGSNOTDAMAGED", "Undamaged outbuildings", numeric=True), - FieldSpec("NOOFCARSONPROPERTY", "Damaged or destroyed cars", numeric=True), + FieldSpec( + "NUMBEROFUNITPERSTRUCTURE", + "Units in structure", + numeric=True, + zero_reading=ZeroReading.UNDECIDABLE, + ), + FieldSpec( + "NOOUTBUILDINGSDAMAGED", + "Damaged outbuildings", + numeric=True, + zero_reading=ZeroReading.MEASUREMENT, + ), + FieldSpec( + "NOOUTBUILDINGSNOTDAMAGED", + "Undamaged outbuildings", + numeric=True, + zero_reading=ZeroReading.MEASUREMENT, + ), + FieldSpec( + "NOOFCARSONPROPERTY", + "Damaged or destroyed cars", + numeric=True, + zero_reading=ZeroReading.MEASUREMENT, + ), FieldSpec( "APN", "APN (parcel)", @@ -793,6 +896,7 @@ def require_columns( unknown_markers=frozenset({"0"}), basis=Basis.INFERRED, numeric=True, + zero_reading=ZeroReading.MARKER, note="Added by the same post-collection spatial join as APN and SITEADDRESS. " "12,148 records hold the literal 0. No structure standing in a California " "wildfire was built in year 0, so a 0 here is the parcel record carrying no " @@ -802,7 +906,10 @@ def require_columns( "docs/MARKERS.md holds the distribution it rests on.", ), FieldSpec( - "ASSESSEDIMPROVEDVALUE", "Assessed improved value (parcel)", numeric=True + "ASSESSEDIMPROVEDVALUE", + "Assessed improved value (parcel)", + numeric=True, + zero_reading=ZeroReading.UNDECIDABLE, ), FieldSpec( "STREETNAME", diff --git a/src/perimeter/schema_export.py b/src/perimeter/schema_export.py index f26a5c1..67ebaa9 100644 --- a/src/perimeter/schema_export.py +++ b/src/perimeter/schema_export.py @@ -45,6 +45,7 @@ from typing import Any from perimeter.artifacts import ARTIFACT_SCHEMA_VERSION, FIELD_STATE_ORDER +from perimeter.schema import ZeroReading from perimeter.sources import DINS, FRAP, Source SCHEMA_DIALECT = "https://json-schema.org/draft/2020-12/schema" @@ -241,13 +242,29 @@ def _field_schema() -> dict[str, Any]: "than folded into either presence or absence. Absent for a field that is " "not numeric." ), + "recorded_zero_reading": { + "type": "string", + "enum": [reading.value for reading in ZeroReading], + "description": ( + "What a recorded zero in this field MEANS, as a reviewer has ruled. " + "`measurement`: a zero is a finding. `undecidable`: reviewed, and the " + "published documentation does not settle it, so the zero is left " + "present and must not be read as a finding OR as an absence. " + "`marker`: declared as a marker, so this field's zeros are counted in " + "`explicit_unknown` and `recorded_zero_values` is 0. `unreviewed`: " + "NOBODY HAS RULED -- a zero here may be an absence, and the build " + "fails if such a field ever publishes one. Read beside " + "`recorded_zero_values`, never instead of it, and see " + "docs/MARKERS.md section 7. Absent for a field that is not numeric." + ), + }, "note": _string( "The field registry's note on this field. Absent where the registry " "carries none." ), }, ) - optional = {"recorded_zero_values", "note"} + optional = {"recorded_zero_values", "recorded_zero_reading", "note"} schema["required"] = sorted(set(schema["required"]) - optional) return schema @@ -287,6 +304,48 @@ def _access_schema() -> dict[str, Any]: ) +def _zero_review_schema() -> dict[str, Any]: + """How much of the zero question has been answered, and how much has not. + + Two numbers on purpose (issue #83). "Every field publishing a zero has been reviewed" + is a true and reassuring sentence that says nothing about the fields measured as + numbers holding no zero *today* -- and a zero arriving in one of those is exactly the + case that used to be published as a measurement nobody had ruled on. + """ + return _object( + "Coverage of the zero question over this artifact's fields. Every figure is " + "derived from the fields beside it; none is stated anywhere.", + { + "fields_measured_as_numbers": _integer( + "The denominator: every field that can carry a zero, and so every field " + "this project either has or has not ruled on." + ), + "fields_with_a_reviewed_zero_reading": _integer( + "Of those, how many carry a `recorded_zero_reading` other than " + "`unreviewed`. `undecidable` counts as reviewed: a review that reached no " + "verdict is still a review, and it is a finding rather than a gap." + ), + "fields_without_one": { + "type": "array", + "items": {"type": "string"}, + "description": ( + "The rest, named rather than counted away. A zero appearing in any of " + "these fails the build until somebody rules on it." + ), + }, + "fields_publishing_a_recorded_zero": _integer( + "Of the fields measured as numbers, how many actually carry a zero in " + "this retrieval." + ), + "fields_publishing_a_recorded_zero_with_a_reviewed_reading": _integer( + "Of those, how many carry a reading. Smaller than the figure above means " + "a zero nobody has ruled on is being published as a value, which is the " + "condition `tests/test_schema.py` fails the build on." + ), + }, + ) + + def _envelope(properties: dict[str, Any], *, measurement: str) -> dict[str, Any]: """The keys both artifacts share, plus this artifact's own.""" shared = { @@ -310,6 +369,7 @@ def _envelope(properties: dict[str, Any], *, measurement: str) -> dict[str, Any] "measurement": _string("What this artifact measures, in one sentence."), "source": _source_schema(), "records": _integer("Records measured."), + "recorded_zero_review": _zero_review_schema(), } return {**shared, **properties} diff --git a/tests/test_artifact_schemas.py b/tests/test_artifact_schemas.py index 6d5c0b2..504f5b7 100644 --- a/tests/test_artifact_schemas.py +++ b/tests/test_artifact_schemas.py @@ -62,7 +62,7 @@ #: The keys `perimeter.artifacts._field_json` writes conditionally, and nothing else. #: Every other key in every object of both artifacts is required. -OPTIONAL_FIELD_KEYS = {"recorded_zero_values", "note"} +OPTIONAL_FIELD_KEYS = {"recorded_zero_values", "recorded_zero_reading", "note"} @pytest.fixture(scope="module") diff --git a/tests/test_published_site_is_current.py b/tests/test_published_site_is_current.py index 9eef313..81fd100 100644 --- a/tests/test_published_site_is_current.py +++ b/tests/test_published_site_is_current.py @@ -150,7 +150,22 @@ def spine(payload: Mapping[str, Any]) -> dict[str, Any]: "measurement": payload["measurement"], "source_keys": sorted(payload["source"]), "fields": [ - [field["name"], field["label"], field["marker_basis"], sorted(field)] + [ + field["name"], + field["label"], + field["marker_basis"], + # The zero reading is a registry decision, not a measurement, so a fixture + # build and a real one must agree on it exactly as they do on the basis + # beside it. `sorted(field)` alone catches a field that stops carrying the + # key and misses the case that matters: a ruling silently changed or + # withdrawn in `schema.py` while the published artifact keeps the old word. + # Measured 2026-09-13: removing `NOOFCARSONPROPERTY`'s ruling left all 534 + # tests in tests/test_schema.py green, because every gate there reads the + # committed artifact and `make site-check` -- the only thing that rebuilds + # it -- needs data/raw/ and cannot run in CI. + field.get("recorded_zero_reading"), + sorted(field), + ] for field in payload["fields"] ], } diff --git a/tests/test_schema.py b/tests/test_schema.py index 8f7684f..03faa20 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -11,6 +11,8 @@ import pytest from perimeter.cells import Cell, CellState, SentinelDriftError +from perimeter.coverage import FieldCoverage, field_coverage, zero_review +from perimeter.records import Record from perimeter.schema import ( DINS_FIELDS, DINS_FIELDS_BY_NAME, @@ -19,6 +21,7 @@ Basis, FieldSpec, SchemaDriftError, + ZeroReading, require_columns, ) @@ -696,3 +699,264 @@ def test_the_gate_refuses_a_count_it_cannot_read() -> None: assert problems[0].startswith( "section 7 says 'Several' fields publish recorded zeros" ) + + +# -------------------------------------------------------------------------------------- +# A zero nobody ruled on is not a measurement (ADR-0006, issue #83) +# -------------------------------------------------------------------------------------- + +#: The last column of section 7's table, where the registry's declared reading is repeated +#: for a human. Matched as a backticked token so "reviewed" in prose cannot satisfy it. +_ZERO_READING_IN_ROW = re.compile( + r"^\|\s*`(?P[A-Z0-9_]+)`\s*\|[^|]*\|[^|]*\|(?P[^|]*)\|" +) + + +def _section_seven_readings(doc: str) -> dict[str, str]: + """Field name to the reading word section 7's table states for it.""" + section = _zero_section(doc) + found: dict[str, str] = {} + for line in section.splitlines(): + match = _ZERO_READING_IN_ROW.match(line) + if match is None: + continue + words = [ + reading.value + for reading in ZeroReading + if f"`{reading.value}`" in match["declared"] + ] + if len(words) == 1: + found[match["name"]] = words[0] + return found + + +@pytest.mark.parametrize("spec", ALL_FIELDS, ids=lambda s: s.name) +def test_only_a_field_measured_as_a_number_rules_on_its_zeros(spec: FieldSpec) -> None: + """A field with no zeros to rule on has not ruled on any. + + Without this, a free-text field could be marked `measurement` and would enter the + reviewed numerator of a question that does not apply to it -- a coverage figure made + to look better by widening it with rows nobody had to examine. + """ + if spec.numeric: + return + assert spec.zero_reading is ZeroReading.UNREVIEWED, ( + f"{spec.name} is not measured as a number, so it has no zeros to rule on, but it " + f"declares zero_reading={spec.zero_reading.value!r}" + ) + + +@pytest.mark.parametrize("spec", ALL_FIELDS, ids=lambda s: s.name) +def test_a_marker_reading_and_a_declared_zero_marker_are_the_same_fact( + spec: FieldSpec, +) -> None: + """`marker` in the registry and `0` in the marker set must travel together. + + Either half alone is a lie in one direction: the word without the declaration says the + zeros are counted as unknown when they are still counted as values, and the + declaration without the word leaves the audit unable to see why the field publishes + none. + """ + declared = "0" in spec.unknown_markers or "0" in spec.unknown_codes + says_marker = spec.zero_reading is ZeroReading.MARKER + assert declared == says_marker, ( + f"{spec.name}: zero_reading is {spec.zero_reading.value!r} but `0` is " + f"{'' if declared else 'not '}declared as a marker for it" + ) + + +def unruled_published_zeros(payload: dict[str, Any]) -> list[tuple[str, int]]: + """Fields publishing a recorded zero while saying nobody has ruled on what one means. + + THE gate for issue #83. `numeric_zeros_missing_from_the_audit` asks whether the field + is NAMED in docs/MARKERS.md, which a field can be while the document says nothing + about its zeros; and until `zero_reading` existed nothing could ask the stronger + question at all, because a reviewed decision and an unexamined field both published + `marker_basis: "none"`. + """ + return [ + (field["name"], field["recorded_zero_values"]) + for field in payload["fields"] + if field.get("recorded_zero_values") + and field.get("recorded_zero_reading") == ZeroReading.UNREVIEWED.value + ] + + +@pytest.mark.parametrize("name", NUMERIC_ARTIFACTS) +def test_no_published_zero_is_one_nobody_ruled_on(name: str) -> None: + unruled = unruled_published_zeros(_published(name)) + assert unruled == [], ( + f"{name}: these fields publish a recorded zero while declaring that nobody has " + f"ruled on what a zero there means, so an absence may be being published as a " + f"measurement: {unruled}. Read the field's form and set `zero_reading` on it in " + f"src/perimeter/schema.py, then write the reading up in docs/MARKERS.md §7." + ) + + +def test_the_unruled_zero_gate_catches_the_shape_issue_83_found() -> None: + """The five fields as they stood when #83 was filed, run back through the gate. + + A positive control: the gate above passes on the committed artifacts, and a passing + gate is worth nothing until it has been shown to fail on the thing it is for. + """ + before = { + "fields": [ + { + "name": "NOOFCARSONPROPERTY", + "recorded_zero_values": 55831, + "recorded_zero_reading": "unreviewed", + }, + { + "name": "NUMBEROFUNITPERSTRUCTURE", + "recorded_zero_values": 58411, + "recorded_zero_reading": "unreviewed", + }, + { + "name": "YEARBUILT", + "recorded_zero_values": 0, + "recorded_zero_reading": "marker", + }, + {"name": "CITY", "markers": {"NA": 1}}, + ] + } + assert unruled_published_zeros(before) == [ + ("NOOFCARSONPROPERTY", 55831), + ("NUMBEROFUNITPERSTRUCTURE", 58411), + ] + + +@pytest.mark.parametrize("spec", ALL_FIELDS, ids=lambda s: s.name) +def test_every_ruling_on_a_zero_is_written_up_in_the_audit(spec: FieldSpec) -> None: + """A reading nobody wrote up is a judgment call nobody can inspect. + + The sibling of `test_every_judgment_call_is_written_up_in_the_audit`, on the other + axis. `Basis.NONE` and `ZeroReading.UNREVIEWED` are different absences and a field can + be in one without being in the other -- which is the whole point of the second axis. + """ + if spec.zero_reading is ZeroReading.UNREVIEWED: + return + assert f"`{spec.name}`" in MARKERS_DOC, ( + f"{spec.name} declares zero_reading={spec.zero_reading.value!r} and " + f"docs/MARKERS.md never names it" + ) + + +def test_section_seven_states_the_reading_the_registry_carries() -> None: + """The document and the registry, held to each other by an exact token. + + Both directions. A row whose word drifts from the registry is caught, and so is a row + that stops carrying a word at all -- because a row with no word parses as absent here + and the field it names is in the registry with one. + """ + stated = _section_seven_readings(MARKERS_DOC) + published = { + field["name"]: field.get("recorded_zero_reading") + for name in NUMERIC_ARTIFACTS + for field in _published(name)["fields"] + if field.get("recorded_zero_values") + } + assert stated, ( + "section 7's table no longer states a reading word for any field, so this gate " + "checks nothing" + ) + assert stated == published, ( + f"docs/MARKERS.md section 7 and the artifacts disagree about which reading each " + f"field's zeros carry: doc={stated}, artifacts={published}" + ) + + +def _coverage_of(spec: FieldSpec, values: Sequence[object]) -> FieldCoverage: + """One field's coverage over a handful of made-up cells.""" + records = [ + Record(identifier=str(index), cells={spec.name: spec.classify(raw, where="t")}) + for index, raw in enumerate(values) + ] + return field_coverage(records, spec) + + +def test_the_two_numbers_are_counted_separately() -> None: + """`reviewed` counts rulings; `examinable` counts questions. Never the same walk. + + A control, not a restatement of :func:`zero_review`. Every other gate on this block + reads the committed artifact, where the two numbers happen to be 6 and 8 because six + fields have been ruled on -- so a build that counted every examinable field as + reviewed would publish `8 of 8` and every one of those gates would still pass, because + each of them recomputes the block from the same payload the block was written into. + Measured 2026-09-13: `reviewed=len(numeric)` in `zero_review` left all 1,158 tests + green. This is the one that reads the function's own answer over a field set whose two + numbers are known to differ. + """ + ruled = FieldSpec( + "RULED", "Ruled", numeric=True, zero_reading=ZeroReading.MEASUREMENT + ) + unruled = FieldSpec("UNRULED", "Unruled", numeric=True) + text = FieldSpec("TEXT", "Not a number") + review = zero_review( + [ + _coverage_of(ruled, ["0", "1"]), + _coverage_of(unruled, ["0", "2"]), + _coverage_of(text, ["a", "b"]), + ] + ) + assert review.examinable == 2, "the free-text field is not a zero question" + assert review.reviewed == 1 + assert review.unreviewed_fields == ("UNRULED",) + assert review.unreviewed == 1 + assert review.publishing_zeros == 2 + assert review.publishing_zeros_reviewed == 1 + + +def test_a_field_with_no_zeros_still_counts_as_a_question() -> None: + """The reason `examinable` is not "fields publishing a zero". + + `LATITUDE` and `LONGITUDE` hold no zeros in this retrieval and nobody has ruled on + them. A denominator of "fields publishing a zero today" would report 5 of 5 and be + true, and the first zero to arrive in either would be published as a measurement + nobody had ruled on -- the exact shape issue #83 reported. + """ + unruled = FieldSpec("NOZEROS", "No zeros here", numeric=True) + review = zero_review([_coverage_of(unruled, ["1", "2"])]) + assert review.examinable == 1 + assert review.reviewed == 0 + assert review.publishing_zeros == 0 + assert review.publishing_zeros_reviewed == 0 + + +@pytest.mark.parametrize("name", NUMERIC_ARTIFACTS) +def test_the_published_zero_review_re_derives_from_the_fields_beside_it( + name: str, +) -> None: + """The two numbers, recomputed from the rows they describe. + + A summary block is exactly the shape that goes stale silently: it is small, it reads + as a headline, and nothing else in the artifact contradicts it. Recomputing it here + from the same document means the published figure cannot be one the fields do not + support. + """ + payload = _published(name) + numeric = [field for field in payload["fields"] if "recorded_zero_reading" in field] + review = payload["recorded_zero_review"] + assert review["fields_measured_as_numbers"] == len(numeric) + assert review["fields_with_a_reviewed_zero_reading"] == sum( + 1 + for field in numeric + if field["recorded_zero_reading"] != ZeroReading.UNREVIEWED.value + ) + assert review["fields_without_one"] == [ + field["name"] + for field in numeric + if field["recorded_zero_reading"] == ZeroReading.UNREVIEWED.value + ] + with_zeros = [field for field in numeric if field["recorded_zero_values"]] + assert review["fields_publishing_a_recorded_zero"] == len(with_zeros) + assert review["fields_publishing_a_recorded_zero_with_a_reviewed_reading"] == sum( + 1 + for field in with_zeros + if field["recorded_zero_reading"] != ZeroReading.UNREVIEWED.value + ) + # The denominator and the numerator are not the same number, and a build where they + # are would make the gap this issue is about invisible again. + assert ( + review["fields_with_a_reviewed_zero_reading"] + <= review["fields_measured_as_numbers"] + )