A refresh began from a crash, one refusal at a time

CHANGELOG.md

@@ -6,6 +6,39 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this project a
 
 ## [Unreleased]
 
+### Added, a survey command that inventories a retrieval's candidate markers
+
+- **`python -m perimeter.survey` (`make survey`).** A refresh began from a crash. The
+  build refuses a cell holding something that reads like a missing-data marker in a field
+  that has not declared it, which is right, and which reports exactly one cell: a new
+  retrieval was worked through one refusal at a time. The survey reads the same files the
+  build reads and reports every candidate at once, per measured field, with counts.
+
+  It **declares nothing**. It writes no schema, edits no registry, and no part of the
+  build reads its output. Every candidate carries `basis: unreviewed`, in `survey.json`
+  and in the `survey.md` draft written in `docs/MARKERS.md`'s shape, so it cannot be
+  mistaken for a decision somebody made. The intended sequence is survey, review, declare,
+  build, diff, and the reviewing step is a person's.
+
+  **The survey and the gate read one list.** The sentinel decision moved out of
+  `FieldSpec.classify` into `FieldSpec.undeclared_marker`, which both now call. A survey
+  carrying its own copy of the rule would go quiet at exactly the moment the rule moved,
+  and a test holds the two readers to the same verdict over every value in the sentinel
+  vocabulary, for every field in both registries.
+
+  Three distinctions it keeps, being the same three the artifacts keep. A field above the
+  listing bound publishes its distinct count and says the listing was withheld, never an
+  empty list. A field with no published domain publishes `null` for values outside one,
+  never `0`, per ADR-0010. A numeric field in which nothing parsed as a number has no zero
+  share, not a zero one.
+
+  A zero the registry has already ruled on is reported and not flagged. `YEARBUILT`
+  declares the literal `0` for the 12,148 parcel records carrying no year, and re-asking
+  that question on every run is how a review aid stops being read.
+
+- **`SentinelDriftError` now names the survey command** in its message, so the first
+  refusal of a refresh points at the tool that lists the rest.
+
 ### Fixed, a distribution name that could never have been released
 
 - **`pyproject.toml` declared `name = "perimeter"`, which is somebody else's package.**

Makefile

@@ -183,3 +183,26 @@ determinism:
 # Network. Run by hand, never from a build. See PROVENANCE.md.
 acquire:
 	uv run python -m perimeter.acquire --out data/raw
+
+# Inventory the candidate markers in an acquired retrieval, before a build refuses one.
+#
+# Offline, and deliberately NOT a prerequisite of anything. It declares nothing, writes no
+# schema, and no part of the build reads its output: it is a review aid for the person
+# doing a refresh, and a target that ran it automatically would turn a list of open
+# questions into something a build appears to have answered.
+#
+# It is not part of `verify` for the same reason `acquire` is not: it reads data/raw/,
+# which is never in git and never in CI. Point it at the fixtures to see the shape:
+#
+#   make survey SURVEY_DINS=fixtures/dins_postfire.sample.json \
+#               SURVEY_PERIMETERS=fixtures/frap_perimeters.sample.json
+#
+SURVEY_DINS ?= data/raw/dins_postfire.json
+SURVEY_PERIMETERS ?= data/raw/frap_perimeters.json
+SURVEY_OUT ?= build/survey
+
+survey:
+	uv run python -m perimeter.survey \
+		--perimeters $(SURVEY_PERIMETERS) \
+		--dins $(SURVEY_DINS) \
+		--out $(SURVEY_OUT)

README.md

@@ -163,6 +163,42 @@ make site-offline    # build from committed fixtures; runs anywhere, no network
 make diff OLD=a.json NEW=b.json   # compare two coverage artifacts leaf by leaf
 ```
 
+### Before a refresh: what the new file holds
+
+A refresh used to begin from a crash. The build refuses a cell holding something that
+reads like a missing-data marker in a field nobody has reviewed for it, which is the right
+behaviour and a poor way to find out: it reports one cell, in one field, and says nothing
+about the next one, so a new retrieval gets worked through one refusal at a time.
+
+`make survey` reads the same files the build reads and reports every candidate at once:
+
+```sh
+make survey                                   # over data/raw/, after make acquire
+make survey SURVEY_DINS=fixtures/dins_postfire.sample.json \
+            SURVEY_PERIMETERS=fixtures/frap_perimeters.sample.json
+```
+
+Per measured field it writes the distinct values with counts, the values that read like a
+missing-data marker the field has not declared, the values outside a published domain, and
+for a numeric field how many of its records hold a zero. Output is `survey.json` and a
+`survey.md` draft in `docs/MARKERS.md`'s shape, where every entry reads `basis:
+unreviewed`.
+
+It declares nothing. It writes no schema, edits no registry, and no part of the build
+reads it. The sequence is survey, review, declare, build, diff, and the reviewing step is
+a person's.
+
+Three things it will not round off, because they are the same three the artifacts refuse
+to round off:
+
+- A field with more distinct values than the listing bound reports **the bound and its
+  distinct count**, never an empty list. A reader must be able to tell "this field holds
+  nothing" from "this field holds too much to show".
+- A field with **no published domain** reports `null` for values outside one, not `0`.
+  There was no comparison to have a result.
+- A numeric field in which nothing parsed as a number has **no zero share**, not a zero
+  one.
+
 ### What a refresh moved
 
 The figures on these pages move only when the pinned retrievals are deliberately
@@ -288,6 +324,7 @@ here is how much of each published field is actually filled in, and what the bla
 | `src/perimeter/artifacts.py` | Deterministic JSON |
 | `src/perimeter/render.py` | The static pages |
 | `src/perimeter/acquire.py` | The only code that touches the network. Run by hand, never in CI |
+| `src/perimeter/survey.py` | Inventory a retrieval's candidate markers before a build refuses one. Declares nothing |
 | `tools/a11y.mjs` | axe-core over the built pages in a headless DOM; an undecided rule is not a pass |
 | `tools/determinism.sh` | Compare two build trees; refuse an empty or missing one |
 | `site/` | The built pages and their JSON artifacts. Tracked in git, and published by `.github/workflows/pages.yml` |

src/perimeter/schema.py

@@ -124,6 +124,29 @@ def declares_vocabulary(self) -> bool:
             self.unknown_codes or self.unknown_markers or self.recorded_absences
         )
 
+    def undeclared_marker(self, text: str) -> str | None:
+        """The normalized marker ``text`` would stop the build with, or ``None``.
+
+        This is the sentinel decision itself, lifted out of :meth:`classify` so that the
+        gate and :mod:`perimeter.survey` read one list rather than two that can drift.
+        The survey exists to find these before a build refuses one, and a survey working
+        from its own copy of the rule would go quiet at exactly the moment the rule moved.
+
+        ``text`` is expected already stripped, as :meth:`classify` hands it over. The
+        order below is the order that matters: a published domain code and a published
+        recorded absence are checked first, so a field that legitimately publishes
+        ``None`` or ``N/A`` as a finding keeps it as a finding and is not reported as a
+        candidate.
+        """
+        if text in self.unknown_codes or text in self.recorded_absences:
+            return None
+        marker = normalize_marker(text)
+        if marker in self.unknown_markers:
+            return None
+        if marker in SUSPECTED_SENTINELS:
+            return marker
+        return None
+
     def classify(self, raw: object, *, where: str) -> Cell:
         """Sort one cell into present, explicit-unknown, or not-recorded.
 
@@ -143,13 +166,16 @@ def classify(self, raw: object, *, where: str) -> Cell:
         marker = normalize_marker(text)
         if marker in self.unknown_markers:
             return Cell.explicit_unknown(marker)
-        if marker in SUSPECTED_SENTINELS:
+        if self.undeclared_marker(text) is not None:
             raise SentinelDriftError(
                 f"{where}.{self.name}: cell {text!r} reads as a missing-data marker but "
                 f"is not one this project has reviewed for this field "
                 f"(reviewed markers: {sorted(self.unknown_markers | self.unknown_codes)}; "
                 f"published absences: {sorted(self.recorded_absences)}); "
-                "guessing here would publish an absence as a value"
+                "guessing here would publish an absence as a value. "
+                "`python -m perimeter.survey` inventories every candidate in a "
+                "retrieval at once, so a refresh does not have to be driven one "
+                "refusal at a time"
             )
         return self._present(text, where=where)