A refresh began from a crash, one refusal at a time

[ChelseaKR]

What was wrong

A refresh began from a crash. SentinelDriftError fails the build the first time a cell holds something that reads like a missing-data marker in a field nobody has reviewed for it. That is the right behaviour and a poor way to find out: it reports one cell, in one field, and says nothing about the next one. Working through a new retrieval meant running the build, reading a refusal, reviewing a field, and running it again, once per undeclared marker in the file.

What changed

python -m perimeter.survey (make survey) reads the same files the build reads and reports every candidate at once. Per measured field: the distinct values with counts, the values that read like a missing-data marker the field has not declared, the values outside a published domain, and for a numeric field how many records hold a zero. Output is survey.json and a survey.md draft in docs/MARKERS.md's shape.

It declares nothing. No schema, no registry edit, and no part of the build reads its output. Every candidate carries basis: unreviewed, in both files and in the artifact's own header text, so it cannot be mistaken for a decision somebody made. The sequence is survey, review, declare, build, diff, and the reviewing step is a person's.

The survey and the gate read one list. The sentinel decision moved out of FieldSpec.classify into FieldSpec.undeclared_marker, which both callers now use. A survey carrying its own copy of the rule would go quiet at exactly the moment the rule moved. SentinelDriftError also now names the survey command, so the first refusal of a refresh points at the tool that lists the rest.

Three distinctions carried over from the artifacts, because this module is a third place they can be lost:

• A field above the listing bound reports its distinct count and says the listing was withheld. Never an empty list, which would read as a field holding nothing.
• A field with no published domain reports null for values outside one, never 0. ADR-0010.
• A numeric field in which nothing parsed as a number has no zero share, not a zero one.

A zero the registry has already ruled on is reported and not flagged. YEARBUILT declares the literal 0, and re-asking that question every run is how a review aid stops being read.

What it found on the pinned retrieval

Run over data/raw/ at the 2026-08-07 pin (132,522 DINS records, 23,334 FRAP records, 4.6s):

No undeclared marker candidates in either file. The registry covers every marker in the pinned retrieval, which is what a green build already implied and is now measured rather than inferred.

Five numeric fields whose zeros nobody has ruled on, four of them severe:

Field	Zeros	Of numbers	Share	Declared basis
NOOFCARSONPROPERTY	55,831	56,188	99.4%	none
NOOUTBUILDINGSDAMAGED	59,628	61,939	96.3%	none
NOOUTBUILDINGSNOTDAMAGED	58,888	61,958	95.0%	none
NUMBEROFUNITPERSTRUCTURE	58,411	61,929	94.3%	none
ASSESSEDIMPROVEDVALUE	6,613	126,482	5.2%	none

This is ADR-0006's shape exactly, and it is the same shape as YEARBUILT: 58,411 structures recorded as containing zero units is not a plausible reading of a count. Nothing here declares any of them and no artifact or page moves in this pull request. They are a reviewer's call and are written up as a separate issue.

One check on the arithmetic worth stating: the survey independently reproduces YEARBUILT's 12,148 zeros, the figure section 7 of docs/MARKERS.md already records, and correctly reports it as declared rather than open.

How it was verified

make lint, make format, mypy --strict: green. uv run pytest: 978 passed, branch coverage 98.80% overall and 99% on survey.py.

Two negative controls, both fired.

1. The survey and the gate read one list. Baseline git hash-object src/perimeter/schema.py = e3bd6ca…. Reordered classify so the sentinel net ran before the recorded-absence check, which is how the two readers realistically come apart; re-hashed and asserted the hash moved to 7e932c7…. Restored and asserted the hash returned to e3bd6ca… exactly, with __pycache__ cleared between every run.

   The first run of this control passed, and that first run is the honest measurement. The planted values were only SUSPECTED_SENTINELS, which is lowercase, while recorded_absences holds the publisher's own casing (None for a street type, N/A for a propane-tank distance). Nothing in the fixture reached the branch where the order decides anything: the fixture sat where the failure was impossible. Widening it to plant each field's own declared values in their published casing made the control fire, on PROPANETANKDISTANCE and N/A. That widening is its own commit.

2. Absence is not a value. Baseline git hash-object src/perimeter/survey.py = b032766…. Changed the no-published-domain branch from None to (); hash moved to cc030db…; test_no_published_domain_is_null_and_not_an_empty_list went red naming SITEADDRESS and assert () is None. Restored, hash back to b032766… exactly.

The determinism test runs across processes, not twice in one interpreter, under PYTHONHASHSEED values 0, 1 and 524287, over the full ten-record DINS fixture rather than one crafted row. A companion test pins that the fixture holds at least eight fields with more than one distinct value, so the determinism test cannot quietly become vacuous: one row is always in order.

Closes #66

Prepared with AI assistance; reviewed before submission.