Skip to content

docs(review): falsification + shared-field-semantics guards in pre-PR checklist - #1

Merged
alonbilu merged 2 commits into
mainfrom
chore/review-falsification-and-shared-field-guards
Jul 1, 2026
Merged

docs(review): falsification + shared-field-semantics guards in pre-PR checklist#1
alonbilu merged 2 commits into
mainfrom
chore/review-falsification-and-shared-field-guards

Conversation

@alonbilu

@alonbilu alonbilu commented Jul 1, 2026

Copy link
Copy Markdown
Owner

Two guards for the /review checklist, distilled from a real miss on a downstream project (a validator rule read an LLM-echoed field as advisory while sibling rules read the same field as committed — every isolated test passed, the two were contradictory together; it survived several spec revisions before review caught it).

Added under Testing:

  1. Falsify the acceptance test — confirm a new rule/acceptance test goes RED without the change under test. A test that passes because it never exercised the case is worse than none; fixtures shaped to match the implementation (single-mode / edge-case seed data) are the tell.
  2. Shared-field semantics — when a new check reads a field another rule/consumer already reads, verify they agree on what it means. Divergent interpretations pass isolation yet contradict each other. Grep the field name across rules/consumers.

Guard (2) is the one that would actually have caught the miss — falsification alone wouldn't have, because the original (wrong) test and the bug agreed.

🤖 Generated with Claude Code

Alon Bilu and others added 2 commits July 1, 2026 21:39
…e-PR checklist

Two guards distilled from a real miss (a validator rule read an LLM-echoed field as advisory
while sibling rules read it as committed — passed every isolated test, contradictory together):
1. Falsify a new rule/acceptance test — confirm it goes red without the change (catches vacuity).
2. When a new check reads a field another rule already consumes, verify they agree on its meaning.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaced by adding the review guards: each seeded repo has its own copy of review.md et al.,
which silently drifts behind the canonical starter — same failure class as a canonical value
living in two places. Deferred; needs a decision before the repo count grows.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alonbilu
alonbilu force-pushed the chore/review-falsification-and-shared-field-guards branch from 3d831a0 to 8779350 Compare July 1, 2026 21:39
@alonbilu

alonbilu commented Jul 1, 2026

Copy link
Copy Markdown
Owner Author

Updated (force-push):

  • Generalized the shared-field guard so it reads as a pattern for any shared field (denormalized copy, user-supplied value, upstream/LLM echo, cached flag) with the committed-vs-advisory case as one illustration — not a domain-specific footnote a reader pattern-matches too narrowly.
  • Added a backlog note (docs/FUTURE-ROADMAP.md, commit 8779350): adding these guards surfaces a meta-question — each seeded repo vendors its own copy of review.md et al., which silently drifts behind the canonical starter (the same canonical-value-in-two-places failure class the guard itself is about). Deferred; flagged to decide before the repo count grows (vendor-per-repo + drift check, or reference a single source).

@alonbilu
alonbilu merged commit 7785272 into main Jul 1, 2026
0 of 3 checks passed
@alonbilu
alonbilu deleted the chore/review-falsification-and-shared-field-guards branch July 1, 2026 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant