Skip to content

Review the interface a change publishes, not just its implementation - #45

Open
bockets wants to merge 1 commit into
mainfrom
review-prompt-interface-contract-rules
Open

Review the interface a change publishes, not just its implementation#45
bockets wants to merge 1 commit into
mainfrom
review-prompt-interface-contract-rules

Conversation

@bockets

@bockets bockets commented Aug 31, 2026

Copy link
Copy Markdown
Member

Adds prompts/_shared/interface-contract-rules.md, injected into the Codex first pass and both synthesize arms.

Why

Three defects shipped through review on BiggerPockets/biggerpockets#30718 and were found only by driving the server by hand against a review app afterwards.

  • An MCP tool required a market criterion that no tool enumerated. The browser funnel resolves it from a landing-page widget rather than asking it as a question, so a caller following the tools' own documentation could not produce a runnable call — the documented path failed 100% of the time, and reported "Something went wrong".
  • run_finder_search told callers to pass contact criteria "exactly as describe_finder_questions enumerates them". For the lender finder, that tool enumerated none of them.
  • Tool arguments arrive from the MCP SDK symbolized; the finder reads them by string. Every answer a client sent read as no answer at all. Unit specs were green because they built the argument hash themselves and picked the key type the implementation expected.

All three were invisible in the diff for the same reason: the implementation was self-consistent, and what was wrong was the contract it offered a caller who cannot see it. Nothing in the existing rule blocks points a reviewer at that. Completeness rules ask whether the work was finished; these ask whether the published interface is usable by someone holding only the interface.

What it asks

Four things checkable from a diff:

  1. Every required input is obtainable through the interface itself — with a specific nudge toward values the existing UI supplies out of band, which are the ones a browser-shaped author is least likely to notice a non-browser caller cannot produce, and a nudge to actually open the operation a cross-reference names.
  2. A missing or unusable input is named in the error, not rescued into a generic failure.
  3. A value crossing a serialization boundary is read the way it arrives, and is tested through that boundary — flagging the absence of such a test as a real gap, since a test that builds the payload itself structurally cannot catch a key-type mismatch.
  4. A whitelist filter is judged by what it drops today, not by the comment beside it.

Scope

Included from all three prompts so Codex and the synthesize arms can't drift. Editing a _shared block bumps the derived prompt version for every consuming repo, which is the intended Roll.

I ran scripts/resolve-prompts.sh locally: both prompts resolve, no markers remain, and both templates are include-expanded.

Three defects shipped through review on BiggerPockets/biggerpockets#30718 and were
found only by driving the server by hand afterwards. All three were invisible in the
diff for the same reason: the implementation was self-consistent, and what was wrong
was the contract it offered a caller who cannot see it.

An MCP tool required a criterion no tool enumerated, because the browser funnel
supplies it from a landing-page widget rather than a question. One tool told callers
to pass values "exactly as" another enumerates them, and that other tool enumerated
none of them. And a payload written with symbol keys was read with string keys, so
every answer a client sent read as no answer at all, with the unit specs green
because they built the hash themselves.

The new shared block asks four things a reviewer can check from a diff: that every
required input is obtainable through the interface, that a missing one is named
rather than rescued into a generic failure, that a value crossing a serialization
boundary is read the way it arrives and is tested through that boundary, and that a
whitelist filter is judged by what it drops today rather than by the comment beside
it.

Included by the Codex first pass and both synthesize arms, so the two cannot drift.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant