Skip to content

feat(round): check WHICH solver produced a PoC, instead of trusting the backend that ran it - #238

Merged
ai-hpc merged 2 commits into
mainfrom
ai-hpc/check-the-solver-pin
Sep 10, 2026
Merged

ai-hpc merged 2 commits into
mainfrom
ai-hpc/check-the-solver-pin

Conversation

@ai-hpc

@ai-hpc ai-hpc commented Sep 10, 2026

Copy link
Copy Markdown
Member

Pairs with cybergym-backend #42, which publishes the pin and puts the receipt on the submission wire. Together they finish our side of cathedral-sandbox#142 item 4.

Why

The backend runs every miner's agent itself, so "only the approved solver was credited" is a claim the backend makes about itself, and nothing downstream could check it. That matters more here than usual: our corpus is public OSS-Fuzz bugs with published reference PoCs, so an enclave run of a lookup table is exactly as genuinely attested as an enclave run of an agent that derived the crash. The measurement is the only thing separating them — and a validator that doesn't check it is trusting the party it exists to check.

What

solver_refusal holds each submission to the pin the backend publishes at /v2/solver:

check refused when
workload_sha256 absent, or not the approved measurement
intel_verified the hardware quote did not verify
report_data_match the receipt does not bind this dispatch
payload_bound present and false — the enclave did not read what the backend sent

Fails closed on every missing piece, with one deliberate exception: payload_bound absent is not refused. An older backend omits the key, and refusing every miner over a field that does not exist would be this validator's fault, not theirs. Present-and-false is a real failure and is refused.

A refused submission scores zero without being benchmarked. Its PoCs may well crash the target; a crash nobody can attribute to the approved solver is what this lane exists not to pay for, and rebuilding a corpus for it also spends round time we don't have.

Refusal is a verdict, not an abstention. evaluated=False means "I could not judge this" and the backend excludes it from the average. A refusal was judged and must count. The two are one field apart and a payout hangs on the difference — so it is tested explicitly.

Turning it on

CYBERGYM_REQUIRE_APPROVED_SOLVER=1. Off by default, with a startup warning naming what is not being checked. Demanding the pin when the backend publishes none raises rather than returning None — silently not checking would leave a validator believing it enforces something it does not. It equally must not zero the field for that: the existing benchmark-failure path records the error, does not advance reported_round, and retries, so the validator abstains. A backend that publishes nothing is not the miners' fault.

Verification

151 passed across the v2 round suite (adds 22): every refusal reason, the case-insensitive compare, the older-backend exemption, verdict-vs-abstention, the never-benchmarked property, off-by-default, pin resolution including the malformed and unpublished cases, and the daemon path end to end — including that no verdicts are posted when the pin is demanded but missing.

…he backend that ran it

The backend runs every miner's agent itself, so "only the approved solver was credited" is a claim
it makes about itself and nothing downstream could check. That matters here more than usual: the
corpus is public OSS-Fuzz bugs with published reference PoCs, so an enclave run of a lookup table
is exactly as genuinely attested as an enclave run of an agent that derived the crash. The
measurement is the only thing separating them.

Submissions now carry the receipt (backend #42), so `solver_refusal` holds each one to the pin the
backend PUBLISHES at /v2/solver: the right workload_sha256, a hardware quote that verified, a
report that binds this dispatch, and — when the backend reports on it — an enclave that read the
payload it was sent. Every missing piece fails closed. A refused submission scores zero WITHOUT
being benchmarked: its PoCs may well crash the target, but a crash nobody can attribute to the
approved solver is what this lane exists not to pay for, and rebuilding a corpus for it wastes
round time.

Refusal is a VERDICT, not an abstention. `evaluated=False` means "I could not judge this" and the
backend excludes it from the average; a refusal was judged and must count. The two are one field
apart and a payout hangs on the difference.

CYBERGYM_REQUIRE_APPROVED_SOLVER=1 turns it on, off by default with a warning that says what is
not being checked. Demanding the pin when the backend publishes none RAISES rather than returning
None: silently not checking would leave a validator believing it enforces something it does not.
It equally must not zero the whole field for that — the existing benchmark-failure path records it
and retries, so the validator abstains. A backend that publishes nothing is not the miners' fault.
…s formatter

CI caught F821 on the fetch_solver annotation; ruff format then rewrote four files. Verified the
refusal logic survived the reformat rather than assuming it did.
@ai-hpc

ai-hpc commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

Self-review before merge (owner-directed — jared asked for the pin to be publishable and checkable; posting the pass here for post-merge audit).

Risk-direction first, because this is the first change that can zero a miner who solved tasks:

  • Who can lose money, and for whose fault. A refusal costs the miner their whole round. It fires only on evidence the backend publishes about a run the backend performed — so I checked each reason for "could this be our fault, not theirs": missing receipt, wrong measurement, failed quote, unbound report → all mean the run cannot be attributed, which is the miner's requirement to meet. payload_bound absent is the one case that would punish miners for a backend that doesn't implement the field, and it is explicitly exempted (present-and-false is refused). Tested both ways.
  • The mass-zero scenario. If the operator sets CYBERGYM_REQUIRE_APPROVED_SOLVER=1 against a backend that publishes no pin, the naive behaviour is every miner refused. That is the worst outcome in this diff, so it raises before any submission is judged, nothing is posted, and the existing benchmark-failure path (fix(round): three failures that each cost a whole round's weights without looking broken #237) records it and retries — the validator abstains. There is a test asserting client.posted is None for exactly that path.
  • Verdict vs abstention. evaluated=False is excluded from the backend's average; a refusal must count. One field apart, a payout hangs on it, so it is asserted directly rather than implied.
  • Case and format. The comparison lowercases both sides; a malformed published pin (sha256: prefix, wrong length, non-hex) is refused at resolution rather than silently never matching.
  • Off by default, with a startup warning that names what is not being checked — an upgrade cannot start refusing miners without an operator deciding to.
  • Cost, not just correctness: refused submissions never enter the benchmark plan, so a rejected miner doesn't spend a corpus rebuild of this validator's round time.
  • After ruff format rewrote four files I re-verified the refusal logic was still intact rather than assuming, then re-ran the suite (151 pass).

Residual, stated: this checks which program ran, not that the program could not look the answer up. That needs the egress-restricted enclave (cathedral-sandbox#142 item 3), still blocked on the mint.

@ai-hpc
ai-hpc merged commit 7ef2312 into main Sep 10, 2026
9 checks passed
@ai-hpc
ai-hpc deleted the ai-hpc/check-the-solver-pin branch September 10, 2026 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant