fix(round): score against the published task set + the validator's deployable side - #231
Conversation
…er's own submission Found by the v2 end-to-end dry run. A miner whose agent tripped its run budget after two tasks submitted only those two, both solved, and scored 100 — taking the king slot (0.84 of the lane) from a miner that attempted all six and solved five. The denominator was `len(submission.tasks)`, which the miner controls, so the winning move was to withhold your failures. The score is now out of the round's authoritative task set, published by the server and identical for every miner. That closes it from both directions: a task the miner never submitted counts unsolved, and a task it invented that is not in the set is ignored rather than padding the numerator. `RoundClient` gains `fetch_round_tasks`, and `benchmark_and_report` refuses to score a round that published no task set rather than falling back to a miner-supplied denominator. Abstention is unaffected: a validator that ran out of time still reports unevaluated rather than turning "I did not finish" into a real zero. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fferential, off-chain daemon Fills in the three seams the merged round loop left injected, so a validator can actually run: * `cybergym_round_client` — the concrete `RoundClient` over the v2 round API, stdlib only. Every failure raises: a 500 or a timeout must never read as "no submissions" or "no scores", because an empty field composes an all-burn board and a transport hiccup would burn a round's emission. * `cybergym_round_benchmark` — the real differential. Rebuilds from the server-supplied proof and requires crash-on-vulnerable AND clean-on-patched, with the strict sanitizer-report + expected- death crash rule mirrored from cathedral_distill (reproduced, not imported, so the payout path carries no optional cross-repo dependency). A rule naming a sanitizer this detector cannot recognise is refused rather than silently making the task unsolvable for the whole field. * `cybergym_round_daemon` — polls the block, steps the loop, records weights. Off-chain the backend is the block authority and the daemon adopts its geometry, so a validator cannot drift onto a different schedule than the server it evaluates for. The weight sink is a local trail, not a substrate call; swapping it is the only change on-chain requires. The tie-break nonce is explicitly a TEST stand-in — deterministic so validators agree, and NOT adversarially safe. `RoundConfig` makes the round geometry configurable (production values stay the defaults), which is what lets a full submit -> evaluate -> compose cycle run in a minute instead of three days. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… be filtered away
Found running the daemon as a real process against a real backend. `compose_and_set` composed the
board, then handed the setter `{hotkey: share for s in board.standings if s.lane_share > 0}` — so
`board.lane_burn` never reached the chain. A round with no qualifying miner composed "forfeit the
whole lane" and then set an EMPTY vector. An empty vector is not a forfeit; it is a malformed set,
and the lane's allocation went nowhere instead of to the sandbox lane, which is the rule it was
meant to honour. The existing test asserted `calls == [{}]` — the bug, written down.
`SetWeightsFn` now takes a `LaneWeights` carrying the miner shares AND the burn together, so the
share cannot be dropped by filtering. The two always sum to 1, and the e2e now asserts that.
The same fix corrects a second case the process run exposed: before its first compose a validator
re-asserted an EMPTY vector on the keep-alive, which asserts nothing while looking like a healthy
weight set. `RuntimeState.last_burn` starts at 1, so until it has composed anything a validator
honestly asserts that the lane forfeits.
Also adds `cybergym_round_main`, the validator entrypoint. It REFUSES to start without
`CYBERGYM_ALLOW_OFFCHAIN=1`: a daemon that records weights to a file instead of setting them on
chain looks healthy while the chain zeroes it, so that has to be a decision, not a default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Second finding from the deployment run (pushed to this branch)Running the daemon as a real process against a real backend surfaced a second payout bug, distinct from the denominator one.
The existing test asserted
The same change fixes a second case the process run exposed: before its first compose, a validator re-asserted an empty vector on the 300-block keep-alive — asserting nothing while looking like a healthy weight set. Verified as two real processesBackend and two validators, separate OS processes over real sockets, 72-block rounds at 0.3s/block: Three-way tie broken deterministically by the nonce — both validators picked the same king independently. 107 tests pass across the v2 modules. |
…rator visibility After the weights are set, the validator tells the backend what it composed so the dashboard can show whether the validators agree — the operational question that matters most. DISPLAY ONLY: it feeds no score and no average, so a backend that ignores or mangles it cannot change a payout. Best-effort by design — `report_weights` returns False instead of raising, and it runs AFTER the set, because failing to report what we set must never stop us from setting it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…clamped zero Found reading the live dashboard off the deployed box. The reporting used `max(block // round_blocks - 1, 0)`, so the very first compose — which scores round -1, i.e. nothing — was filed under round 0. The dashboard then showed round 0 already composed as an all-burn board while its real compose was still a whole round away. It now uses `submission_round_being_scored`, the same function the runtime composes from, and reports nothing when there is no scored round yet. The weights are still SET in that case; skipping the set would let the chain zero the validator. Only the report is withheld. Display-only, so it could never move a payout — but it was wrong exactly where an operator looks to answer "did the validators agree on this round?". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#226 `ruff check` and `ruff format --check` run over cathedral_thin in CI, and three merges this morning (#226, #229, #230) each landed with that gate failing. I merged them without reading the checks, so main has been red all day and every subsequent PR inherited a red baseline. Removes three unused imports and applies ruff format across the v2 round modules, including cybergym_round_scoring.py which came in unformatted with #229. No behaviour change: the 109 v2 round tests pass identically before and after. Merging this repairs main as well as this branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Self-review before merge (owner-directed; posted for post-merge audit)Merging on jared's instruction. Two reward-path fixes and the deployable side. What I checked before merging:
Correction I owe on this PR: the lint gate ( Verified end to end on the box: real corpus, real enclave runs, real differential, validators converging on one vector. |
Two commits. The first is a reward-path security fix; the second is the deployment wiring that found it.
1. The score denominator was the miner's to choose
The v2 end-to-end dry run put five miners through a real round. One of them (
5MinerEcho) tripped its run budget after two tasks, so the sandbox kept only those two PoCs — both solved — and it scored 100, taking the king slot and 0.84 of the lane from a miner that attempted all six tasks and solved five.The denominator was
len(submission.tasks). A miner controls that. The winning move was to withhold your failures.The score is now out of the round's authoritative task set, published by the server (
GET /v2/tasks) and identical for everyone. That closes both directions:RoundClientgainsfetch_round_tasks, andbenchmark_and_reportrefuses to score a round that published no task set rather than falling back to the miner-supplied denominator. Abstention is untouched: a validator that ran out of time still reports unevaluated rather than turning "I did not finish" into a real zero.2. The deployable side
The merged round loop left four boundaries injected. These fill them in:
cybergym_round_client— the concreteRoundClientover the v2 API, stdlib only. Every failure raises. A 500 or a timeout must never read as "no submissions": an empty field composes an all-burn board, so a swallowed error would burn a round's emission on a hiccup.cybergym_round_benchmark— the real differential. Crash on vulnerable and clean on patched; one half alone proves nothing. The strict crash rule (canonical sanitizer report plus the death the task is known to die) is mirrored fromcathedral_distill.cybergym_repro— reproduced rather than imported so the payout path carries no optional cross-repo dependency. Hardening over the original: a rule naming a sanitizer this detector cannot recognise is refused, instead of silently making that task unsolvable for the entire field.cybergym_round_daemon— polls the block, steps the loop, records weights. Off-chain the backend is the block authority and the daemon adopts its geometry, so a validator cannot drift onto a different schedule than the server it evaluates for. The weight sink is a local trail, not a substrate call; swapping it is the only change going on-chain requires.RoundConfig— round geometry configurable, production values still the defaults. This is what lets a full submit → evaluate → compose cycle run in a minute.The tie-break nonce is explicitly a TEST stand-in (
offchain_nonce): deterministic so validators agree, and not adversarially safe. Production must pass a real block-hash nonce.Verification
0.84 / 0.07 / 0.03 / 0.03 / 0.03.tests/thin/are a missingbittensor_walletin this environment, untouched by this branch.No chain is touched anywhere in this PR.