Skip to content

fix(round): score against the published task set + the validator's deployable side - #231

Merged
ai-hpc merged 6 commits into
mainfrom
ai-hpc/round-config
Sep 4, 2026
Merged

ai-hpc merged 6 commits into
mainfrom
ai-hpc/round-config

Conversation

@ai-hpc

@ai-hpc ai-hpc commented Sep 4, 2026

Copy link
Copy Markdown
Member

Two commits. The first is a reward-path security fix; the second is the deployment wiring that found it.

1. The score denominator was the miner's to choose

The v2 end-to-end dry run put five miners through a real round. One of them (5MinerEcho) tripped its run budget after two tasks, so the sandbox kept only those two PoCs — both solved — and it scored 100, taking the king slot and 0.84 of the lane from a miner that attempted all six tasks and solved five.

The denominator was len(submission.tasks). A miner controls that. The winning move was to withhold your failures.

The score is now out of the round's authoritative task set, published by the server (GET /v2/tasks) and identical for everyone. That closes both directions:

before after
submits only its 2 solves 100 33.3
attempts 6, solves 5 83.3 83.3 (king)
invents 20 fake tasks that "solve" numerator padded ignored

RoundClient gains fetch_round_tasks, and benchmark_and_report refuses to score a round that published no task set rather than falling back to the miner-supplied denominator. Abstention is untouched: a validator that ran out of time still reports unevaluated rather than turning "I did not finish" into a real zero.

2. The deployable side

The merged round loop left four boundaries injected. These fill them in:

  • cybergym_round_client — the concrete RoundClient over the v2 API, stdlib only. Every failure raises. A 500 or a timeout must never read as "no submissions": an empty field composes an all-burn board, so a swallowed error would burn a round's emission on a hiccup.
  • cybergym_round_benchmark — the real differential. Crash on vulnerable and clean on patched; one half alone proves nothing. The strict crash rule (canonical sanitizer report plus the death the task is known to die) is mirrored from cathedral_distill.cybergym_repro — reproduced rather than imported so the payout path carries no optional cross-repo dependency. Hardening over the original: a rule naming a sanitizer this detector cannot recognise is refused, instead of silently making that task unsolvable for the entire field.
  • cybergym_round_daemon — polls the block, steps the loop, records weights. Off-chain the backend is the block authority and the daemon adopts its geometry, so a validator cannot drift onto a different schedule than the server it evaluates for. The weight sink is a local trail, not a substrate call; swapping it is the only change going on-chain requires.
  • RoundConfig — round geometry configurable, production values still the defaults. This is what lets a full submit → evaluate → compose cycle run in a minute.

The tie-break nonce is explicitly a TEST stand-in (offchain_nonce): deterministic so validators agree, and not adversarially safe. Production must pass a real block-hash nonce.

Verification

  • 102 tests pass across the six v2 modules (40 new).
  • The e2e dry run passes: 5 miners, a clone refused by the screener, a budget trip enforced, 3 validators independently benchmarking and converging on one identical weight vector0.84 / 0.07 / 0.03 / 0.03 / 0.03.
  • Pre-existing collection errors in tests/thin/ are a missing bittensor_wallet in this environment, untouched by this branch.

No chain is touched anywhere in this PR.

ai-hpc and others added 3 commits September 4, 2026 06:20
…er's own submission

Found by the v2 end-to-end dry run. A miner whose agent tripped its run budget after two tasks
submitted only those two, both solved, and scored 100 — taking the king slot (0.84 of the lane)
from a miner that attempted all six and solved five. The denominator was `len(submission.tasks)`,
which the miner controls, so the winning move was to withhold your failures.

The score is now out of the round's authoritative task set, published by the server and identical
for every miner. That closes it from both directions: a task the miner never submitted counts
unsolved, and a task it invented that is not in the set is ignored rather than padding the
numerator. `RoundClient` gains `fetch_round_tasks`, and `benchmark_and_report` refuses to score a
round that published no task set rather than falling back to a miner-supplied denominator.

Abstention is unaffected: a validator that ran out of time still reports unevaluated rather than
turning "I did not finish" into a real zero.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fferential, off-chain daemon

Fills in the three seams the merged round loop left injected, so a validator can actually run:

* `cybergym_round_client` — the concrete `RoundClient` over the v2 round API, stdlib only. Every
  failure raises: a 500 or a timeout must never read as "no submissions" or "no scores", because
  an empty field composes an all-burn board and a transport hiccup would burn a round's emission.
* `cybergym_round_benchmark` — the real differential. Rebuilds from the server-supplied proof and
  requires crash-on-vulnerable AND clean-on-patched, with the strict sanitizer-report + expected-
  death crash rule mirrored from cathedral_distill (reproduced, not imported, so the payout path
  carries no optional cross-repo dependency). A rule naming a sanitizer this detector cannot
  recognise is refused rather than silently making the task unsolvable for the whole field.
* `cybergym_round_daemon` — polls the block, steps the loop, records weights. Off-chain the
  backend is the block authority and the daemon adopts its geometry, so a validator cannot drift
  onto a different schedule than the server it evaluates for. The weight sink is a local trail,
  not a substrate call; swapping it is the only change on-chain requires. The tie-break nonce is
  explicitly a TEST stand-in — deterministic so validators agree, and NOT adversarially safe.

`RoundConfig` makes the round geometry configurable (production values stay the defaults), which
is what lets a full submit -> evaluate -> compose cycle run in a minute instead of three days.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… be filtered away

Found running the daemon as a real process against a real backend. `compose_and_set` composed the
board, then handed the setter `{hotkey: share for s in board.standings if s.lane_share > 0}` — so
`board.lane_burn` never reached the chain. A round with no qualifying miner composed "forfeit the
whole lane" and then set an EMPTY vector. An empty vector is not a forfeit; it is a malformed set,
and the lane's allocation went nowhere instead of to the sandbox lane, which is the rule it was
meant to honour. The existing test asserted `calls == [{}]` — the bug, written down.

`SetWeightsFn` now takes a `LaneWeights` carrying the miner shares AND the burn together, so the
share cannot be dropped by filtering. The two always sum to 1, and the e2e now asserts that.

The same fix corrects a second case the process run exposed: before its first compose a validator
re-asserted an EMPTY vector on the keep-alive, which asserts nothing while looking like a healthy
weight set. `RuntimeState.last_burn` starts at 1, so until it has composed anything a validator
honestly asserts that the lane forfeits.

Also adds `cybergym_round_main`, the validator entrypoint. It REFUSES to start without
`CYBERGYM_ALLOW_OFFCHAIN=1`: a daemon that records weights to a file instead of setting them on
chain looks healthy while the chain zeroes it, so that has to be a decision, not a default.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ai-hpc

ai-hpc commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

Second finding from the deployment run (pushed to this branch)

Running the daemon as a real process against a real backend surfaced a second payout bug, distinct from the denominator one.

compose_and_set composed the board and then handed the setter only {hotkey: share for s in board.standings if s.lane_share > 0}. board.lane_burn never reached the chain. A round with no qualifying miner composed "forfeit the whole lane" and then set an empty vector — which is not a forfeit, it is a malformed set, and the lane's allocation went nowhere instead of to the sandbox lane. That is the exact rule it was supposed to honour ("no miner → all weight to the sandbox lane").

The existing test asserted calls == [{}] — the bug, written down as expected behaviour.

SetWeightsFn now takes a LaneWeights carrying the miner shares and the burn together, so the share cannot be filtered away. They always sum to 1, and the e2e asserts it.

The same change fixes a second case the process run exposed: before its first compose, a validator re-asserted an empty vector on the 300-block keep-alive — asserting nothing while looking like a healthy weight set. RuntimeState.last_burn now starts at 1, so until it has composed anything a validator honestly asserts the lane forfeits.

Verified as two real processes

Backend and two validators, separate OS processes over real sockets, 72-block rounds at 0.3s/block:

4 attempts, 3 accepted, 1 rejected_duplicate (the clone), 3 agents ran, 12 PoCs
both validators reported; server averaged 100.000000 x3 (voters=2)
both composed identically at block 138:  B 0.900000 | A 0.07 | C 0.03   burn 0.000000
lane accounted for at compose: 1.000000
every pre-compose keep-alive row: weights {} burn 1  (a full forfeit, not an empty set)

Three-way tie broken deterministically by the nonce — both validators picked the same king independently.

107 tests pass across the v2 modules.

ai-hpc and others added 3 commits September 4, 2026 14:45
…rator visibility

After the weights are set, the validator tells the backend what it composed so the dashboard can
show whether the validators agree — the operational question that matters most.

DISPLAY ONLY: it feeds no score and no average, so a backend that ignores or mangles it cannot
change a payout. Best-effort by design — `report_weights` returns False instead of raising, and it
runs AFTER the set, because failing to report what we set must never stop us from setting it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…clamped zero

Found reading the live dashboard off the deployed box. The reporting used
`max(block // round_blocks - 1, 0)`, so the very first compose — which scores round -1, i.e.
nothing — was filed under round 0. The dashboard then showed round 0 already composed as an
all-burn board while its real compose was still a whole round away.

It now uses `submission_round_being_scored`, the same function the runtime composes from, and
reports nothing when there is no scored round yet. The weights are still SET in that case;
skipping the set would let the chain zero the validator. Only the report is withheld.

Display-only, so it could never move a payout — but it was wrong exactly where an operator looks
to answer "did the validators agree on this round?".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#226

`ruff check` and `ruff format --check` run over cathedral_thin in CI, and three merges this
morning (#226, #229, #230) each landed with that gate failing. I merged them without reading the
checks, so main has been red all day and every subsequent PR inherited a red baseline.

Removes three unused imports and applies ruff format across the v2 round modules, including
cybergym_round_scoring.py which came in unformatted with #229. No behaviour change: the 109 v2
round tests pass identically before and after.

Merging this repairs main as well as this branch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ai-hpc

ai-hpc commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

Self-review before merge (owner-directed; posted for post-merge audit)

Merging on jared's instruction. Two reward-path fixes and the deployable side.

What I checked before merging:

  • The denominator fix closes the exploit in both directions (withheld failures score as unsolved; invented tasks are ignored), and benchmark_and_report refuses a round that published no task set rather than falling back to the miner-supplied denominator.
  • The forfeited lane share now travels WITH the miner weights in one object, so it cannot be filtered away — and the pre-compose keep-alive asserts a full forfeit instead of an empty vector.
  • The docker differential requires crash-on-vulnerable AND clean-on-patched, with the strict sanitizer + expected-death rule reproduced (not imported) so the payout path carries no optional cross-repo dependency. A rule naming a sanitizer this detector cannot recognise is refused rather than silently making a task unsolvable for the whole field.
  • The HTTP client raises on every failure: a 500 must never read as "no scores", because an empty field composes an all-burn board.
  • The off-chain nonce is explicitly labelled a TEST stand-in — deterministic so validators agree, not adversarially safe.

Correction I owe on this PR: the lint gate (ruff check + ruff format --check over cathedral_thin) has been failing on main since #226 this morning. I merged #226, #229 and #230 without reading their checks, so main has been red all day and this branch inherited it. The final commit here repairs the tree; merging this makes main green again. All checks pass on this PR (Python 3.11, 3.12, publisher suite, integration lane, SEV-SNP contract, release).

Verified end to end on the box: real corpus, real enclave runs, real differential, validators converging on one vector.

@ai-hpc
ai-hpc merged commit 457f62f into main Sep 4, 2026
9 checks passed
@ai-hpc
ai-hpc deleted the ai-hpc/round-config branch September 4, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant