Skip to content

Model hub: verification sandbox runner (P4) - #49

Open
w4ffl35 wants to merge 2 commits into
mainfrom
feat/verification-sandbox-runner
Open

w4ffl35 wants to merge 2 commits into
mainfrom
feat/verification-sandbox-runner

Conversation

@w4ffl35

@w4ffl35 w4ffl35 commented Sep 17, 2026

Copy link
Copy Markdown
Member

Closes #48.

What this adds

  • .github/workflows/hub-verify.yml: a new workflow, triggered by
    repository_dispatch (hub-artifact-verify), that runs on
    ubuntu-latest only — never the self-hosted spikeforge-ci runner
    deploy-hetzner.yml uses, since that runner holds the Hetzner deploy
    key and running torch.load on a stranger's bytes there is the
    scenario plans/hub_accounts_plan.md §7.2 names as the most
    predictable way this design could be compromised. It's hardened with
    step-security/harden-runner in block mode, allow-listing GitHub's
    own checkout/runner endpoints, the PyPI/PyTorch install endpoints, and
    the two hub-api endpoints the job talks to (cdn.spikeforge.net,
    hub.spikeforge.net), and has a 20-minute wall-clock timeout.
  • scripts/hub_verify/: the runner itself, reusing the existing pipeline
    rather than reimplementing it — spikeforge.serving.bundle for
    checksum/manifest verification and the weights_only=True load,
    spikeforge_hub.inspect/.compat, spikeforge.nir_bridge.validate
    for the same NIR export + reference-interpreter drift check
    spikeforge_hub.import_model runs for the curated catalog, and
    spikeforge_targets.energy for the SOP/MAC/AC report. A rejection
    names the failing step and the actual reason (e.g. "bundle: weights
    are not loadable: ...", "drift: the exported NIR graph drifts from
    the trained module at ...") rather than a generic failure.
  • A decompressed-size cap in spikeforge/serving/bundle.py's archive
    reader (bundle_manifest.MAX_DECOMPRESSED_BYTES, 256 MiB — matching
    the plan's own per-artifact quota, §5.3), checked against each zip
    entry's declared size before any entry is read. Before this change
    _read_archive had no cap at all, so a zip bomb was an unbounded read
    into memory rather than a clean, named rejection.

The contract this implements

Neither this endpoint nor its hub-api counterpart existed in code as of
this session (spikeforge-hub-api#4 is open, no branch/PR). The one
binding requirement I found in committed code is
hub_api/catalog/trust.py::label_for, which only shows
machine-checked when version.verification["passed"] is the JSON
boolean true. Everything else below is this repo's proposal for the
rest of the contract; issue #48 and hub-api#4 both describe it only in
prose ("pass/fail, reasons, NIR/compat/energy summary"), so it needs a
look from whoever picks up hub-api#4 before both sides are wired
together for real:

  • POST {SPIKEFORGE_HUB_API_BASE_URL}/internal/v1/verifications/{version_id}
    with a JSON body: version_id, passed (bool), reasons (list of
    strings, empty iff passed), checks (bundle/inspect/compat/
    drift/energy detail), summary, generated_at, runner.
  • Signed with header X-Spikeforge-Hub-Signature: sha256=<hex hmac>,
    HMAC-SHA256 over the exact raw request body bytes (the same
    convention GitHub itself uses for webhook signatures), under the
    shared secret SPIKEFORGE_HUB_VERIFICATION_SECRET — matching
    config.verification_secret / SPIKEFORGE_HUB_VERIFICATION_SECRET
    already declared in hub_api/config.py.
  • Dispatch: repository_dispatch with event_type: hub-artifact-verify and client_payload: {version_id, artifact_url}.
    The callback base URL is deliberately not read from the dispatch
    payload — it's a fixed repository variable
    (vars.SPIKEFORGE_HUB_API_BASE_URL) this job trusts, not
    attacker-influenced data.
  • Pass/fail rule this runner applies: bundle integrity/format/weights-
    loadability and the drift check are hard gates; compat's preset
    classification and the energy report are informational only (a
    .spkf bundle already carries its own explicit TopologySpec, so
    unlike a bare downloaded artifact it never needs to match a known
    preset to be verifiable).

This repo needs two things configured before the workflow can run for
real: the SPIKEFORGE_HUB_VERIFICATION_SECRET secret and the
SPIKEFORGE_HUB_API_BASE_URL repository variable.

One thing worth flagging

My review copy of this repo has plans/hub_accounts_plan.md (the design
of record this issue points at) only on a local, unpushed commit — it
is not present on origin/main as of this branch. This PR's code
comments cite it by section (§5.3, §7, §7.2); that's fine once the plan
lands on main, but a reviewer diffing against main today won't find
the file. Worth checking before merge.

Testing

  • pytest (full suite, -m "not network"): 1372 passed, 8 skipped, 3
    deselected — no regressions in the existing bundle/hub suites.
  • pytest tests/test_hub_verify_*.py tests/test_serving_bundle.py: 34
    passed, including the new decompressed-size-cap test and an
    end-to-end pipeline test that builds a real .spkf bundle and runs it
    through the full funnel.
  • ruff check .: clean.
  • mypy spikeforge/serving/bundle.py spikeforge/serving/bundle_manifest.py:
    clean (note: spikeforge.serving.bundle is already in mypy.ini's
    ignore_errors list, pre-existing). scripts/hub_verify isn't part
    of the mypy/ci.yml gate (only spikeforge/,
    spikeforge_targets/, spikeforge_hub/, spikeforge_io/,
    spikeforge_serve/, spikeforge_clients/, server/ are), but I ran
    it manually against the new package and it's clean too.
  • What I could not test: the workflow itself has not run on GitHub
    Actions, and there is no live hub-api deployment or real
    SPIKEFORGE_HUB_VERIFICATION_SECRET to exercise the callback against
    end to end. The harden-runner egress allowlist is my best-effort
    read of what GitHub Actions' own infrastructure needs; I'd recommend
    a first run with egress-policy: audit to confirm it before relying
    on block.

A GitHub Actions workflow, triggered by repository_dispatch from
spikeforge-hub-api, that fetches one community-uploaded artifact from a
signed URL and runs it through the existing verification pipeline on a
GitHub-hosted (never self-hosted) runner: bundle checksum/manifest
verification and the weights_only=True load already used by bundle.py,
spikeforge_hub.inspect and .compat, the NIR export and reference-
interpreter drift check spikeforge_hub.import_model already runs for the
curated catalog, and a spikeforge_targets.energy report. It posts a
signed pass/fail report to the hub API's (not yet built)
POST /internal/v1/verifications/{version_id}, naming which step failed
and why on rejection rather than a generic failure message.

Also adds a decompressed-size cap to bundle.py's archive reader, which
had none: a zip bomb was previously an unbounded read into memory rather
than a clean, named rejection.

The self-hosted spikeforge-ci runner (deploy-hetzner.yml) is deliberately
never used here, since it holds the Hetzner deploy key and running
torch.load on a stranger's bytes there is the plan's own worst-case
scenario (plans/hub_accounts_plan.md §7.2).
Every job here sets up Python with `cache: pip`, and on this fleet that
step fails outright: "No file ... matched to [**/requirements.txt or
**/pyproject.toml], make sure you have checked out the target
repository". All seven pyproject.toml files are present in `packages/`
at that exact path -- checked on the runner's own workspace while a job
sat failed.

The runners keep `_work` on another drive behind a symlink, and the
action's dependency-file glob does not see through it. It has taken out
`blocked-deps`, `client`, `docs` and all five `extras` jobs, and it took
out the dashboard's release workflow earlier today for the same reason.
It is not reliably reproducible -- an earlier run on main globbed fine --
which makes it worse, not better: a pipeline that merges on green will
see spurious red it cannot distinguish from a real failure.

Removing it costs nothing. A self-hosted runner keeps `~/.cache/pip`
between jobs by itself, so the action was caching a cache.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model hub: verification sandbox runner (P4)

1 participant