Symptom
test-wheel-ubuntu (and other jobs using Jimver/cuda-toolkit with method: network) fail during Install CUDA Toolkit before any Mooncake tests run.
Seen on PR #3969 run https://github.com/kvcache-ai/Mooncake/actions/runs/34425688979/job/102715019979
Log excerpt
E: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/Packages
File has unexpected size (1799720 != 1817210). Mirror sync in progress? [IP: …]
##[error]Error: The process '/usr/bin/sudo' failed with exit code 100
Also warns CUDA apt source is configured twice (archive_uri-…noble.list + cuda-ubuntu2404-x86_64.list).
Context
ci.yml pins Jimver/cuda-toolkit@v0.2.24
_build-efa-wheel.yaml already uses @v0.2.29
- Failure is infra/mirror, not product code; CI Gate then fails because
test-wheel-ubuntu failed
Proposed fix (separate PR)
- Retry the Install CUDA Toolkit step (2–3×) on apt size-mismatch / exit 100
- Bump
Jimver/cuda-toolkit in ci.yml toward the version already used by efa wheel (or newer), and avoid duplicate apt sources if still present
- Optional: document that empty re-run is OK for this flake
Happy to send a small CI-only PR.
Symptom
test-wheel-ubuntu(and other jobs usingJimver/cuda-toolkitwithmethod: network) fail during Install CUDA Toolkit before any Mooncake tests run.Seen on PR #3969 run https://github.com/kvcache-ai/Mooncake/actions/runs/34425688979/job/102715019979
Log excerpt
Also warns CUDA apt source is configured twice (
archive_uri-…noble.list+cuda-ubuntu2404-x86_64.list).Context
ci.ymlpinsJimver/cuda-toolkit@v0.2.24_build-efa-wheel.yamlalready uses@v0.2.29test-wheel-ubuntufailedProposed fix (separate PR)
Jimver/cuda-toolkitinci.ymltoward the version already used by efa wheel (or newer), and avoid duplicate apt sources if still presentHappy to send a small CI-only PR.