Conversation
`qwen3-cuda-compile` checked one package (`pegainfer-server`) while `qwen3-cuda-clippy` lints eight that include it, both with `--all-targets`, so the compile job's surface is strictly contained in the clippy job's. The `feature-model-cuda` matrix ran a `compile` and a `clippy` leg over the same package, feature and `--all-targets` surface; clippy subsumes check, so the `compile` leg is redundant. The corresponding `include` entry is removed with the dimension value, since a leftover `gate: compile` entry would be appended as a new combination with no model fields bound. Ready-for-review PRs go from 16 to 13 executed jobs. No coverage is dropped: every gate the removed jobs covered is still compiled, linted or run by a retained job. Reported in pegainfer-project#975. Signed-off-by: luxing <luicarus@users.noreply.github.com>
|
@codex review |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
FeathBow
left a comment
There was a problem hiding this comment.
Thanks! Before we go further on job counting, I want to flag where the time actually goes. The wall clock on a ready PR is one CUDA lane, about six minutes, and removing three jobs left it there. Job dedup buys summed runner minutes.
The repeated work is nvcc. build.rs spawns it directly, so RUSTC_WRAPPER=sccache never sees it, and every CUDA lane compiles the same seventeen shared translation units under pegainfer-kernels/csrc from cold. That is why a lane can report 92% sccache hits and still spend three and a half minutes in cargo. We paid for that six times per PR before #1048 and pay three times now. The seam is narrow. pegainfer_build::CudaToolkit hands out nvcc as a PathBuf and build.rs spawns it in four or five places, so turning that into a command that can carry a launcher puts sccache in front of nvcc in CI and on cold dev machines too. pegainfer-build already runs in the CPU unit test job and already has a fake toolkit tree helper, so that logic is testable without a GPU. I recommend we spike it next.
Two smaller things. The toolchain pin is declared in rust-toolchain.toml and again in every dtolnay/rust-toolchain step, which is what produced the rustfmt component conflict we fixed by syncing nine copies. The action only sets rustup default, and the toolchain file overrides that as soon as cargo runs in the repo, so I recommend dropping the input and letting the file drive the install. Separately, the CUDA lanes key on model x gate, and with the gate axis gone Qwen3 still sits outside the matrix while the other two lines sit inside it. One row per model line, with the default features row carrying the shared crates, would make the next line a table row.
|
Taking this. Before writing code, I checked what sccache actually caches for I'll do the launcher work as a separate PR since it touches |
Every dtolnay/rust-toolchain step repeated the channel and components that ust-toolchain.toml already declares. When the two disagreed, rustup reported a component conflict; commit ba794bd patched that by hand-syncing the components list across all nine steps, which is the part that does not scale. rustup reads the toolchain file as soon as cargo runs in the repository, and installs the components it lists there, so the duplicated input is redundant. Verified locally: a stable toolchain installed with --profile minimal (no rustfmt, no clippy) picks both up on the first cargo fmt inside a project whose toolchain file requests them. The oolchain: input stays because the action requires it. Reported in pegainfer-project#975. Signed-off-by: luxing <luicarus@users.noreply.github.com>
The CUDA lanes were grouped by model x gate, which left Qwen3 outside the matrix after the compile leg went away while gemma4 and qwen35 stayed inside it. Three model lines, two ways of declaring them. This makes the matrix a single include list with one row per model line, so adding a model line is one more row. The Qwen3 row is the default-feature line, which is the only one that takes no --features, so it is also the row that carries the shared crates (pegainfer-core, -kernels, -kv-cache, -kv-offload, -sample, -bench) alongside pegainfer-server. Coverage is unchanged, verified by expanding the matrix and comparing the resulting command lines: the Qwen3 row still lints the same eight packages with no --features, and the Gemma 4 / Qwen3.5 rows still lint their own package with their own feature. cargo_command and railing_args were only ever clippy and -- -D warnings through the gate dimension, so they are now written directly. Job executions are unchanged at 13; this is a structure change, not a scheduling one. Reported in pegainfer-project#975. Signed-off-by: luxing <luicarus@users.noreply.github.com>
Description
Part of #975.
On current
master, a ready-for-review PR expands to 16 job executions. Three of those jobs cover compile surfaces that are already exercised by retained Clippy jobs. This PR removes those redundant executions without reducing package, feature, or target coverage.1.
feature-model-cuda: remove thecompileleg (2 jobs)The
compileandclippylegs use the same package, feature, release profile, and--all-targetssurface:For the same package / feature / target surface, the retained Clippy invocation exercises the same compilation surface while also running the lints, so the separate
compileleg adds no additional coverage.The
gate: compilevalue and its correspondingincludeentry are removed together. Removing only the dimension value would leave theincludeentry unmatched, causing GitHub Actions to append it as a new matrix combination without the expectedmodel,package, andfeaturebindings.2.
qwen3-cuda-compile: remove the job (1 job)The two Qwen3 jobs currently cover:
The compile job's package set (
pegainfer-server) is a subset of the Clippy job's package set, with the same--all-targetssurface.qwen3-cuda-clippyalready carriesCUDA_PATH,PEGAINFER_CUDA_SM, andPEGAINFER_NVCC_JOBSon its Cargo step, so the CUDA-specific build environment required by the retained command is preserved.Coverage map
Every surface covered by a removed job remains covered by a retained command:
Qwen3 CUDA compilepegainfer-server, default features,--all-targetsQwen3 CUDA Clippy--all-targetssurfaceGemma 4 CUDA Compilepegainfer-gemma4,gemma4,--all-targetsGemma 4 CUDA ClippyQwen3.5 CUDA Compilepegainfer-qwen35,qwen35,--all-targetsQwen3.5 CUDA ClippyAll other gates are unchanged: CPU Clippy, CPU unit tests, simulated frontend E2E, DCO, attribution, formatting, and locked Cargo metadata.
Expected impact
Ready-for-review job executions:
16 → 13
Repeated setup executions:
actions/checkoutdtolnay/rust-toolchainmozilla-actions/sccache-actionapt-get install protobuf-compiler libibverbs-devJimver/cuda-toolkitThe direct expected benefit is lower summed runner time and less repeated setup work. The wall-time effect depends on which jobs land on the critical path, so I won't claim an improvement there until it is measured.
Before numbers
These numbers come from the Actions jobs API for the two runs linked in #975.
Here, wall time means the active-job span:
min(job.started_at)→max(job.completed_at)and summed job runtime is:
Σ(job.completed_at - job.started_at)over all executed jobs in the run.
The three jobs removed by this PR account for:
Qwen3 CUDA compileGemma 4 CUDA CompileQwen3.5 CUDA CompileSimply subtracting those jobs from the baseline gives projected summed runtimes of approximately:
32950033957— 33.7% lower;32869418134— 30.8% lower.Those are arithmetic projections, not measured after-results. They assume the retained jobs behave exactly as they did in the baseline runs.
The wall-time effect is less predictable:
32950033957, the longest retained job wasGemma 4 CUDA Clippyat about 6m17s, so removing the compile jobs alone would not necessarily reduce the critical path.32869418134, the 10m56s critical-path job wasQwen3.5 CUDA Compile, which this PR removes. That run therefore could improve substantially if the retainedQwen3.5 CUDA Clippyjob does not exhibit the same slowdown.Because the second run is an outlier, I don't want to infer a wall-time improvement from it without after-data.
Measurement convention
The issue reports 6m39s and 11m00s for the same two baseline runs. Reading them through the Actions API produces a different span depending on the timestamps used:
run_started_at→updated_atThe whole-run span includes runner allocation before the first job starts and workflow finalization after the last one completes, so it is always the larger of the two — by 1m20s in one run and 4s in the other. That gap is scheduling overhead, not compute, and it varies enough between runs that mixing the two definitions would make a before/after comparison unreliable.
The comparison above therefore uses the active-job span,
min(job.started_at)→max(job.completed_at). I'm stating the convention up front so the after-runs are read the same way.After numbers
Once this PR has completed runs of its own, I'll report the same measurements across multiple ordinary PR runs, with links to the raw Actions results:
These measurements need to come from completed GitHub Actions runs: runner scheduling and cache behavior are properties of the CI environment and cannot be reproduced meaningfully by a local build.
Follow-up under #975
The fused CPU gate is intentionally not included in this PR. CPU fusion can reduce summed runner time while also changing the workflow critical path, so I'd prefer to evaluate and measure that tradeoff separately rather than mix it with the straightforward CUDA deduplication here.
Out of scope
Left untouched as specified in #975:
nvcccompilation through a cache launcher;release.ymlis unchanged.Type of Change
CI maintenance / performance optimization.
Checklist
docs/conventions/coding-style.md).CLAUDE.md).Local verification
This PR only changes
.github/workflows/ci.yml; there are no Rust source, manifest, lockfile, orbuild.rschanges.Ran locally:
cargo fmt --all --check— passes (rustc 1.99.0-nightly, matchingrust-toolchain.toml).cargo metadata --locked --no-deps --format-version 1— passes.feature-model-cudaexpands to 2 executions; no unmatched matrixincludeentry remains.The full
## Testssuite fromCLAUDE.mdwas not run locally because the relevant commands require CUDA/GPU/model-weight dependencies unavailable in this environment. The checklist item is therefore left unchecked; the retained CI jobs will exercise those paths on this PR.