fix(roofline): drop only cuda-graph capture, not inductor, for vLLM - #1470
Open
mgehre-amd wants to merge 1 commit into
Open
mgehre-amd wants to merge 1 commit into
mgehre-amd wants to merge 1 commit into
Conversation
The profiling fallback appended --enforce-eager for vLLM, which turns off
torch.compile/inductor as well as graph capture. A trace taken that way
profiles the uncompiled kernels, not the ones the measured runs execute, so
the roofline attributes device time to kernels production never runs.
--compilation-config.cudagraph_mode NONE drops only the capture and keeps
inductor compilation, so the profiled kernels match the measured ones.
Changes:
- The dotted form is deliberate. It is two plain tokens, so no shlex
round-trip in the arg pipeline can damage it; the JSON form
--compilation-config {"cudagraph_mode":"NONE"} only survives because
_grid_server_args._repair_unquoted_json puts back the double quotes a
round-trip strips. vLLM parses the dotted form in
vllm/utils/argparse_utils.py: json.loads("NONE") fails, so it falls through
to the raw string and merges to {"cudagraph_mode":"NONE"}.
- _with_cuda_graph_disabled now dedups on the option's base name rather than
the whole flag string. vLLM's flag is two tokens with a dotted option, so
whole-string matching would never fire and the flag would be re-appended on
every call. Comparing the part before any "." or "=" also treats an
operator-supplied --compilation-config (JSON form) or
--compilation-config.<field> as already present, which matters because vLLM
re-appends a synthesized --compilation-config for dotted args and argparse
would then take the last one silently. The original property is kept:
--disable-cuda-graph-extra is still not mistaken for --disable-cuda-graph,
because base names are compared whole.
- sglang is untouched; --disable-cuda-graph was already capture-only there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The profiling fallback appended
--enforce-eagerfor vLLM. That flag disablesgraph capture as well as torch.compile/inductor. Disabling torch.compile/inductor means
that there are different kernel fusions/kernel choices during profiling then during benchmarking.
vLLM's
--compilation-config.cudagraph_mode NONEdrops only the cuda-graph capture and keeps inductorcompilation, so the profiled kernels match the measured ones.
Linked issue(s): none.
Tests: added/updated? commands run?
Updated the three existing assertions that pinned the vLLM flag string
(
test_baseline_param_overrides.py,test_roofline_executor.py).Breaking changes: no.
PR addresses single concern: yes.
Root cause is upstream (Magpie/TraceLens/GEAK/IntelliKit/AgentKernelArena), ticket filed: no