Syncing up with the main branch - #594
Open
devalshahamd wants to merge 873 commits into
Open
devalshahamd wants to merge 873 commits into
devalshahamd wants to merge 873 commits into
Conversation
Added perf models for new ops
add sorting logic on kernel name
Update patches
# $O(subtree \times N_{launchers})$ traversal in `tree_perf.py`
`get_kernel_launchers` computed subtree GPU time by calling
`_compute_subtree_kernel_time_us(event)`, which called
`loop_and_aggregate_kernels` (a full recursive subtree traversal) *for
every launcher*. Since `add_gpu_ops_to_tree` already propagates all GPU
kernel UIDs up to every ancestor via event["gpu_events"], this is now an
$O(1)$ field lookup.
Split PR at request of @ajassani
#577 (comment)
<!--
Copyright (c) 2024 - 2025 Advanced Micro Devices, Inc. All rights
reserved.
See LICENSE for license information.
-->
# Pull Request Template
> **Note to AMDers:**
> This is a public repository. Please do **not** upload any confidential
or customer data. Make sure all such data has been anonymized or removed
before making this PR. If you need to attach any private files or links,
please insert a Internal OneDrive Link or a Jira Ticket Link instead.
- Rebuild unit_tests.tar.gz and e2e_tests.tar.gz with latest analysis_output_ref golden references (ProfileLens-validated) - Remove redundant analysis_output/ dirs, analysis_stream.ndjson, and other agent artifacts from both archives — each case now contains only the trace file and analysis_output_ref/ - Remove wan2.1_mi300 from e2e test suite (not a standard PyTorch trace) - Update eval-post-processing skill to generate split reports with separate Unit Test and E2E Test sections, per-case metrics (category, platform, runs, avg duration), and handling for single-split runs Made-with: Cursor
Made changes to include the config variables
…nd timesformer Made-with: Cursor
Update golden reference archives and eval post-processing skill
TraceLens Agent Update (4/10)
Expanded instructions for profiling benchmarks with vLLM and SGLang, including common flags and configuration edits for targeted and full-benchmark profiling.
Enhance profiling instructions for vLLM and SGLang
Standalone Analysis README cleanup
* Added initial steady state parsing * Renamed the orchestrator and deleted a redundant skill * Integrated GPU event analyzer metrics with semantic breakdown * Redundancy and hardcode removal * Fixed issue where agent was generating files on the fly * Added improved kernel alignment based on regex filters * Better kernel correlation by using kernel types * Reintroduced LLM into the comparison and alignment step * Added harmonization agent for better correlation between traces * Black + Added Copyright to files * Reformat using local Black after rebase * Copyright header fix
Update Inference_analysis.md
…er YAML Convert report_section_rules.yaml from JSON to proper YAML format so the required AMD copyright header (# comments) can be added, passing the CI copyright regression test. Data is identical — verified against existing repeatability results. Update eval-post-processing skill to use yaml.safe_load(). Made-with: Cursor
Add copyright header to report_section_rules.yaml
Contains commits up to bd84efa
Comparative graph capture support in evals was broken. Now capture folder paths are correctly extracted from trace csv and passed into prompt. Verified that standalone and comparative prompts to agent are still correct.
Added flag in run_repeatability_parallel.sh to skip reference generation
Adding patches for SGLang-dev 5.18.0 image. This version has some of the TraceLens patches integrated (upstreamed PRs), so only shape profiling related patches are added. Another fix is to override torchlib so files to ensure the rocm 7.2.4 so files are used to enable profiling of graph captured region. <!-- Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> Co-authored-by: Deval Shah <devashah@amd.com>
<!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> --------- Co-authored-by: Matt Williams <Matt.Williams+amdeng@amd.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
### Summary
This PR adds a script `kill_eval_jobs.sh` that kills all eval jobs
spawned by `run_repeatability_parallel.sh` and `generate_ref.sh`.
The analysis eval pipeline (`run_repeatability_parallel.sh`, and
`generate_ref.sh`) fans out into a large tree of processes per run: the
orchestrator scripts, their worker subshells, the Cursor `agent`/`pi`
analysis and eval calls, the `node` worker each of those spawns, and the
scripted-eval `python` processes. There was no clean way to stop a run:
The processes from a single run don't share one kill handle. Here is a
live snapshot of the tree from one `generate_ref.sh` run, showing the
different kernel IDs:
| PID | PPID | PGID | SID | Process |
|---|---|---|---|---|
| 110120 | 1 | **110120** | **110085** | `bash …/generate_ref.sh` (the
script; reparented to `init`) |
| 110127 | 110120 | 110120 | 110085 | `bash …/generate_ref.sh` (worker
subshell) |
| 110136 | 110127 | 110120 | 110085 | `bash …/generate_ref.sh` (worker
subshell) |
| 110138 | 110136 | **110138** | 110085 | `timeout 1800 agent …` (starts
a new group) |
| 110139 | 110138 | 110138 | 110085 | `agent …` (Cursor CLI) |
| 110299 | 110139 | 110138 | 110085 | `node …/cursor-agent/…` (the LLM
worker) |
| 110987 | 110139 | **110987** | **110987** | `bash -c …` (agent
tool-call; `setsid` → own group *and* session) |
- **Killing the top-level script's PID** doesn't cascade — its children
reparent to `init` and keep going, and the pipeline's retry loop
respawns new agents.
- **A process-group kill** (`kill -- -PGID`) can't cover the run in one
call, because a single run is spread across *multiple* process groups:
the `timeout` wrapper around each agent call starts a new group, so the
`timeout`/`agent`/`node` worker chain sits outside the script's group.
- **A session kill** doesn't help either — the agent's tool-call
subprocesses call `setsid` and leave the session entirely, so there's no
single session that contains the whole tree.
This PR tags each process with an inherited environment marker and adds
a `kill_eval_jobs.sh` script that that reliably kills an entire eval run
by killing jobs with this environment marker.
### Motivation
The core difficulty is *identifying* which processes belong to an eval
run. Matching by command name is unreliable (the real workers are
generic `node` and `python` processes), and matching by parentage or
process group fails because of reparenting and the agent CLI's own
process group. What every process in a run *does* have in common is that
it descends from one of the two entrypoint scripts. An inherited
environment marker survives reparenting, process-group changes, and
renames.
### Solution
Both entrypoints export `TRACELENS_EVAL_JOB=1`, which is inherited by
every descendant. `kill_eval_jobs.sh` then finds the whole run by
scanning `/proc/<pid>/environ` for that marker and signalling those
PIDs.
### Implementation
**`agent_evals/Analysis/eval_scripts/kill_eval_jobs.sh` (new)**
A standalone cleanup tool. It scans `/proc/<pid>/environ` for
`TRACELENS_EVAL_JOB=1` (guarding on file ownership so it only ever
touches the current user's processes) and signals the matches. Because
the marker is unique to the pipeline, unrelated `agent`, `claude`, or
`python` sessions are never touched.
Two subtleties drove the design:
- **The kill isn't atomic.** A process caught mid-run can fork a child —
a retry-backoff `sleep`, or a fresh agent — in the window between the
scan and the kill; that child inherits the marker but wasn't in the
snapshot. So the tool runs a **bounded rescan loop**: scan, signal,
repeat until a scan comes back empty (or `MAX_PASSES` is reached).
Because the orchestrator scripts are themselves tagged (see below), they
die on the first pass and stop respawning, and later passes just mop up
orphaned stragglers.
- **Graceful then forceful.** The first pass sends `SIGTERM` to let
processes shut down cleanly; every pass after that sends `SIGKILL`. If
anything is still alive after the last pass (e.g. stuck in
uninterruptible I/O), it prints a warning with an inspection command and
exits non-zero.
It also supports `--list` (show what would be killed, kill nothing) and
`-9` (`SIGKILL` from the first pass).
**`agent_evals/Analysis/eval_scripts/generate_ref.sh` and
`run_repeatability_parallel.sh`**
Each entrypoint gains a small guard at the top that sets the marker and
re-execs itself once:
```bash
if [[ "${TRACELENS_EVAL_JOB:-}" != 1 ]]; then
export TRACELENS_EVAL_JOB=1
exec bash "$0" "$@"
fi
```
The re-exec is needed since a process's `/proc/<pid>/environ` is fixed
when it starts. Exporting an environment variable in the script tags its
*children* but not the script itself. Without the re-exec, the scan
would find the agents but not the orchestrator scripts, and a kill would
leave those scripts alive to respawn the agents.
### Tests
Manual verification for both entrypoints:
- Launched `generate_ref.sh` and, separately,
`run_repeatability_parallel.sh`, let each spin up its agent tree, and
confirmed via `/proc/<pid>/environ` that the **orchestrator script
process itself**, its worker subshells, the `timeout`/`agent` launchers,
the `node` worker, and tool-call subprocesses were all tagged.
- Ran `kill_eval_jobs.sh --list` and confirmed it enumerated the full
tree (script + subshells + agent tree).
- Ran `kill_eval_jobs.sh` and independently verified afterward (a fresh
`/proc` scan plus `pgrep`) that **zero** marked processes remained and
nothing respawned.
- Confirmed the precision guard: an unmarked, unrelated `agent`/`python`
process is left untouched, and a marked but generically-named process
(e.g. a plain `sleep`) is still found and killed — demonstrating the
match is by environment marker, not command name.
- Confirmed the frozen-`environ` behavior directly: a runtime-only
`export` does **not** appear in the exporting process's own
`/proc/environ`, while the re-exec form does — validating why the
re-exec is necessary.
Skip creating dataframes for empty data. This avoids warnings such as this: ``` TraceLens/TraceLens/TreePerf/tree_perf.py:557: UserWarning: Input list of events is empty. Returning an empty DataFrame. warnings.warn( TraceLens/TraceLens/TreePerf/tree_perf.py:677: UserWarning: Input DataFrame is empty. Returning an empty summary DataFrame. warnings.warn( ``` Closes #482
## Summary The root `README.md` had several documentation links pointing at pages that either no longer exist or describe something other than what the link text claims. All fixes are contained in `README.md`; the other ten module READMEs across the repo were audited and are clean. ## What was wrong 1. **Stale duplicate "Supported Profile Formats" table.** The README carried two `## Supported Profile Formats` sections. The first used pre-reorg flat paths (`docs/generate_perf_report.md`, `docs/jax_analyses.md`, `docs/generate_perf_report_rocprof.md`, `docs/generate_perf_report_rocprof_pftrace.md`, `docs/generate_perf_report_genesis.md`) — none of which exist after the docs were reorganized under `docs/how-to/`. The `#supported-profile-formats` in-page anchors landed on this broken copy. The second table (further down) was already correct. 2. **Documentation table rows pointing at never-created docs.** The TreePerf row pointed at `docs/how-to/tree-perf-analysis.md` and the GPU Event Analyser row at `docs/how-to/gpu-event-analysis.md`; neither file exists. Both features are actually documented in `docs/how-to/sdk-analysis.md` (the `TreePerfAnalyzer` / `GPUEventAnalyser` SDK reference). ## Changes - Removed the stale duplicate "Supported Profile Formats" table so there is one authoritative section and a clean anchor target. - Preserved that table's unique **Genesis / Taichi** row by porting it into the canonical table, repointed to `docs/how-to/generate-perf-report-genesis.md`. - Repointed the **TreePerf** and **GPU Event Analyser** rows in the Documentation table to `docs/how-to/sdk-analysis.md`. Net effect: every relative link in the README now both resolves and points at a doc whose subject matches the link text.
Create util functions for calculating perf metrics
Create merge_intervals utility function to create list of merged, non-overlapping intervals given list of time intervals
#### NOTE: Reference CSVs have been updated. UID shifts due to additional events being added to trace tree affects UID columns for traces that re-assign UIDs during graph capture merge (vllm_decode_full and xdit_flux.1). xdit_hunyuanvideo and xdit_sd_3.5 are in eager mode so the only changes for these traces are changes in the call stack. ## Summary The tree builder's "event bleed" detection in `build_host_call_stack_tree` uses a strict `>` comparison to decide when an event ends after the current stack top. Due to profiling noise and floating point rounding there can be sub-nanosecond time differences between two events that really end at the same wall-clock time. When that happens, the strict comparison misclassifies a true parent-child pair as a bleed and discards the child. The existing bleed-handling logic (added in #694) handles time jitter between siblings, but not for time jitter between parent-children pairs. See #900: `runner.py(126): <module>` and its descendants (real `python_function` events that end at the same wall-clock time as their parent) were dropped from the tree in one run of a trace but not an identical rerun, purely because `ts + dur` rounded to a slightly different value depending on operand magnitudes. ### Solution Apply the same 1us tolerance already used for handling bleeds to the detection comparison itself: `event.end > stack_top.end + tolerance`. An event that ends within 1us of its parent's end is no longer treated as a bleed. ## Implementation - `TraceLens/Trace2Tree/trace_to_tree.py`: moved the `overlap_tolerance_us = 1.0` constant above the bleed-detection `if` and added it to the detection comparison's right-hand side. ## Tests - Verified against the exact scenario from #900 using local xDiT hunyuanvideo trace. The `call_stack_full` column in `unified_perf_summary` shows the fix restoring the missing top-level frames: **Pre-fix** — `runner.py(126): <module>` is dropped as a false bleed, so the call stack starts one level too deep: ``` runner.py(81): profile └─ base_model.py(611): profile └─ base_model.py(677): _run_timed_pipe └─ hunyuan.py(79): _run_pipe └─ ... ``` **Post-fix** — the same event is correctly nested as a child, restoring the full call stack: ``` runner.py(126): <module> ← restored └─ runner.py(81): profile └─ base_model.py(611): profile └─ base_model.py(677): _run_timed_pipe └─ hunyuan.py(79): _run_pipe └─ ... ``` Closes #900
This PR refreshes the checked-in Sphinx docs lockfile to remove vulnerable transitive dependencies in the Read the Docs environment. The change is limited to the generated dependency snapshot used for docs builds. - **Docs dependency refresh** - updates `cryptography` from `49.0.0` to `50.0.1` - updates `gitpython` from `3.1.55` to `3.1.61` - updates `tornado` from `6.5.7` to `6.5.8` - **Scope** - changes only `/docs/sphinx/requirements.txt` - preserves the existing `rocm-docs-core @ ...@develop` source while advancing the resolved lockfile - **Representative diff** ```txt - cryptography==49.0.0 + cryptography==50.0.1 - gitpython==3.1.55 + gitpython==3.1.61 - tornado==6.5.7 + tornado==6.5.8 ``` Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: gabeweisz <162640284+gabeweisz@users.noreply.github.com>
This pull request extends the `build_docker_vllm.sh` script and adds new
patches to support vLLM versions v0.26.0 through v0.28.0. It introduces
improved graph-capture tracing for CUDA/HIP profiling in these newer
versions, ensuring that all relevant subsystems (decoder, speculator,
encoder) are properly traced. The main changes are grouped as follows:
**Script and Versioning Updates:**
* Updated `build_docker_vllm.sh` to support vLLM versions v26, v27, and
v28, including their corresponding patch files and improved
documentation for tracing behavior in these versions.
[[1]](diffhunk://#diff-e87f62fd28a41719f1735a23b6aacd929284e4f592d3311e5f5ffc13fa24aa26L13-R25)
[[2]](diffhunk://#diff-e87f62fd28a41719f1735a23b6aacd929284e4f592d3311e5f5ffc13fa24aa26R90-R109)
**Graph-Capture Tracing Enhancements (v0.26.0+):**
* Added a new patch (`config_vllm_v0.26.0.patch`) that introduces
`vllm/profiler/graph_capture.py`, providing context-managed helpers to
bind and annotate torch profiler traces during CUDA graph capture for
encoder, decoder, and speculator subsystems.
* Integrated `graph_capture_profiler` and `graph_capture_step` into key
vLLM worker modules to ensure that graph-capture tracing is performed
per subsystem, and trace files are written with subsystem-specific
names.
* Updated tracing logic so that for v0.26.0 and later, the patches focus
on adding graph-capture tracing to areas that upstream vLLM does not yet
cover, rather than duplicating config options now present upstream.
[[1]](diffhunk://#diff-e87f62fd28a41719f1735a23b6aacd929284e4f592d3311e5f5ffc13fa24aa26L13-R25)
[[2]](diffhunk://#diff-e87f62fd28a41719f1735a23b6aacd929284e4f592d3311e5f5ffc13fa24aa26R90-R109)
## Test Result
**Environment:** 1× MI355X (gfx950), TP=1, ROCm 7.2.3, PyTorch 2.11.0,
ROCm base images for
vLLM v0.26.0 / v0.27.0 / v0.28.0 with this change applied. Traces
collected with
`--profiler-config.capture_torch_profiler True
--profiler-config.detailed_trace_annotation True`.
### 1. The gap this closes (v0.26.0, Qwen3-8B +
`AngelSlim/Qwen3-8B_eagle3`, 3 draft tokens)
`capture_torch_profiler` was accepted but silently produced nothing on
the V2 runner, because only
the legacy `GPUModelRunner.capture_model()` path was instrumented:
| Build | Model runner | Graph-capture shards emitted |
|---|---|---|
| stock v0.26.0 | V2 (default for this model) | **0** — no
`capture_traces/` directory created |
| stock v0.26.0 | V1 (`VLLM_USE_V2_MODEL_RUNNER=0`) | 98 |
| **patched** | V2 (default) | **251** (100 decoder + 151 speculator) |
### 2. Coverage matrix (all runs completed, all shards non-empty and
loadable)
| vLLM | Workload | Shards (decoder + subsystem) |
|---|---|---|
| v0.26.0 | Qwen3-8B + EAGLE3, V2 runner | 100 + 151 speculator |
| v0.27.0 | Qwen3-8B + EAGLE3, V2 runner | 100 + 151 speculator |
| v0.28.0 | Qwen3-8B + EAGLE3, V2 runner | 100 + 151 speculator |
| v0.28.0 | Qwen2-VL-2B-Instruct, encoder cudagraphs (budgets
`[256,512]`) | 102 + 2 encoder |
| v0.26.0 | gpt-oss-20b MXFP4 (MoE → legacy V1 runner) | 166 |
The gpt-oss-20b run confirms the legacy V1 path still behaves as before
this change: 166 shards for
83 capture sizes × {PIECEWISE, FULL}.
### 3. Subsystem separation and annotation naming
Each subsystem writes its own trace-file group, so shards from
concurrent capture invocations no
longer share a filename prefix, and every shard carries a
`record_function` annotation identifying
the shape and cudagraph mode:
- `graph_capture_rank_0.*` — `capture_104_FULL`,
`capture_104_PIECEWISE`, … (100 distinct
annotations across 100 shards)
- `graph_capture_rank_0_speculator.*` — `capture_104_draft_FULL`,
`capture_104_draft_PIECEWISE`, …
(151 shards, 102 distinct annotations; the 49 `*_draft_FULL` shapes
captured by both drafter
capture invocations appear once per invocation, each in its own shard
file)
- `graph_capture_rank_0_encoder.*` — `capture_256_encoder_default`,
`capture_512_encoder_default`
<!--
Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved.
See LICENSE for license information.
-->
Co-authored-by: Deval Shah <devashah@amd.com>
## Summary - Add `TraceLens_trace_index` and a `TraceLens.TraceIndex` package for cataloging traces and TraceLens CSV reports into a queryable SQLite index. - Ingest is `append` (one trace) and `build` (a list of traces). Pass `--report-dir` to import an existing CSV report; if omitted, TraceIndex generates a PyTorch performance report from the trace and imports it. - Store parsed `perf_params` / kernel details as JSON. Explode per-op kernels into `op_kernels` (FK `unified_row_id`) and fill `gemm_perf` / `sdpa_perf` / `conv_perf` shape tables so shape questions are SQL filters. - SQLite is the first backend; the table schema is the shared query surface. Documented in the how-to with a schema diagram and example queries. ## Test plan - [x] `python -m pytest tests/test_trace_index.py tests/test_copyright_headers.py -q` - [x] `python -m TraceLens.TraceIndex.cli --help` --------- Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary - Ports `TraceLens_internal/Agent/Analysis/triage` from [AMD-AGI/TraceLens-internal](https://github.com/AMD-AGI/TraceLens-internal) into `TraceLens/Agent/Analysis/triage` so analysis-run checks can ship with the public package. - Retargets module paths from `TraceLens_internal.Agent.Analysis.triage` to `TraceLens.Agent.Analysis.triage` and updates install docs for the public repo. ## Authorship This code was originally written by **Tharun Adithya Srikrishnan** (`@tsrikris`, tsrikris@amd.com) in TraceLens-internal: - [TraceLens Triage](AMD-AGI/TraceLens-internal@85308f4) (2026-06-25) - [TraceLens Traige: Linting](AMD-AGI/TraceLens-internal@2cda09d) (2026-07-09) Source tree: https://github.com/AMD-AGI/TraceLens-internal/commits/main/TraceLens_internal/Agent/Analysis/triage The Git author on this PR’s commit is set to Tharun Adithya Srikrishnan to preserve that attribution. ## Test plan - [ ] `python -c "from TraceLens.Agent.Analysis.triage import run_triage, ALL_CHECKS; print(len(ALL_CHECKS))"` - [ ] `python -m TraceLens.Agent.Analysis.triage.runner --run-dir <analysis_output> --detailed` against a known-good analysis folder - [ ] `bash TraceLens/Agent/Analysis/triage/run_triage.sh <traces_root> ./triage_report 2` against a small batch of analysis outputs - [ ] Confirm Hyperloom/GEAK session checks still run with `--session-dir` when a session tree is available Made with [Cursor](https://cursor.com) --------- Co-authored-by: Tharun Adithya Srikrishnan <tsrikris@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
# Release Notes (9/03) ## Summary This release merges `staging_agent` into `main`, carrying a single feature: a deterministic (no-LLM) fallback analysis path for graph-collapsed inference traces (PR #970). It is additive and parser-compatible — no existing analysis output contract changes. Net: **9 files changed, +1103 / −89**. --- ## Key Features ### 1. Deterministic (no-LLM) Fallback for Graph-Collapsed Traces (PR #970) - Some inference traces are **graph-under-recorded**: a GPU-graph replay or a compiled region collapses the whole workload behind one launch, so the profiler records device kernels but not the per-op decomposition (Python/aten op, shapes, launcher path) the analysis pipeline needs. On these traces the LLM analysis path cannot run at all. - Adds a **trace-quality gate at Step 2**: `check_graph_replay_coverage()` reads the already-written `perf_report_csvs/`, computes the fraction of device time hidden behind graph-replay / compiled-region launches, and trips when it exceeds `GRAPH_REPLAY_FRACTION_MAX`. Benign eager-launch and memcpy plumbing wrappers (present in healthy traces too) are excluded so the signal isolates the pathology. - On a bad verdict, `render_fallback_report()` emits a **parser-compatible `analysis.md`** deterministically: `#### P{rank}:` headings, one `reasoning-candidate` and one `impact-begin kind=p_item` marker per P-item, and the 9-column `**Data:**` table — the same contract the full analysis path emits, so a downstream consumer needs no structural change. No LLM, no GPU. - **Honest unrecoverable cells:** fields a graph-collapsed trace never captured (Operation, Args, Kernel Path, Count, FLOPS/Byte, Efficiency, Bound) render a literal `—` instead of a fabricated value. The raw device symbol is preserved verbatim in `Kernel Name`; the P-item heading shows a display-shortened form. - **P-item count control:** a `%E2E` floor (`MIN_PITEM_PERCENT_E2E`) plus a defensive cap (`MAX_PITEM_COUNT`) collapse noisy traces to the significant few. The percentage denominator is built over all surviving rows before filtering, so dropping the tail never inflates survivors; non-zero drops are reported in the banner, never silent. - **CSV field-size hardening:** the gate reads every trace's CSVs; traces with very large cells previously exceeded `csv`'s default field limit and aborted the gate. A module-level `csv.field_size_limit(...)` is now raised once with a platform-safe clamp (start high, halve on `OverflowError`). - **Pipeline step renumber:** inserting the gate as Step 2 shifted the remaining steps by one, so the orchestrator prep, validation utility, and orchestrator skill / reference / template prose were renumbered in lockstep — prose and print-label only, no logic change. ### 2. Downstream / Contract Impact The change is additive and parser-compatible. The 9-column table header, the `#### P{rank}:` / `reasoning-candidate` / `impact-begin` markers, and their attribute names are all unchanged. The one behavioral note for a downstream consumer is the `—` `Operation` cell: a consumer that today drops rows with an empty Operation should substitute the `Kernel Name` symbol for those rows (a backward-compatible reader relaxation) so fallback reports parse to the intended candidates. --- ## Lines of Code Changed | Section | Files | Insertions | Deletions | Net | |---|---|---|---|---| | Engine (`utils/`) | 4 | +293 | −27 | +266 | | Orchestrator specs / templates | 4 | +94 | −62 | +32 | | Tests | 1 | +716 | 0 | +716 | | **Grand Total** | **9** | **+1103** | **−89** | **+1014** |
<!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> Fix kernel_shape_profiler's torch default-device restore to read the internal override state instead of `torch.get_default_device()` (which always returns a concrete device), preventing a leaked CPU default-device override from corrupting later CUDA tensor creation in the serving path. --------- Co-authored-by: Deval Shah <devashah@amd.com>
Changin SGLang 5.18 patches to apply on the release docker. Moved the older patches targeted to the sglang-dev dockers to different folder. <!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> Co-authored-by: Deval Shah <devashah@amd.com> Co-authored-by: mohbasit <mohbasit@amd.com>
Adds a bundled arch spec for the Radeon 8060S integrated GPU (RDNA 3.5, gfx1151) so roofline analysis works on Ryzen AI Max+. Data is taken from public sources such as https://gpuopen.com/learn/wmma_on_rdna3/. Allow spec files without `memory_gb` field. Specs for APU parts like Strix Halo won't have a fixed `memory_gb` because those use unified memory (sharing with host DDR) that is configurable in the BIOS. --------- Co-authored-by: Tharun Adithya Srikrishnan <tsrikris@amd.com>
New archives as of 09/09/2026 <!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> # Pull Request Template > **Note to AMDers:** > This is a public repository. Please do **not** upload any confidential or customer data. Make sure all such data has been anonymized or removed before making this PR. If you need to attach any private files or links, please insert a Internal OneDrive Link or a Jira Ticket Link instead.
## Summary - Fixes CodeQL `py/clear-text-logging-sensitive-data` on `main` (`tests/manual_test_wheel_agent_install.py`). - Passes `CURSOR_API_KEY` into the container with Docker `-e CURSOR_API_KEY` (copy from host env) instead of embedding the secret in `docker_cmd`, which is printed. ## Test plan - [ ] Confirm CodeQL no longer reports alert #6 after this lands on `main`. - [ ] Optional: run `python tests/manual_test_wheel_agent_install.py` with `CURSOR_API_KEY` set and verify the container still receives the key (Docker `-e NAME` form). Made with [Cursor](https://cursor.com) Co-authored-by: Cursor <cursoragent@cursor.com>
<!-- Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> ## Summary Documents Codex as a supported agent runner for the TraceLens Analysis Agent, now that it has been verified end-to-end against the orchestrator. The two user-facing "supported runners" notes now name Claude Code, Cursor, and Codex, and the orchestrator's subagent rule no longer assumes the Claude-only "Task tool" primitive. Docs-only: 3 files changed, +3 / -5. ## Why The orchestrator skill and its docs described a portable, runner-agnostic workflow but only ever named runners implicitly, and the one operational rule that mattered for portability — how to spawn subagents — was written as "use the Task tool", which is Claude Code / Cursor terminology. A Codex user following the skill hit a primitive that does not exist under their runner (`spawn_agent`) with no translation. These edits close that gap without changing any workflow behavior. ## What changed ### Docs (3 files) | File | Change | |---|---| | `TraceLens/Agent/Analysis/README.md` | Quick Start portability note now names Claude Code, Cursor, or Codex as example runners; merged the adjacent skill-paths note into the same block. | | `docs/how-to/agent.md` | Published how-to "Run the agent from a chat" note mirrors the README wording, naming the three runners. | | `TraceLens/Agent/Analysis/skills/analysis-orchestrator/SKILL.md` | Subagents rule reworded from "Use the Task tool" to "Use your runner's subagent primitive (the Task tool in Claude Code or Cursor, `spawn_agent` under Codex)". | ## Net diff ``` 3 files changed, 3 insertions(+), 5 deletions(-) ```
Extends the patch series to v0.29. Same five files as v0.28: adds vllm/profiler/graph_capture.py and wraps the decoder, speculator and encoder cudagraph capture paths so capture_torch_profiler emits per-subsystem traces on the V2 model runner. Three changes upstream made between v0.28.0 and v0.29 needed resolving: torch.cuda.graph() now takes stream=current_stream() in both the encoder and decoder capture sites, CudaGraphManager.capture() samples free memory around each capture via _capture_mem_samples, and GPUModelRunner passes input buffers through pcp_manager. Generated against v0.29.0rc2 and verified against the v0.29.0 release, whose copies of the four modified files are byte-identical to rc2, so the patch applies with zero fuzz. Tested on vllm/vllm-openai-rocm:v0.29.0 (vLLM 0.29.0, Python 3.12.13, torch 2.12): Qwen3-8B with an EAGLE3 drafter produced 455 capture shards (153 decoder, 302 speculator) over 240/240 requests, and gpt-oss-20b produced 166 decoder shards over 320/320 requests, both with unique annotation keys and an intact serving trace. <!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> # Pull Request Template > **Note to AMDers:** > This is a public repository. Please do **not** upload any confidential or customer data. Make sure all such data has been anonymized or removed before making this PR. If you need to attach any private files or links, please insert a Internal OneDrive Link or a Jira Ticket Link instead.
…ve finder approach (#979) A self-contained, Python package (`TraceLens/TraceUtils/kernel_source/`) that takes a GPU kernel name from a trace and returns the editable source file and line where it's defined, plus whether it's patchable. It looks kernels up by symbol rather than by file path, so it keeps working even when frameworks are reinstalled or upgraded. The package is fully additive and nothing in TraceLens imports it yet, so it can't break anything. ## Files ### New package — `TraceLens/TraceUtils/kernel_source/` (11 files) | File | Purpose | |---|---| | `__init__.py` | Public API exports | | `datatypes.py` | Shared dataclasses (dep-free, breaks import cycles) | | `demangle.py` | 3-tier symbol demangling | | `patchability.py` | Level-0 gate (no I/O) | | `editable.py` | Editable-vs-generated path classifier (includes vLLM `torch_compile_cache`/`inductor_cache` marker fix) | | `index.py` | csrc symbol indexer + cache (includes comment-stripping fix) | | `resolver.py` | Level-1 native symbol resolution | | `triton_pin.py` | Triton `.py` AST line-pinning | | `discovery.py` | Active-finder framework tree discovery | | `contract.py` | Public contract surface | | `cli.py` | `TraceLens_resolve_kernel_source` entry point | ### Tests — `tests/` (6 files) - `test_kernel_source_pipeline.py` — 58 tests, end-to-end across all 9 stages (comprehensive suite) - `test_kernel_source_contract.py`, `_discovery.py`, `_gate.py`, `_index.py`, `_triton.py` — focused per-module tests ### Tooling (1 file) - `tools/validate_kernel_source.py` — real-world parity harness (verdicts vs ground-truth `patchable`) ### Modified (1 file) - `setup.py` — adds optional `kernel_source` extra (`itanium-demangler`) + `TraceLens_resolve_kernel_source` console script ## How downstream tooling calls this for source mapping Downstream optimization tooling consumes this as a library via a thin adapter, selected behind an opt-in env flag: ```python from TraceLens.TraceUtils.kernel_source import resolve result = resolve( "<KERNEL_NAME>", # kernel_name: device symbol from the trace (mangled or plain) ["<SEARCH_PATH_1>", "..."], # search_paths: dirs to search; pass None to auto-discover op_name="<OP_NAME>", # optional: launching op name, helps the gate call_stack=[], # optional: call-stack frames, helps the gate run_gate=True, # optional: set False to skip the gate and resolve directly ) print(result.source_file, result.line, result.patchable) ``` Flow: gate → resolve survivors against the currently-installed framework trees → normalize the verdict into the caller's fields. Validated on a real 35-kernel trace: 7 hit / 7 CK-gated / 19 pass-through / 0 disagreements vs the legacy resolver. ## Running the tests ```bash pip install -e '.[dev,kernel_source]' python -m pytest tests/test_kernel_source_pipeline.py -v # 58 tests, self-contained ``` Minimal container (no heavy deps required): ```bash docker run --rm -v "$PWD":/work -w /work python:3.11-slim bash -lc "pip install -q pytest && python -m pytest tests/test_kernel_source_pipeline.py -v" ``` --------- Co-authored-by: Hasssan <ahasssan@ctr2-alola-ctrl-01.amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary Governance and docs cleanup: - `AGENTS.md` is now the single authoring contract, and `CLAUDE.md` collapses to a one-line pointer at it. - Trace-collection guidance moves from the standalone `collect-traces.md` page into the two report docs readers already land on; the orphan page is removed. - The Project Structure tree moves from `CONTRIBUTING.md` into `README.md`, corrected to match disk. ## Changes - **AGENTS.md / CLAUDE.md.** Rewrote `AGENTS.md` as a lean rules-of-engagement contract (change scope, reuse, correctness, hygiene); reduced `CLAUDE.md` to `Read [AGENTS.md](AGENTS.md).` - **Trace collection.** Capture snippet and profiler-flags table now open `generate-perf-report-pytorch.md` as `## Collect a trace`; the sanity-check list is now `## Before you trust a report` in the inference report doc. Removed `collect-traces.md` and repointed every reference (`README.md`, `docs/index.rst`, `docs/how-to/agent.md`, Agent `Analysis/README.md`) at the new `#collect-a-trace` anchor. - **Project Structure tree.** Moved to `README.md`, fixing the unclosed-fence bug and dead Black ToC entries in `CONTRIBUTING.md`. Added the missing `Agent/`, `TraceIndex/`, `TraceUtils/`, `notebooks/`, and `scripts/` dirs; fixed the tree glyph and the `GPUEventAnalyser` spelling.
## Problem
`_arch_product_name` (`TraceLens/PerfModel/benchmarking/microbench.py`)
decides the `name` field of the arch JSON that `microbench` generates.
It had two paths:
1. A memory-tier heuristic (`>= 280 -> MI355X`, `>= 180 -> MI300X`).
2. Otherwise, the last whitespace-separated word of the device string.
For client parts, the device string `"AMD Radeon 8060S Graphics"` is
turned into
```json
{ "name": "Graphics", ... }
```
This PR fixes it to extract `"Radeon_8060S"`
Co-authored-by: Tharun Adithya Srikrishnan <tsrikris@amd.com>
Adding a patch for vLLM nightly with ROCm 10.0 vllm/vllm-openai-rocm:nightly-rocm100-dc36fcce902a63eab06c1b93a5c4a5ee178a0c56 <!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> Co-authored-by: Deval Shah <devashah@amd.com>
Adds @Ahmedhasssan-aig as code owner for `TraceLens/TraceUtils/kernel_source/` so changes to the kernel source mapping utility require review from the area owner. Co-authored-by: Hasssan <ahasssan@ctr2-alola-ctrl-01.amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Syncing up with the main branch