Conversation
## Summary - Add `TraceLens_trace_index` and a `TraceLens.TraceIndex` package for cataloging traces and TraceLens CSV reports into a queryable SQLite index. - Ingest is `append` (one trace) and `build` (a list of traces). Pass `--report-dir` to import an existing CSV report; if omitted, TraceIndex generates a PyTorch performance report from the trace and imports it. - Store parsed `perf_params` / kernel details as JSON. Explode per-op kernels into `op_kernels` (FK `unified_row_id`) and fill `gemm_perf` / `sdpa_perf` / `conv_perf` shape tables so shape questions are SQL filters. - SQLite is the first backend; the table schema is the shared query surface. Documented in the how-to with a schema diagram and example queries. ## Test plan - [x] `python -m pytest tests/test_trace_index.py tests/test_copyright_headers.py -q` - [x] `python -m TraceLens.TraceIndex.cli --help` --------- Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary - Ports `TraceLens_internal/Agent/Analysis/triage` from [AMD-AGI/TraceLens-internal](https://github.com/AMD-AGI/TraceLens-internal) into `TraceLens/Agent/Analysis/triage` so analysis-run checks can ship with the public package. - Retargets module paths from `TraceLens_internal.Agent.Analysis.triage` to `TraceLens.Agent.Analysis.triage` and updates install docs for the public repo. ## Authorship This code was originally written by **Tharun Adithya Srikrishnan** (`@tsrikris`, tsrikris@amd.com) in TraceLens-internal: - [TraceLens Triage](AMD-AGI/TraceLens-internal@85308f4) (2026-06-25) - [TraceLens Traige: Linting](AMD-AGI/TraceLens-internal@2cda09d) (2026-07-09) Source tree: https://github.com/AMD-AGI/TraceLens-internal/commits/main/TraceLens_internal/Agent/Analysis/triage The Git author on this PR’s commit is set to Tharun Adithya Srikrishnan to preserve that attribution. ## Test plan - [ ] `python -c "from TraceLens.Agent.Analysis.triage import run_triage, ALL_CHECKS; print(len(ALL_CHECKS))"` - [ ] `python -m TraceLens.Agent.Analysis.triage.runner --run-dir <analysis_output> --detailed` against a known-good analysis folder - [ ] `bash TraceLens/Agent/Analysis/triage/run_triage.sh <traces_root> ./triage_report 2` against a small batch of analysis outputs - [ ] Confirm Hyperloom/GEAK session checks still run with `--session-dir` when a session tree is available Made with [Cursor](https://cursor.com) --------- Co-authored-by: Tharun Adithya Srikrishnan <tsrikris@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
# Release Notes (9/03) ## Summary This release merges `staging_agent` into `main`, carrying a single feature: a deterministic (no-LLM) fallback analysis path for graph-collapsed inference traces (PR #970). It is additive and parser-compatible — no existing analysis output contract changes. Net: **9 files changed, +1103 / −89**. --- ## Key Features ### 1. Deterministic (no-LLM) Fallback for Graph-Collapsed Traces (PR #970) - Some inference traces are **graph-under-recorded**: a GPU-graph replay or a compiled region collapses the whole workload behind one launch, so the profiler records device kernels but not the per-op decomposition (Python/aten op, shapes, launcher path) the analysis pipeline needs. On these traces the LLM analysis path cannot run at all. - Adds a **trace-quality gate at Step 2**: `check_graph_replay_coverage()` reads the already-written `perf_report_csvs/`, computes the fraction of device time hidden behind graph-replay / compiled-region launches, and trips when it exceeds `GRAPH_REPLAY_FRACTION_MAX`. Benign eager-launch and memcpy plumbing wrappers (present in healthy traces too) are excluded so the signal isolates the pathology. - On a bad verdict, `render_fallback_report()` emits a **parser-compatible `analysis.md`** deterministically: `#### P{rank}:` headings, one `reasoning-candidate` and one `impact-begin kind=p_item` marker per P-item, and the 9-column `**Data:**` table — the same contract the full analysis path emits, so a downstream consumer needs no structural change. No LLM, no GPU. - **Honest unrecoverable cells:** fields a graph-collapsed trace never captured (Operation, Args, Kernel Path, Count, FLOPS/Byte, Efficiency, Bound) render a literal `—` instead of a fabricated value. The raw device symbol is preserved verbatim in `Kernel Name`; the P-item heading shows a display-shortened form. - **P-item count control:** a `%E2E` floor (`MIN_PITEM_PERCENT_E2E`) plus a defensive cap (`MAX_PITEM_COUNT`) collapse noisy traces to the significant few. The percentage denominator is built over all surviving rows before filtering, so dropping the tail never inflates survivors; non-zero drops are reported in the banner, never silent. - **CSV field-size hardening:** the gate reads every trace's CSVs; traces with very large cells previously exceeded `csv`'s default field limit and aborted the gate. A module-level `csv.field_size_limit(...)` is now raised once with a platform-safe clamp (start high, halve on `OverflowError`). - **Pipeline step renumber:** inserting the gate as Step 2 shifted the remaining steps by one, so the orchestrator prep, validation utility, and orchestrator skill / reference / template prose were renumbered in lockstep — prose and print-label only, no logic change. ### 2. Downstream / Contract Impact The change is additive and parser-compatible. The 9-column table header, the `#### P{rank}:` / `reasoning-candidate` / `impact-begin` markers, and their attribute names are all unchanged. The one behavioral note for a downstream consumer is the `—` `Operation` cell: a consumer that today drops rows with an empty Operation should substitute the `Kernel Name` symbol for those rows (a backward-compatible reader relaxation) so fallback reports parse to the intended candidates. --- ## Lines of Code Changed | Section | Files | Insertions | Deletions | Net | |---|---|---|---|---| | Engine (`utils/`) | 4 | +293 | −27 | +266 | | Orchestrator specs / templates | 4 | +94 | −62 | +32 | | Tests | 1 | +716 | 0 | +716 | | **Grand Total** | **9** | **+1103** | **−89** | **+1014** |
<!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> Fix kernel_shape_profiler's torch default-device restore to read the internal override state instead of `torch.get_default_device()` (which always returns a concrete device), preventing a leaked CPU default-device override from corrupting later CUDA tensor creation in the serving path. --------- Co-authored-by: Deval Shah <devashah@amd.com>
Changin SGLang 5.18 patches to apply on the release docker. Moved the older patches targeted to the sglang-dev dockers to different folder. <!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> Co-authored-by: Deval Shah <devashah@amd.com> Co-authored-by: mohbasit <mohbasit@amd.com>
Adds a bundled arch spec for the Radeon 8060S integrated GPU (RDNA 3.5, gfx1151) so roofline analysis works on Ryzen AI Max+. Data is taken from public sources such as https://gpuopen.com/learn/wmma_on_rdna3/. Allow spec files without `memory_gb` field. Specs for APU parts like Strix Halo won't have a fixed `memory_gb` because those use unified memory (sharing with host DDR) that is configurable in the BIOS. --------- Co-authored-by: Tharun Adithya Srikrishnan <tsrikris@amd.com>
New archives as of 09/09/2026 <!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> # Pull Request Template > **Note to AMDers:** > This is a public repository. Please do **not** upload any confidential or customer data. Make sure all such data has been anonymized or removed before making this PR. If you need to attach any private files or links, please insert a Internal OneDrive Link or a Jira Ticket Link instead.
## Summary - Fixes CodeQL `py/clear-text-logging-sensitive-data` on `main` (`tests/manual_test_wheel_agent_install.py`). - Passes `CURSOR_API_KEY` into the container with Docker `-e CURSOR_API_KEY` (copy from host env) instead of embedding the secret in `docker_cmd`, which is printed. ## Test plan - [ ] Confirm CodeQL no longer reports alert #6 after this lands on `main`. - [ ] Optional: run `python tests/manual_test_wheel_agent_install.py` with `CURSOR_API_KEY` set and verify the container still receives the key (Docker `-e NAME` form). Made with [Cursor](https://cursor.com) Co-authored-by: Cursor <cursoragent@cursor.com>
<!-- Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> ## Summary Documents Codex as a supported agent runner for the TraceLens Analysis Agent, now that it has been verified end-to-end against the orchestrator. The two user-facing "supported runners" notes now name Claude Code, Cursor, and Codex, and the orchestrator's subagent rule no longer assumes the Claude-only "Task tool" primitive. Docs-only: 3 files changed, +3 / -5. ## Why The orchestrator skill and its docs described a portable, runner-agnostic workflow but only ever named runners implicitly, and the one operational rule that mattered for portability — how to spawn subagents — was written as "use the Task tool", which is Claude Code / Cursor terminology. A Codex user following the skill hit a primitive that does not exist under their runner (`spawn_agent`) with no translation. These edits close that gap without changing any workflow behavior. ## What changed ### Docs (3 files) | File | Change | |---|---| | `TraceLens/Agent/Analysis/README.md` | Quick Start portability note now names Claude Code, Cursor, or Codex as example runners; merged the adjacent skill-paths note into the same block. | | `docs/how-to/agent.md` | Published how-to "Run the agent from a chat" note mirrors the README wording, naming the three runners. | | `TraceLens/Agent/Analysis/skills/analysis-orchestrator/SKILL.md` | Subagents rule reworded from "Use the Task tool" to "Use your runner's subagent primitive (the Task tool in Claude Code or Cursor, `spawn_agent` under Codex)". | ## Net diff ``` 3 files changed, 3 insertions(+), 5 deletions(-) ```
Extends the patch series to v0.29. Same five files as v0.28: adds vllm/profiler/graph_capture.py and wraps the decoder, speculator and encoder cudagraph capture paths so capture_torch_profiler emits per-subsystem traces on the V2 model runner. Three changes upstream made between v0.28.0 and v0.29 needed resolving: torch.cuda.graph() now takes stream=current_stream() in both the encoder and decoder capture sites, CudaGraphManager.capture() samples free memory around each capture via _capture_mem_samples, and GPUModelRunner passes input buffers through pcp_manager. Generated against v0.29.0rc2 and verified against the v0.29.0 release, whose copies of the four modified files are byte-identical to rc2, so the patch applies with zero fuzz. Tested on vllm/vllm-openai-rocm:v0.29.0 (vLLM 0.29.0, Python 3.12.13, torch 2.12): Qwen3-8B with an EAGLE3 drafter produced 455 capture shards (153 decoder, 302 speculator) over 240/240 requests, and gpt-oss-20b produced 166 decoder shards over 320/320 requests, both with unique annotation keys and an intact serving trace. <!-- Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved. See LICENSE for license information. --> # Pull Request Template > **Note to AMDers:** > This is a public repository. Please do **not** upload any confidential or customer data. Make sure all such data has been anonymized or removed before making this PR. If you need to attach any private files or links, please insert a Internal OneDrive Link or a Jira Ticket Link instead.
…ve finder approach (#979) A self-contained, Python package (`TraceLens/TraceUtils/kernel_source/`) that takes a GPU kernel name from a trace and returns the editable source file and line where it's defined, plus whether it's patchable. It looks kernels up by symbol rather than by file path, so it keeps working even when frameworks are reinstalled or upgraded. The package is fully additive and nothing in TraceLens imports it yet, so it can't break anything. ## Files ### New package — `TraceLens/TraceUtils/kernel_source/` (11 files) | File | Purpose | |---|---| | `__init__.py` | Public API exports | | `datatypes.py` | Shared dataclasses (dep-free, breaks import cycles) | | `demangle.py` | 3-tier symbol demangling | | `patchability.py` | Level-0 gate (no I/O) | | `editable.py` | Editable-vs-generated path classifier (includes vLLM `torch_compile_cache`/`inductor_cache` marker fix) | | `index.py` | csrc symbol indexer + cache (includes comment-stripping fix) | | `resolver.py` | Level-1 native symbol resolution | | `triton_pin.py` | Triton `.py` AST line-pinning | | `discovery.py` | Active-finder framework tree discovery | | `contract.py` | Public contract surface | | `cli.py` | `TraceLens_resolve_kernel_source` entry point | ### Tests — `tests/` (6 files) - `test_kernel_source_pipeline.py` — 58 tests, end-to-end across all 9 stages (comprehensive suite) - `test_kernel_source_contract.py`, `_discovery.py`, `_gate.py`, `_index.py`, `_triton.py` — focused per-module tests ### Tooling (1 file) - `tools/validate_kernel_source.py` — real-world parity harness (verdicts vs ground-truth `patchable`) ### Modified (1 file) - `setup.py` — adds optional `kernel_source` extra (`itanium-demangler`) + `TraceLens_resolve_kernel_source` console script ## How downstream tooling calls this for source mapping Downstream optimization tooling consumes this as a library via a thin adapter, selected behind an opt-in env flag: ```python from TraceLens.TraceUtils.kernel_source import resolve result = resolve( "<KERNEL_NAME>", # kernel_name: device symbol from the trace (mangled or plain) ["<SEARCH_PATH_1>", "..."], # search_paths: dirs to search; pass None to auto-discover op_name="<OP_NAME>", # optional: launching op name, helps the gate call_stack=[], # optional: call-stack frames, helps the gate run_gate=True, # optional: set False to skip the gate and resolve directly ) print(result.source_file, result.line, result.patchable) ``` Flow: gate → resolve survivors against the currently-installed framework trees → normalize the verdict into the caller's fields. Validated on a real 35-kernel trace: 7 hit / 7 CK-gated / 19 pass-through / 0 disagreements vs the legacy resolver. ## Running the tests ```bash pip install -e '.[dev,kernel_source]' python -m pytest tests/test_kernel_source_pipeline.py -v # 58 tests, self-contained ``` Minimal container (no heavy deps required): ```bash docker run --rm -v "$PWD":/work -w /work python:3.11-slim bash -lc "pip install -q pytest && python -m pytest tests/test_kernel_source_pipeline.py -v" ``` --------- Co-authored-by: Hasssan <ahasssan@ctr2-alola-ctrl-01.amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary Governance and docs cleanup: - `AGENTS.md` is now the single authoring contract, and `CLAUDE.md` collapses to a one-line pointer at it. - Trace-collection guidance moves from the standalone `collect-traces.md` page into the two report docs readers already land on; the orphan page is removed. - The Project Structure tree moves from `CONTRIBUTING.md` into `README.md`, corrected to match disk. ## Changes - **AGENTS.md / CLAUDE.md.** Rewrote `AGENTS.md` as a lean rules-of-engagement contract (change scope, reuse, correctness, hygiene); reduced `CLAUDE.md` to `Read [AGENTS.md](AGENTS.md).` - **Trace collection.** Capture snippet and profiler-flags table now open `generate-perf-report-pytorch.md` as `## Collect a trace`; the sanity-check list is now `## Before you trust a report` in the inference report doc. Removed `collect-traces.md` and repointed every reference (`README.md`, `docs/index.rst`, `docs/how-to/agent.md`, Agent `Analysis/README.md`) at the new `#collect-a-trace` anchor. - **Project Structure tree.** Moved to `README.md`, fixing the unclosed-fence bug and dead Black ToC entries in `CONTRIBUTING.md`. Added the missing `Agent/`, `TraceIndex/`, `TraceUtils/`, `notebooks/`, and `scripts/` dirs; fixed the tree glyph and the `GPUEventAnalyser` spelling.
tsrikris
requested review from
ajassani,
devalshahamd and
gabeweisz
as code owners
September 14, 2026 22:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every operation row in
analysis.mdnow carries its ownimpact_score— a per-rowdecomposition of the group-summed P-item card impact, in the same
%E2Ecurrency. Thescore is stamped at source and surfaced as a single end-of-table marker per compute
Data table, so downstream consumers (e.g. Hyperloom) can rank individual kernel rows by
impact instead of inheriting the card total. No existing impact number changes — the card
scores are byte-identical; this adds per-row visibility on top.
What changed
_row_impact_score(op, baseline_ms, comparison_scope)inanalysis_utils.pybecomes the single owner of the impact ladder(quantified roofline-gap branch, heuristic branch).
build_operation_metricscalls it tostamp
impact_score+impact_estimate_methodon each op;compute_impact_estimatesisrefactored to call the same helper and derive its low/high band as fixed ratios of the
mid — its output stays byte-identical.
op_rowmarker. Compute analyzers emit one paired marker after each compute-tier**Data:**table:<!-- impact-begin kind=op_row rank=N impacts=v0,v1,... -->…<!-- impact-end -->— a CSV of per-rowimpact_scorein table order,—for null, onevalue per row. No new table column; the values live only in the marker. Instruction added
to
sub_agent_spec.mdalongside the existing marker instructions.op_row.MarkerValidator(validation_utils.py) gains theop_rowkind (KNOWN_KINDS,REQUIRED_ATTRS_BY_KIND) plus a CSV-count-vs-row-countcheck;
workflow_scripted_evals.pygainsmarker_eval_5(theop_roweval) in itsregistry. CSV parsing in both is whitespace-tolerant.
op_rowcases inMarkerValidatorandmarker_eval_5test suites; per-rowadditivity assertion (Σ member rows == card within
_ROLLUP_IMPACT_TOL); positive stampassertions in
test_build_operation_metricsincl. theNonecases.Design notes:
_row_impact_scoredeliberately excludes themin_impact_scorenoise floor(a row may legitimately carry a sub-floor score — that's correct visibility; the floor is an
estimates-array filtering policy). It stamps mid + method only (no per-row low/high — nothing
renders them). The stamp lands after the fusion-map block so it reads
fusion_flagged.Comparative mode
No carve-out needed. In comparative,
impact_scoreis a t2/t1 ratio-gap (regressed rowsstamp 0) — inherited from
compute_impact_estimates, not introduced here. The heuristicbranch is standalone-only, so comparative rows either take the quantified branch or stamp
None; additivity still holds. Downstream readers skip comparative Data tables by headergate, so the marker is inert there.
Files
category_analyses/analysis_utils.py_row_impact_scorehelper + 2 call sites; stamp inbuild_operation_metricsskills/analysis-orchestrator/templates/sub_agent_spec.mdop_rowmarker instructionutils/validation_utils.pyMarkerValidatorop_rowkind + CSV-count checkagent_evals/Analysis/eval_utils/workflow_scripted_evals.pymarker_eval_5tests/test_analysis_agent_utils.py,tests/test_analysis_agent_evals.pySeparately, TraceLens-internal
agent_extension.pygets a 2-line comment only: theop_rowmarker is consumer-facing and intentionally not rehydrated (rows carry
time_msdirectly);_render_legacyalready returnsNonefor unknown kinds, so no logic change is needed.Test plan
pytest tests/test_analysis_agent_utils.py tests/test_analysis_agent_category_utils.py tests/test_analysis_agent_evals.py— 596 passed.compute_impact_estimatestests(
test_analysis_agent_utils.py:316-419,test_analysis_agent_category_utils.py:500-627)pass unchanged.
MarkerValidatorop_row: valid passes; missingrank/impacts, unpaired, andCSV-count mismatch all fail.
impact_score== card mid within_ROLLUP_IMPACT_TOL.op_rowmarkers; multi-row card3.55+3.29+1.77 = 8.61== gemm P1 card mid.op_rowmarkers; 5-row card3.22+1.86+1.12+0.73+0.15 = 7.08== gemmcard mid.
kind=p_item(category, low, mid, high)triplet identical to the priorbaseline;
marker_eval_5PASS; no structural regression.black+ copyright headers on changed files.