Semantic analysis agent and evals by amoskvic · Pull Request #951 · AMD-AGI/TraceLens

amoskvic · 2026-08-18T09:06:05Z

Overview

This PR introduces a revamped semantic analysis approach (fallback for the comparative graph mode, when there is no graph capture), as well as a new type of eval (partial eval): evaluation based on a partial execution of the general agentic workflow, with pass/fail determined by quality metrics derived from a gold standard.

General approach

The philosophy was to try to keep the changes to the main orchestrator to the absolute minimum. When the orchestrator detects the fallback condition (comparative + graph, but no capture info), a semantic subagent is invoked. The semantic subagent produces a diff_stats file using an LLM-driven analysis, with an analogous structure to a regular diff_stats. After that, the main orchestrator picks up from there (the new --precomputed_diff_stats_csv flag on generate_perf_report_pytorch.py / generate_perf_report_pytorch_inference.py (skips the internal TraceDiff and loads the semantic diff_stats.csv directly)).

Structure

The evals live in agent_evals/Analysis/partial_tests/, a new folder alongside the existing full-orchestrator eval harness. The main entry point is agent_evals/Analysis/partial_tests/run_partial_tests.sh. It invokes the semantic-comparison-agent workflow on the DECODE-only trace pairs listed in partial_test_cases.csv (one row per model: DeepSeek-R1, Qwen3-30B-A3B), then compares the resulting diff_stats.csv with the gold standard one (analysis_output_ref/semantic_purity_gold_diff_stats.csv, obtained by running the usual comparative mode with capture info on the same DECODE trace pair). Helper functions in agent_evals/Analysis/eval_utils/compare_lca_partitions.py (shared with tests/test_compare_lca_partitions.py) are primarily for computing the quality metrics to compare two different diff_stats files; partial_tests/eval_utils/semantic_partition_scripted_evals.py records per-run metrics (informational), and partial_tests/eval_utils/semantic_purity_aggregate.py is the actual pass/fail gate (averages strict_forward across repeats against a per-model floor).

The semantic agent logic itself lives in TraceLens/Agent/Analysis/skills/analysis-orchestrator/agents/semantic-comparison-agent.md (plus its three sub-agents: kernel-stem-preprocessing-agent.md, kernel-unification-agent.md, kernel-coherence-agent.md), with helper python functions in TraceLens/Agent/Analysis/semantic_analyses/ (extract_trace_data.py, extract_tree_context.py, pattern_finder.py, build_semantic_labels.py, kernel_unification.py, kernel_coherence.py, generate_semantic_diff.py, match_and_compare.py) and TraceLens/Agent/Analysis/utils/classify_kernels.py.

Validation

The changes were validated using the approach described above, with the current solution strongly outperforming the previous semantic analysis approach and a number of simpler baselines (e.g. random kernel grouping). One limitation: as of the time of the creation of this PR, the tests were run from a local agent, not using the on-cluster docker. This extra verification step is in progress.

Extra notes:

Removed dead code superseded by this approach: the old block-alignment harmonization
agent (semantic-harmonization-agent.md) and its python backend (harmonization.py).
On the semantic-purity gate floors: set below the worst single observed run
with a small margin (see semantic_purity_aggregate.py docstring).

…tic agent; will require iteration

…unts mismatch AMD capture traces sometimes record extra HIP dispatch API calls (hipExtModuleLaunchKernel, hipLaunchKernel) that get consolidated during graph optimization, causing a count mismatch vs graph kernels. Previously the merge was skipped entirely for that execution root, leaving its kernels as flat children of hipGraphLaunch with no cpu_op ancestry. Adds _align_capture_to_graph which scans forward through capture dispatch events to match each graph kernel by name, discarding any unmatched extras. verify_subtree_events now returns a (success, aligned_capture_events) tuple so the caller can proceed with the aligned subset when counts differ but names align. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

…rnel dispatcher

…aph_capture_changes

amoskvic · 2026-09-03T11:34:31Z

Update

Addressed all code/prompt quality comments.
Added unit tests.
Stripped away lots of code that was either outdated or scaffolding for future cross-framework comparisons.
Streamlined control flow, removing the overly complicated & costly machinery for "detecting" the need for semantic analysis.
Removed unnecessary trace loads.

Some things to be addressed in the future:

The pattern finder needs to be replaced with the updated SILO team implementation (if it proves usable for the task).
The pattern finder is currently excluded from unit testing as it should be replaced in the future.
Trace splitting should be deduplicated and replaced with the main trace split implementation. This might require modifications to the trace split functionality/api, and I'd like compartmentalize it into a separate follow-up PR. This PR, while large, is self-contained, and, for the most part, doesn't modify existing behavior outside of semantic analysis. I think it'd be good to keep it this way.

A note on comments:
There are a few (very few) comment threads I left unresolved where input would be helpful.

The rest is resolved. When I didn't write out a justification, the reasons for resolving them were either "the file/function in question is stripped away entirely," or "the code is improved and the issue fixed".

tsrikris · 2026-09-03T17:37:34Z

+   - If **Graph replay + capture**, ask for the **Capture Folder Path(s)**:
+     - `standalone`: one folder → `<capture_folder_path_1>`. Ask: "Please provide the full path to the graph capture traces folder"
+     - `comparative`: one folder per trace → `<capture_folder_path_1>` (primary/trace1) and `<capture_folder_path_2>` (comparison/trace2). Ask: "Please provide the graph capture traces folder for the primary trace and for the comparison trace."
+   - **Comparative + graph replay** (do not abort): collect capture folders for both traces when available. If capture is not available, the comparison uses the semantic path (see Step 0.5).


"do not abort" seems redundant

I would instead reframe this section as, keeping 2 as is, something like
3. Graph replay only (<inference_exec_mode> = graph_replay_only):

standalone: Abort

comparative: Semantic analysis

tsrikris · 2026-09-03T17:38:58Z


+## Step 0.5: Comparison Method (comparative only)
+
+For `standalone`, skip this step.


This might be removable with the prev. comment

tsrikris · 2026-09-03T17:39:47Z

 ## Step 1: Generate Performance Report

-Use **`<analysis_mode>`** to determine which CLI tool to run and then **`<comparison_scope>`** to determine arguments.
+Use **`<analysis_mode>`** to determine which CLI tool to run and then **`<comparison_scope>`** (and, for comparative, **`<comparison_method>`** from Step 0.5) to determine arguments.


Use <analysis_mode> to determine which CLI tool to run and then <comparison_scope> to determine arguments. can stay as is

tsrikris · 2026-09-03T17:41:29Z


+## Step 1.5: Semantic Comparative Ordering (`<comparison_method>` = `semantic` only)
+
+When `<comparison_scope>` = `comparative` and `<comparison_method>` = `semantic`, run Step 1 in this order:


<inference_exec_mode> = graph_replay_only and scope comparative directly implies semantic, would just simplify the verbiage here

'''
Step 1.5: Semantic Comparative Ordering (<inference_exec_mode> = graph_replay_only )

....

...

'''

tsrikris · 2026-09-03T17:43:55Z

 - **GPU Graph Replay**: raw trace JSON contains `hipGraphLaunch` or `cudaGraphLaunch`.
-  - **Default mode** (analysis_mode = `default`): Inform the user with `[DIAG:trace_quality:GPU_GRAPH_REPLAY]` that GPU graph replay was detected and that the default analysis mode supports typical PyTorch traces. **Abort** -- do not retry or continue.
-  - **Inference mode** (analysis_mode = `inference`): Graph launches are expected and supported if graph capture folder is provided, do not abort. If inference_exec_mode is `eager` (no capture folder was provided), continue.
+  - **Comparative scope**: graph replay is **supported**; the comparison method (tracediff+capture vs semantic) was set in Step 0.5 from capture availability, with no trace classification. Ensure capture folders were collected for both traces when available.


would just remove 692-696. Seems redundant with all that's been added

tsrikris · 2026-09-03T17:47:15Z

@@ -0,0 +1,137 @@
+<!--
+Copyright (c) 2024 - 2025 Advanced Micro Devices, Inc. All rights reserved.


Note to change copyright dates to 2026

tsrikris · 2026-09-03T17:48:36Z

+```json
+{
+  "context_renames": {
+    "(platform 1 name):6": "qkv_projection",


Feedback for follow up PR: Create a util function for this along with a validator, JSON generation leads to token wastage.

tsrikris · 2026-09-03T17:55:50Z

@@ -0,0 +1,75 @@
+###############################################################################
+# Copyright (c) 2024 - 2025 Advanced Micro Devices, Inc. All rights reserved.


Copyright flag

tsrikris · 2026-09-03T17:56:20Z

+def parse_filename_metadata(filepath: str) -> Dict[str, Any]:
+    """
+    Parse isl, osl, conc, tp from trace filename.
+    Example: mi355_tp1_isl1024_osl8_conc4_opt_asm64x256.pt.trace.json.gz


Recommend removing trace name

tsrikris · 2026-09-03T17:59:44Z

+    return result, kernels
+
+
+def _write_split_regions(split_result, trace_path, output_dir):  # pragma: no cover


Clarifying question: are all intermediates for semantic stored inside a separate directory? Just making sure it's inside a sub-dir in the results directory of the agent

tsrikris · 2026-09-03T18:01:45Z

+    return n
+
+
+def load_json(path):


I believe there's a top level DataLoader class for opening traces

tsrikris · 2026-09-03T18:03:28Z

Can this be merged with another file? Or do you prefer leaving it separate

tsrikris · 2026-09-03T18:03:50Z

+
+
+def _load(path):
+    return load_json(path)


Redundant load? Can directly call helper function

tsrikris · 2026-09-03T18:06:05Z

    TraceLens/PerfModel/benchmarking/*
    TraceLens/PerfModel/origami_helper.py
    TraceLens/PerfModel/run_perf_model.py
+    # pattern_finder.py is slated for replacement by an external package; omit until then


Note for follow up PR

tsrikris · 2026-09-03T18:06:56Z

Recommending aligning file nomenclature with other set of evals

tsrikris · 2026-09-03T18:09:06Z

@@ -0,0 +1,8 @@
+# Expanded fixtures -- regenerated from fixtures/*.tar.gz by run_partial_tests.sh.
+# Only the tarballs under fixtures/ are committed (matches the e2e_tests_* convention).
+/semantic_purity_deepseek_r1/


Results of tests should be in a separate directory. Check other evals directory for structure

This should go a level above in the .gitignore under evals directory.

tsrikris · 2026-09-03T18:12:23Z

+<!--
+Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.
+
+See LICENSE for license information.


Recommend merging this with the README in evals. Two levels of README seems messy. Can come in a follow up PR.

tsrikris · 2026-09-03T18:12:58Z

+# below the worst healthy run and ~10x above collapse, so ordinary bucketing
+# variance never false-alarms while any real collapse fails hard.
+MIN_SHARED_BUCKETS = {
+    "semantic_purity_deepseek_r1": 10,


Is this needed? Adding a new trace would require updating this then. Would suggest just a single value

tsrikris · 2026-09-03T18:13:44Z

+CANDIDATE_RELPATH = os.path.join("tracediff_output", "diff_stats.csv")
+
+
+def _write(results_path: str, rows: list[dict]) -> None:


Confirming there is no such function inside the eval_utils?

tsrikris · 2026-09-03T18:14:09Z

+semantic_partition_scripted_evals.py records per-run LCA-partition purity
+metrics but does not gate on them (see that module's docstring) -- the
+semantic-bucketing method has real run-to-run variance (observed on
+Qwen3-30B-A3B), so a single run is not a reliable regression signal.


Specific comment needs to be removed

tsrikris · 2026-09-03T18:15:55Z

Receommend renaming all these as test_analysis_agent_semantic)...

tsrikris

Additional comments/clarifications added.

@amoskvic could you please take a look?

Addressing comments on code
Synching changes to staging_agent's latest PR
PR also requires updating the main TraceLens Agent README and the docs

@Ahmedhasssan-aig could you review the evals directory?
@kyle-hoffmeyer @ajassani on the Reporting and split_trace_annotation and a general review too

kyle-hoffmeyer · 2026-09-04T22:53:51Z

+            "--comparison_json_path; unlike --comparison_json_path it may be "
+            "combined with --capture_folder."


Docstring is false. --comparison_json_path can be used with --capture_folder

kyle-hoffmeyer · 2026-09-04T23:00:56Z

+        type=str,
+        default=None,
+        help=(
+            "Path to a precomputed TraceDiff-schema diff_stats.csv (e.g. from "


If the schema of the CSV output of semantic analysis is the same as TraceDiff, then maybe we generalize this to just say "Path to precomputed TraceDiff output". This would be useful in general for developers who have TraceDiff output and want to generate a comparative report without running through TraceDiff again.
I'd also change flag name to be more interpretable. Maybe "precomputed_tracediff_csv"?

kyle-hoffmeyer and others added 30 commits June 3, 2026 14:47

add graph capture + tracediff extension support

fea42ad

staging: Initial Commit

20a28b2

refactor: removed gap report generation from semantic agent

d147736

fix: added compatibility updated fields to tracediff output for mapping

3518c70

feat: appended kernel list with communication kernels

694e0bb

feat: initial changes to the analysis orchestrator to integrate seman…

88ab6bd

…tic agent; will require iteration

prevent caches from overlapping

356d9fe

dont treat hipgraphlaunch as cuda_runtime

64e6b0a

keep traversing when one node has only one child

7bfd0ba

greedy matching

d3501ed

Bug fix with empty rows in sem agent comparative final output

bc937e6

made regex more robust

c3c9556

collapse single node children during WF matching

5d927d2

stop collapsing when kernel launcher is encountered

a2559af

Merge branch 'upstream-main' into feat/comparative_graph_capture_support

919fb32

remove timestamp shifting logic

8d7dd69

formatting

565642f

moving things around

b025e56

formatting

f116da2

Merge branch 'main' into feat/comparative_graph_capture_support

70a9aa8

print warning if graph root has no kernels

fb57556

positional + name matching iif name matching alone fails

b29c6fd

remove gate

35e9b5a

support graph capture comparative

f5cd48d

disambiguate WF candidates with same name. consider cuda_driver as ke…

a760dcf

…rnel dispatcher

merge main

000157b

Merge remote-tracking branch 'upstream/main' into feat/comparative_gr…

db32265

…aph_capture_changes

reorder graph events to match capture event ordering

39a23a0

Initial basic eval scripts

aeea45e

Arseny added 3 commits September 3, 2026 00:31

Further refinements, mostly stylistic/minor

d8a3780

Further pruning

2af6074

Further stylistic improvements

6e2a83d

amoskvic requested review from gabeweisz and tsrikris September 3, 2026 11:15

tsrikris reviewed Sep 3, 2026

View reviewed changes

tsrikris requested changes Sep 3, 2026

View reviewed changes

kyle-hoffmeyer reviewed Sep 4, 2026

View reviewed changes


		## Step 0.5: Comparison Method (comparative only)

		For `standalone`, skip this step.


		## Step 1.5: Semantic Comparative Ordering (`<comparison_method>` = `semantic` only)

		When `<comparison_scope>` = `comparative` and `<comparison_method>` = `semantic`, run Step 1 in this order:

		@@ -0,0 +1,137 @@
		<!--
		Copyright (c) 2024 - 2025 Advanced Micro Devices, Inc. All rights reserved.

		@@ -0,0 +1,75 @@
		###############################################################################
		# Copyright (c) 2024 - 2025 Advanced Micro Devices, Inc. All rights reserved.

		return result, kernels


		def _write_split_regions(split_result, trace_path, output_dir): # pragma: no cover

		CANDIDATE_RELPATH = os.path.join("tracediff_output", "diff_stats.csv")


		def _write(results_path: str, rows: list[dict]) -> None:

		"--comparison_json_path; unlike --comparison_json_path it may be "
		"combined with --capture_folder."

Uh oh!

Conversation

amoskvic commented Aug 18, 2026

Overview

General approach

Structure

Validation

Extra notes:

Uh oh!

amoskvic commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update

Uh oh!

tsrikris Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tsrikris Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tsrikris left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kyle-hoffmeyer Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

amoskvic commented Sep 3, 2026 •

edited

Loading

tsrikris Sep 3, 2026 •

edited

Loading

tsrikris Sep 3, 2026 •

edited

Loading

tsrikris left a comment •

edited

Loading

kyle-hoffmeyer Sep 4, 2026 •

edited

Loading