Skip to content

TreePerfAnalyzer.from_file() always crashes when capture_trace_filepath is passed #1013

Description

@mgehre-amd

Summary

TreePerfAnalyzer.from_file(..., capture_trace_filepath=...) — the documented SDK entry point for graph-replay + graph-capture inference traces — raises KeyError: t_end unconditionally. There is no value of add_python_func that makes it work.

Repro

Repo at 210fb5c8, using bundled traces:

from TraceLens.TreePerf import TreePerfAnalyzer

TreePerfAnalyzer.from_file(
    "tests/traces/inference/xdit_flux.1/profile_trace_rank_0.json.gz",
    capture_trace_filepath="tests/traces/inference/xdit_flux.1/capture_traces",
)
Traceback (most recent call last):
  File "TraceLens/TreePerf/tree_perf.py", line 240, in from_file
    return TreePerfAnalyzer(
  File "TraceLens/TreePerf/tree_perf.py", line 288, in __init__
    self.tree.build_tree(add_python_func=add_python_func)
  File "TraceLens/Trace2Tree/trace_to_tree.py", line 149, in build_host_call_stack_tree
    >= stack[-1][TraceEventUtils.TraceKeys.TimeEnd]
KeyError: <TraceKeys.TimeEnd: 't_end'>

The capture merge itself succeeds — alignment reports matched 993 capture events to 993 graph kernels before the crash. Reproduces identically with add_python_func=True and add_python_func=False, and on both xdit_flux.1 and xdit_flux.1_aiter_sage.

Cause

from_file (tree_perf.py:230-247) replaces the tree with the output of merge_capture_trace_into_graph, which returns an already-built tree. It then passes that tree to TreePerfAnalyzer.__init__, which re-runs build_tree because rebuild_tree defaults to True (tree_perf.py:288).

The merged tree contains events with no t_end, which build_host_call_stack_tree dereferences unconditionally:

events missing t_end: 17133
  ac2g          17063
  None             62
  cuda_runtime      8

Workaround

TreePerfAnalyzer.from_file(..., capture_trace_filepath=..., rebuild_tree=False)

Verified: yields a usable tree with 11,076 kernel events and correct device totals.

Suggested fix

Have from_file force rebuild_tree=False on the capture-merge path — rebuilding an already-merged tree is wrong independently of the crash.

Why CI misses this

tests/test_inference_perf_report.py calls merge_capture_trace_into_graph directly and passes the resulting tree onward, so the from_file convenience path has no coverage. Worth adding a smoke test for it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions