Summary
TreePerfAnalyzer.from_file(..., capture_trace_filepath=...) — the documented SDK entry point for graph-replay + graph-capture inference traces — raises KeyError: t_end unconditionally. There is no value of add_python_func that makes it work.
Repro
Repo at 210fb5c8, using bundled traces:
from TraceLens.TreePerf import TreePerfAnalyzer
TreePerfAnalyzer.from_file(
"tests/traces/inference/xdit_flux.1/profile_trace_rank_0.json.gz",
capture_trace_filepath="tests/traces/inference/xdit_flux.1/capture_traces",
)
Traceback (most recent call last):
File "TraceLens/TreePerf/tree_perf.py", line 240, in from_file
return TreePerfAnalyzer(
File "TraceLens/TreePerf/tree_perf.py", line 288, in __init__
self.tree.build_tree(add_python_func=add_python_func)
File "TraceLens/Trace2Tree/trace_to_tree.py", line 149, in build_host_call_stack_tree
>= stack[-1][TraceEventUtils.TraceKeys.TimeEnd]
KeyError: <TraceKeys.TimeEnd: 't_end'>
The capture merge itself succeeds — alignment reports matched 993 capture events to 993 graph kernels before the crash. Reproduces identically with add_python_func=True and add_python_func=False, and on both xdit_flux.1 and xdit_flux.1_aiter_sage.
Cause
from_file (tree_perf.py:230-247) replaces the tree with the output of merge_capture_trace_into_graph, which returns an already-built tree. It then passes that tree to TreePerfAnalyzer.__init__, which re-runs build_tree because rebuild_tree defaults to True (tree_perf.py:288).
The merged tree contains events with no t_end, which build_host_call_stack_tree dereferences unconditionally:
events missing t_end: 17133
ac2g 17063
None 62
cuda_runtime 8
Workaround
TreePerfAnalyzer.from_file(..., capture_trace_filepath=..., rebuild_tree=False)
Verified: yields a usable tree with 11,076 kernel events and correct device totals.
Suggested fix
Have from_file force rebuild_tree=False on the capture-merge path — rebuilding an already-merged tree is wrong independently of the crash.
Why CI misses this
tests/test_inference_perf_report.py calls merge_capture_trace_into_graph directly and passes the resulting tree onward, so the from_file convenience path has no coverage. Worth adding a smoke test for it.
Summary
TreePerfAnalyzer.from_file(..., capture_trace_filepath=...)— the documented SDK entry point for graph-replay + graph-capture inference traces — raisesKeyError: t_endunconditionally. There is no value ofadd_python_functhat makes it work.Repro
Repo at
210fb5c8, using bundled traces:The capture merge itself succeeds — alignment reports
matched 993 capture events to 993 graph kernelsbefore the crash. Reproduces identically withadd_python_func=Trueandadd_python_func=False, and on bothxdit_flux.1andxdit_flux.1_aiter_sage.Cause
from_file(tree_perf.py:230-247) replaces the tree with the output ofmerge_capture_trace_into_graph, which returns an already-built tree. It then passes that tree toTreePerfAnalyzer.__init__, which re-runsbuild_treebecauserebuild_treedefaults toTrue(tree_perf.py:288).The merged tree contains events with no
t_end, whichbuild_host_call_stack_treedereferences unconditionally:Workaround
Verified: yields a usable tree with 11,076 kernel events and correct device totals.
Suggested fix
Have
from_fileforcerebuild_tree=Falseon the capture-merge path — rebuilding an already-merged tree is wrong independently of the crash.Why CI misses this
tests/test_inference_perf_report.pycallsmerge_capture_trace_into_graphdirectly and passes the resulting tree onward, so thefrom_fileconvenience path has no coverage. Worth adding a smoke test for it.