Description
_check_marker_top_ops in agent_evals/Analysis/eval_utils/workflow_scripted_evals.py counts <!-- top-ops-row ... --> markers but never parses the table those markers annotate. It requires at least one marker to exist, rather than one marker per row.
EVAL_RUBRICS.md states the requirement per row — "each table row within the wrapper has a marker" — which is vacuously satisfied when the table has no rows. So a top_ops block that is correctly empty, because no finding is quantifiable, is reported as a template defect.
Evidence
Standalone harness run, one test case, NUM_REPEATS=1. Trace: vLLM decode steady state, BF16 decoder-only LLM, MI355X, bs64/conc64, rank 0, captured under GPU graph replay.
Result: 66 checks, 64 PASS, 2 FAIL. One of the two failures is this check.
From eval_summary.csv:
index : marker_eval_1
category : Marker Identification
issue_summary : Top Operations markers (kind=top_ops)
result : FAIL
details : No <!-- top-ops-row ... --> markers found inside top_ops block
root_cause : template
recommended_fix : Add or fix top_ops markers in analysis.md
The section it refers to, from the generated analysis.md:
### Top Operations
No quantified priority categories: every compute finding in this trace is a
heuristic (unmodeled) estimate, so the deterministic rollup of quantified
savings is empty.
<!-- impact-begin kind=top_ops -->
| Rank | Category | Time (ms) | % of Compute Time | Ops |
|------|----------|-----------|-------------------|-----|
<!-- impact-end -->
The wrapper is present and correctly paired. The table has a header row, a separator row, and no data rows. The report states the reason directly above the table, and the executive summary of the same report agrees:
All 12 compute findings are heuristic — the graph-replayed kernels have no performance models, so impact estimates are derived from E2E GPU-time share rather than roofline gap projections.
Not a flake. Stage 1 (golden reference) and Stage 2 (candidate) run the analysis independently. Both produced the same empty block:
candidate data rows = 0 top-ops-row markers = 0
golden ref data rows = 0 top-ops-row markers = 0
The agent behaved identically both times. Any trace where no compute finding is quantifiable fails this check deterministically; traces captured under GPU graph replay fall into that class, since the replayed kernels have no performance models.
Description
_check_marker_top_opsin agent_evals/Analysis/eval_utils/workflow_scripted_evals.py counts<!-- top-ops-row ... -->markers but never parses the table those markers annotate. It requires at least one marker to exist, rather than one marker per row.EVAL_RUBRICS.mdstates the requirement per row — "each table row within the wrapper has a marker" — which is vacuously satisfied when the table has no rows. So a top_ops block that is correctly empty, because no finding is quantifiable, is reported as a template defect.Evidence
Standalone harness run, one test case, NUM_REPEATS=1. Trace: vLLM decode steady state, BF16 decoder-only LLM, MI355X, bs64/conc64, rank 0, captured under GPU graph replay.
Result: 66 checks, 64 PASS, 2 FAIL. One of the two failures is this check.
From eval_summary.csv:
The section it refers to, from the generated analysis.md:
The wrapper is present and correctly paired. The table has a header row, a separator row, and no data rows. The report states the reason directly above the table, and the executive summary of the same report agrees:
Not a flake. Stage 1 (golden reference) and Stage 2 (candidate) run the analysis independently. Both produced the same empty block:
The agent behaved identically both times. Any trace where no compute finding is quantifiable fails this check deterministically; traces captured under GPU graph replay fall into that class, since the replayed kernels have no performance models.