Skip to content

marker_eval_1 reports FAIL for a top_ops block that is correctly empty #1002

Description

@meinali-566

Description

_check_marker_top_ops in agent_evals/Analysis/eval_utils/workflow_scripted_evals.py counts <!-- top-ops-row ... --> markers but never parses the table those markers annotate. It requires at least one marker to exist, rather than one marker per row.

EVAL_RUBRICS.md states the requirement per row — "each table row within the wrapper has a marker" — which is vacuously satisfied when the table has no rows. So a top_ops block that is correctly empty, because no finding is quantifiable, is reported as a template defect.

Evidence

Standalone harness run, one test case, NUM_REPEATS=1. Trace: vLLM decode steady state, BF16 decoder-only LLM, MI355X, bs64/conc64, rank 0, captured under GPU graph replay.

Result: 66 checks, 64 PASS, 2 FAIL. One of the two failures is this check.

From eval_summary.csv:

index           : marker_eval_1
category        : Marker Identification
issue_summary   : Top Operations markers (kind=top_ops)
result          : FAIL
details         : No <!-- top-ops-row ... --> markers found inside top_ops block
root_cause      : template
recommended_fix : Add or fix top_ops markers in analysis.md

The section it refers to, from the generated analysis.md:

### Top Operations
No quantified priority categories: every compute finding in this trace is a
heuristic (unmodeled) estimate, so the deterministic rollup of quantified
savings is empty.
<!-- impact-begin kind=top_ops -->
| Rank | Category | Time (ms) | % of Compute Time | Ops |
|------|----------|-----------|-------------------|-----|
<!-- impact-end -->

The wrapper is present and correctly paired. The table has a header row, a separator row, and no data rows. The report states the reason directly above the table, and the executive summary of the same report agrees:

All 12 compute findings are heuristic — the graph-replayed kernels have no performance models, so impact estimates are derived from E2E GPU-time share rather than roofline gap projections.

Not a flake. Stage 1 (golden reference) and Stage 2 (candidate) run the analysis independently. Both produced the same empty block:

candidate    data rows = 0    top-ops-row markers = 0
golden ref   data rows = 0    top-ops-row markers = 0

The agent behaved identically both times. Any trace where no compute finding is quantifiable fails this check deterministically; traces captured under GPU graph replay fall into that class, since the replayed kernels have no performance models.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions