Skip to content

TraceLens analysis produces inconsistent CSV schemas and row counts across identical runs #1007

Description

@meinali-566

Problem

The analysis run completed but its output CSVs deviate from the golden reference: ops_summary.csv is missing the required column 'total_subtree_kernel_time_ms', and ops_summary_by_category.csv / ops_unique_args.csv have substantially different row counts (41 vs 12 and 317 vs 307), causing the quality_eval_1 check to fail.

Reproduction context

  • Model: GLM-5.2-MXFP4
  • Framework: sglang
  • GPU: MI355X
  • Trace SHA256: 1c858d3c2de4a204e482287c5769a873d01a54f62846c6dd0c82ff5d99ef81c0
  • Harness exit: 0

Evidence

  • The run failed at the eval stage with exactly one failing check out of 54: error_class='eval_failed', fail_count=1, pass_count=53, check_count=54 (check/result_summary)
  • The failing check is a data-quality comparison of analysis output against the golden reference: check_index='quality_eval_1', category='Quality', result='FAIL', root_cause='data' (check/failed_checks)
  • Analysis output ops_summary.csv is missing a required column present in the reference: ops_summary.csv: missing required columns: {'total_subtree_kernel_time_ms'} (check/failed_checks details)
  • Analysis output row counts diverge significantly from the golden reference: ops_summary_by_category.csv: row count 41 vs ref 12; ops_unique_args.csv: row count 317 vs ref 307 (check/failed_checks details)
  • The analysis phase itself ran to completion and produced output; the harness exited cleanly: harness_exit_code=0, analysis_produced=true, 'Phase 1 complete.' and 'Phase 2 complete.' logged; Summary: 55/56 PASS (check/log/harness.log)
  • Golden reference was generated successfully in the same run, so the comparison baseline is valid: Golden Reference Generation Complete: Total: 1 | Generated: 1 | Failed: 0 (check/log/harness.log)

Failed checks

  • quality_eval_1 [data]: ops_summary.csv: missing required columns: {'total_subtree_kernel_time_ms'}; ops_summary_by_category.csv: row count 41 vs ref 12; ops_unique_args.csv: row count 317 vs ref 307

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions