Skip to content

Fix eval concurrency argument compatibility - #89

Merged
haofrank merged 7 commits into
mainfrom
fix/eval-concurrency-compat
Sep 2, 2026
Merged

Fix eval concurrency argument compatibility#89
haofrank merged 7 commits into
mainfrom
fix/eval-concurrency-compat

Conversation

@haofrank

@haofrank haofrank commented Sep 1, 2026

Copy link
Copy Markdown
Member

Summary

Fix accuracy evaluation failures caused by passing the unsupported
--concurrent-requests argument to InferenceX run_eval, while ensuring
accuracy artifacts are reliably persisted for both local and remote evaluation.

Changes

  • Configure lm-eval concurrency through EVAL_CONCURRENT_REQUESTS.
  • Add MAGPIE_EVAL_CONCURRENCY to control accuracy concurrency independently
    from performance benchmark concurrency.
  • Apply the concurrency fix consistently to SGLang, vLLM, and ATOM benchmark
    scripts on MI300X and MI355X.
  • Use the following concurrency precedence:
    1. MAGPIE_EVAL_CONCURRENCY
    2. EVAL_CONCURRENT_REQUESTS
    3. benchmark CONC
  • Preserve raw lm-eval JSON/JSONL artifacts under lm_eval/.
  • Generate a stable accuracy_report.json beside benchmark_report.json.
  • Support batched concurrency values such as
    EVAL_CONCURRENT_REQUESTS="2 4" without losing staged results.
  • Record the actual accuracy concurrency in meta_env.json, rather than the
    performance benchmark concurrency.
  • Propagate accuracy-report persistence failures for remote evaluations instead
    of reporting a false success.
  • Leave source lm-eval artifacts untouched when copying them into the benchmark
    workspace.
  • Update benchmarking documentation and the Magpie skill reference.

Output

When accuracy evaluation is enabled, the benchmark workspace now includes:

results/benchmark_<framework>_<timestamp>/
├── benchmark_report.json
├── accuracy_report.json
└── lm_eval/
    ├── results_*.json
    ├── samples_*.jsonl
    └── meta_env.json

@haofrank
haofrank requested a review from weilei0120 September 1, 2026 23:36
@haofrank
haofrank requested review from a team and sinarafati-amd as code owners September 1, 2026 23:36
sinarafati-amd
sinarafati-amd previously approved these changes Sep 1, 2026

@sinarafati-amd sinarafati-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@sinarafati-amd sinarafati-amd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@haofrank
haofrank merged commit 8a7529c into main Sep 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants