Skip to content

Add persistent benchmark server reuse - #93

Merged
haofrank merged 3 commits into
mainfrom
docker-server-reuse
Sep 10, 2026
Merged

Add persistent benchmark server reuse#93
haofrank merged 3 commits into
mainfrom
docker-server-reuse

Conversation

@haofrank

@haofrank haofrank commented Sep 9, 2026

Copy link
Copy Markdown
Member

Summary

  • add persistent benchmark server lifecycle support for both local and Docker run modes
  • split the built-in SGLang, vLLM, and ATOM launch scripts into reusable server and client phases
  • reuse only Magpie-owned servers whose image and server configuration match the current benchmark
  • preserve Docker GPU mappings across client runs and provide explicit final cleanup
  • document the workflow and add a Docker SGLang example configuration

Motivation

Model sweeps often run several client configurations against the same inference server configuration. Restarting and reloading the model for every concurrency point adds substantial overhead and may introduce differences in server initialization. This change allows compatible benchmark invocations to share one warmed server while keeping lifecycle ownership and cleanup inside Magpie.

Validation

  • 156 passed across the benchmark lifecycle, benchmark support, and launcher compatibility tests
  • validated the Docker lifecycle end to end with a multi-point SGLang benchmark on MI355X

Usage

Set benchmark.server_lifecycle.enabled: true, keep cleanup: false for intermediate client runs, and set cleanup: true for the final run. Client-only fields such as concurrency and sequence lengths may change; incompatible server-side configuration is rejected unless reuse is explicitly forced.

@haofrank
haofrank requested review from a team and sinarafati-amd as code owners September 9, 2026 21:06
@haofrank
haofrank merged commit fee7b9e into main Sep 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants