Skip to content

Refactor simulator around pluggable serving topology interface #2

Description

@fhalde

The current simulator mixes two concerns:

  • Workload arrivals, simulation lifecycle, and result aggregation
  • The actual GPU serving model: replica queues, prefill, decode batching, KV pressure, preemption, and utilization

This was fine for the initial design, but it makes the simulator harder to extend to new serving topologies such as prefill/decode disaggregation & maybe more which I'm unaware of at the moment.

Motivation

This split would make the simulator easier to reason about and extend. Arrivals should not need to know whether a request is served by one replica, a prefill pool followed by a decode pool, or a multi-stage distributed topology. They should only submit requests into a serving system.

The metrics layer also needs a stable interface so each topology can expose queue depth, memory pressure, utilization, preemptions, drops, and latency contributors in a consistent way.

Proposed Design

Split the simulator into a generic outer loop and pluggable serving topology implementations.

The generic simulation driver should own:

  • SimPy environment setup
  • workload RNG and request generation
  • arrival process
  • simulation duration / drain window
  • collection of standardized metrics
  • construction of SimResult

The serving topology should own:

  • request routing
  • queues
  • prefill/decode execution
  • batching behavior
  • KV accounting
  • preemption / eviction policy
  • per-resource utilization
  • topology-specific sampling

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions