The current simulator mixes two concerns:
- Workload arrivals, simulation lifecycle, and result aggregation
- The actual GPU serving model: replica queues, prefill, decode batching, KV pressure, preemption, and utilization
This was fine for the initial design, but it makes the simulator harder to extend to new serving topologies such as prefill/decode disaggregation & maybe more which I'm unaware of at the moment.
Motivation
This split would make the simulator easier to reason about and extend. Arrivals should not need to know whether a request is served by one replica, a prefill pool followed by a decode pool, or a multi-stage distributed topology. They should only submit requests into a serving system.
The metrics layer also needs a stable interface so each topology can expose queue depth, memory pressure, utilization, preemptions, drops, and latency contributors in a consistent way.
Proposed Design
Split the simulator into a generic outer loop and pluggable serving topology implementations.
The generic simulation driver should own:
- SimPy environment setup
- workload RNG and request generation
- arrival process
- simulation duration / drain window
- collection of standardized metrics
- construction of
SimResult
The serving topology should own:
- request routing
- queues
- prefill/decode execution
- batching behavior
- KV accounting
- preemption / eviction policy
- per-resource utilization
- topology-specific sampling
The current simulator mixes two concerns:
This was fine for the initial design, but it makes the simulator harder to extend to new serving topologies such as prefill/decode disaggregation & maybe more which I'm unaware of at the moment.
Motivation
This split would make the simulator easier to reason about and extend. Arrivals should not need to know whether a request is served by one replica, a prefill pool followed by a decode pool, or a multi-stage distributed topology. They should only submit requests into a serving system.
The metrics layer also needs a stable interface so each topology can expose queue depth, memory pressure, utilization, preemptions, drops, and latency contributors in a consistent way.
Proposed Design
Split the simulator into a generic outer loop and pluggable serving topology implementations.
The generic simulation driver should own:
SimResultThe serving topology should own: