feat(sim): profiled replay

[Mrtroll486]

Summary

Pr 1/3 of issue #1039

This PR adds a runnable, deterministic offline serving simulator to pegainfer-sim and provides the versioned timing-profile and step-worker foundations needed by it.

The offline simulator does not depend on GPUs, model weights, HTTP, or wall-clock time. It replays requests through a single logical event loop and produces identical event ordering and reports for the same scenario, workload, timing profile, and seed.

Background

The existing online SimScheduler is useful for validating the OpenAI/vLLM frontend, HTTP streaming, and metrics. It is not suitable for fleet-level scheduling experiments because its results include HTTP, Tokio, operating-system scheduling, and real-time waiting overhead, and because a large HTTP run is not a simulation of a large serving fleet.

This PR adds an independent logical-clock replay module instead of scaling the online simulator with threads or servers. The online simulator continues to cover protocol and integration behavior; the offline simulator focuses on scheduling, routing, and fleet A/B experiments.

Production Invariant

• The same scenario, workload, timing profile, and random seed produce a byte-identical report.
• Every request reaches exactly one terminal state: Completed or Rejected.
• No step exceeds max_num_seqs or max_batched_tokens.
• No request exceeds max_model_len.
• Increasing the logical worker count does not create one OS thread, Tokio task, or HTTP server per worker.
• The existing online simulator protocol remains compatible when profile mode is not explicitly used.

Main Changes

Offline logical-clock simulator

• Represents logical time as integer microseconds.
• Drives the entire fleet from one BinaryHeap event loop.
• Applies stable same-time event ordering:
  1. Worker step completion;
  2. Request arrival;
  3. Worker scheduling;
  4. Monotonic sequence-number tie-break.
• Implements the Waiting -> Prefill -> Decode -> Completed/Rejected lifecycle.
• Supports max_num_seqs, max_batched_tokens, and max_model_len.
• Implements decode-first continuous batching.
• Supports optional per-request prefill chunking.
• Generates one token per active decode request per step.

Fleet and routing

• Workers contain data state only; they do not own tasks, threads, or servers.
• Supports round-robin routing.
• Supports seeded-random routing using a fixed SplitMix64 implementation.
• Replays do not depend on a third-party RNG implementation or version.

Timing and JSON contracts

Adds separate, strict, versioned JSON contracts for:

• Scenario;
• Workload;
• Timing profile;
• Simulation report.

PR1 provides an extensible tagged timing-model enum with a fixed synthetic model:

• Fixed per-step overhead;
• Per-prefill-token cost;
• Per-decode-token cost.

Reports include:

• SHA-256 digests of the exact input bytes;
• Request placement and terminal outcome;
• Queue time, TTFT, ITL, and E2E latency;
• Token timestamps;
• Request and output-token throughput;
• Worker busy time, utilization, and peak running/waiting counts;
• An optional full simulation trace.

Standalone CLI

Adds pegainfer-sim-replay, which runs without starting the HTTP frontend:

cargo run --release -p pegainfer-sim \
  --bin pegainfer-sim-replay -- \
  --scenario pegainfer-sim/examples/offline/scenario.json \
  --workload pegainfer-sim/examples/offline/workload.json \
  --timing-profile pegainfer-sim/examples/offline/timing-profile.json \
  --output /tmp/pegainfer-sim-report.json

The repository includes runnable scenario, workload, and timing-profile examples.

Online simulator foundations and hardening

The preceding commits in this branch also provide:

• Versioned engine timing profiles with strict validation;
• Reusable step-based worker state;
• Profile-driven online scheduling and timing;
• Server CLI profile loading and strict out-of-domain handling;
• An online profile frontend E2E gate;
• Admission checks before completion allocation, preventing oversized max_tokens requests from causing OOM;
• Correct propagation of fallback-token-id in profile mode;
• Error-returning timing conversion instead of runtime panic;
• A distinct rejection type for whole-prefill step-budget failures.

These facilities provide the foundation for future benchmark-driven calibration, while the PR1 offline timing model remains explicitly synthetic.

Acceptance Evidence

• Hand-calculated workloads produce the expected event timestamps.
• Repeated fixed-seed runs produce byte-identical reports.
• Round-robin and random routing produce different placements and latency results for the same workload.
• Sequence capacity, queueing, context rejection, and one-time terminal-state checks pass.
• Every step stays within the configured sequence and token budgets.
• A u32::MAX output request is rejected before allocation.
• A 1024-worker replay completes in one logical event loop.
• The standalone CLI produces a parseable report.
• Existing online frontend, streaming, and metrics regression tests pass.

Verification

cargo fmt --all -- --check

cargo clippy --release \
  -p pegainfer-sim \
  --all-targets -- -D warnings

NO_PROXY=127.0.0.1,localhost \
no_proxy=127.0.0.1,localhost \
cargo test --release \
  -p pegainfer-sim \
  --lib \
  --test frontend_e2e \
  --test offline_replay

cargo test --release \
  -p pegainfer-frontend \
  --lib

Results:

• pegainfer-sim unit tests: 20/20 passed;
• Online frontend E2E tests: 17/17 passed;
• Offline replay tests: 7/7 passed;
• pegainfer-frontend unit tests: 65/65 passed;
• Formatting, clippy, and diff checks passed.

Non-Goals

This PR does not include:

• Fitting timing profiles from real PegaInfer or vLLM benchmarks;
• Claims about real GPU performance or latency accuracy;
• KV-cache identity, capacity, or eviction;
• KV-aware routing or prefix-cache hit/miss modeling;
• Prefill/decode disaggregation;
• Worker failure, retry, or recovery modeling;
• Heterogeneous fleets;
• Speculative-decoding simulation;
• Automatic search for an optimal routing or scheduling policy.

These capabilities are reserved for follow-up calibration and KV-aware routing work.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist

• My code follows the style guidelines of this project (see docs/conventions/coding-style.md).
• I have performed a self-review of my own code.
• I have formatted my commits according to Commitizen conventions.
• I have run the local test suite and all tests pass (see CLAUDE.md).

[Mrtroll486]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a91522033c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Keep legacy TPOT independent of batch width

When the CLI runs without --profile, build_runtime still attaches this generated profile, so concurrent requests share one decode step. Multiplying the duration by decode_reqs makes each token interval N * tpot_ms for a batch of N requests, whereas the previous legacy scheduler advanced every ready request after the same fixed TPOT. Consequently, existing vllm bench workloads with concurrency greater than one report artificially increasing TPOT and latency; retain the legacy scheduler for this mode or make its decode-step duration batch-invariant.

Useful? React with 👍 / 👎.

[Mrtroll486]

Solved.

[chatgpt-codex-connector]

Install a logger before promising fallback warnings

In the pegainfer-sim executable, neither main.rs nor pegainfer-frontend installs a logger for the log facade, so this warning is discarded. Whenever the default WarnAndFallback policy encounters a shape outside the measured grid, the run silently switches to parametric timing and the benchmark operator cannot tell that results are no longer profile-backed; initialize logging or surface this warning through the executable's configured stderr/reporting path.

Useful? React with 👍 / 👎.

[Mrtroll486]

Solved.

[Mrtroll486]

2 review suggestions provided by codex has been fixed, regression test added.

[xiaguan]

The overall online worker implementation looks reasonable. I ran the existing library/HTTP tests and exercised the unchanged head with vllm bench serve: all 64 requests completed at each client concurrency of 1, 4, and 8, with queueing appearing above the configured worker capacity. This used a synthetic timing profile, so it validates the serving path rather than prediction accuracy.

The main design question is the JSON contract. Please post a proposal in a comment for discussion before implementing further schema/model changes. We should first establish the minimum useful inputs and how we will obtain their values.

Some thoughts for that discussion:

• Start with the prediction interface and calibration data. What workload description goes in, and what duration comes out? A forward-pass duration is a useful target; request queueing can then be accounted for by the simulator. Explain how measurements will be collected and how prediction error will be checked before committing to a dense three-dimensional table.
• Consider simpler fitted functions. AIConfigurator's forward-pass model has a regression path using prefill token count for prefill, and decode request count plus total KV tokens for decode. Its prefill regression can fit a piecewise-linear curve. These are useful references, not a requirement to reproduce AIC or assume its accuracy transfers here. Discuss whether a separate mixed-workload model is needed.
• Ablate the required fields. Keeping schema_version makes sense. For every other field, identify the behavior it controls or the ambiguity it prevents. scheduler.policy currently has only one implementation; profile_id could be represented by the file name. Model/hardware/backend identity should remain associated with the calibration dataset, but we should decide which metadata must be in the runtime JSON. My worker-level check found identical plans, outputs, and estimated durations after changing descriptive metadata, and after zeroing fallback coefficients in strict mode; that does not establish that dataset identity is dispensable.
• Justify independent knobs and failure behavior. Does a separate per-request max_chunk_tokens add useful behavior beyond consuming the remaining batch token budget? Removing it changes scheduling, so this deserves a concrete workload example. Likewise, explain whether out-of-domain prediction needs a fallback model in the first version or should report missing coverage.

A small proposed JSON example, a purpose for each retained field, and a calibration/validation outline would be enough to start the discussion. We can agree on that before expanding the implementation.

Separately, there is one reproduced correctness issue on e8ae2e18: in apply_outcome, an abort observed after complete_step can retire the ledger entry and remove request metadata while leaving a nonfinished request in WorkerState. A later step then panics on the closed ledger entry and interrupts other requests. A public-engine test submitting 128 requests and concurrently cancelling 127 reproduced this on the first round in two runs. Temporarily removing the request from the worker in that cleanup path made 30 rounds pass with the uncancelled request completing; restoring the original code reproduced the failure. Please keep worker/metadata/ledger cancellation consistent and cover the surviving-request behavior.

[Mrtroll486]

The correctness issue mentioned above has been fixed. The proposal follows:

Prediction Contract

The timing predictor should answer one question:

StepShape -> forward-pass duration

The initial shape is:

StepShape {
    decode_reqs,
    sum_decode_ctx_tokens,
    prefill_tokens_in_step,
}

These values come from the simulated worker scheduler after admission and
step planning, immediately before the step executes.

The returned duration represents target-engine execution time for that
scheduled step. Request queueing, admission delay, HTTP/frontend overhead,
and accumulation into TTFT/TPOT/E2E latency remain simulator
responsibilities.

When target-engine step telemetry is available, it can provide the shape and
measured duration directly. Black-box request metrics alone cannot uniquely
recover internal step shapes, so controlled black-box workloads will be used
together with optional engine traces/FPM data. Held-out request-level
behavior remains the final fidelity gate when exact step telemetry is not
available.

Work In This PR vs. PR2

This PR will validate the runtime contract and scheduler behavior:

• ablate runtime/profile fields;
• test whether max_chunk_tokens has independent scheduling value;
• define strict out-of-domain behavior;
• keep timing behind the stable StepShape -> Duration interface;
• retain the current grid as a provisional baseline while allowing simpler
  predictor implementations to be compared.

PR2 will perform real timing calibration:

• collect target-engine serving measurements;
• normalize them into calibration observations;
• fit prefill, decode, and optional mixed-workload models;
• compare those models with the current grid on held-out workloads;
• select the timing representation based on measured error and coverage.

Candidate Timing Models

The initial comparison will include:

Prefill:
    T_prefill = piecewise_linear(prefill_tokens_in_step)

Decode:
    T_decode = f(decode_reqs, sum_decode_ctx_tokens)

Additive:
    T_step = intercept + T_prefill + T_decode

A separate aggregate/mixed correction will only be added if held-out mixed
steps show systematic residual error:

T_step =
    intercept
  + T_prefill
  + T_decode
  + T_mixed(prefill_tokens_in_step,
            decode_reqs,
            sum_decode_ctx_tokens)

T_mixed does not initially need to be a dense three-dimensional table. It
could be a small interaction term, a two-dimensional aggregate regression,
or a sparse set of correction cells. The current 3D grid remains one
candidate, not the assumed final representation.

Illustrative Minimal Profile

The exact predictor payload remains a tagged implementation choice, but the
runtime contract could look like:

{
  "schema_version": 1,
  "calibration": {
    "target_engine": "vllm",
    "engine_version": "<version>",
    "model_id": "<model>",
    "model_revision": "<revision>",
    "model_config_sha256": "<sha256>",
    "gpu": "<gpu>",
    "server_config_sha256": "<sha256>",
    "source_sha256": "<sha256>"
  },
  "scheduler": {
    "policy": "vllm_v1",
    "max_num_seqs": 4,
    "max_num_batched_tokens": 2048,
    "max_model_len": 32768,
    "prefill": {
      "mode": "chunked",
      "max_chunk_tokens": 512
    }
  },
  "predictor": {
    "kind": "regression_v1",
    "out_of_domain": "error",
    "prefill": {
      "knots": [[0, 0], [512, 1800], [2048, 6100]]
    },
    "decode": {
      "intercept_us": 100,
      "request_us": 20,
      "context_token_us": 0.02
    },
    "mixed_correction": null
  }
}

The numeric values above are illustrative only. Full raw benchmark inputs,
server flags, collection commands, and environment details belong in the
calibration bundle manifest. The runtime artifact only needs
compatibility-critical identity and source digests.

Field Purpose and Current-PR Ablations

Field	Purpose or ambiguity prevented	Experiment / proposed treatment
schema_version	Defines parsing and semantics	Keep; unknown versions fail closed
profile_id	Presentation and logging only	Remove as a required field; use file name and source digest
calibration identity	Prevents using measurements from another engine, model, GPU, or config	Mutate each field and verify predictions are unchanged while compatibility/reporting still identifies the mismatch
scheduler.policy	Defines admission and chunking semantics that numeric limits alone do not describe	Keep as scheduler-adapter identity, not a timing input; unsupported values fail closed
max_num_seqs	Limits concurrent running requests	Vary independently and compare admission and waiting sequences
max_num_batched_tokens	Limits total decode plus prefill work in a step	Vary independently and compare generated StepShapes
max_model_len	Defines valid request/context domain	Test boundary acceptance and rejection
prefill mode	Selects whole versus chunked scheduling	Replay the same requests under both modes and compare step plans
max_chunk_tokens	May prevent one prefill from consuming all remaining step budget	Run the concrete scheduling experiment below
predictor payload	Maps StepShape to duration	Keep behind a tagged predictor interface; compare representations in PR2
fallback coefficients	Produce unvalidated values outside measured coverage	Do not require them in the first contract; use explicit missing-coverage errors until a fallback is validated

Descriptive metadata changing neither step plans nor durations is expected.
That does not make calibration identity dispensable: identity protects
artifact selection and reproducibility rather than controlling the predictor
directly.

max_chunk_tokens Experiment

Use:

max_num_seqs = 4
max_num_batched_tokens = 8
prefill mode = chunked

At the start of the step:

A, B: two running decode requests
C: waiting request with a 100-token prompt
D: waiting request with a 1-token prompt

A and B consume two decode tokens, leaving six batch tokens.

Compare:

max_chunk_tokens = 6:
    C consumes all six remaining tokens; D waits.

max_chunk_tokens = 2:
    C consumes two tokens; D can be admitted in the same step.

Record the exact step sequence, admission step, D's queueing/TTFT, C's
completion time, and total throughput. Retain the independent knob only if
this behavior is useful and cannot be expressed by the global batch-token
budget alone.

Out-of-Domain Experiment

Construct a small measured domain and test:

• an in-domain shape;
• one shape outside each individual axis;
• a shape outside multiple axes.

For each case, verify:

• whether prediction returns a duration or a structured missing-coverage
  error;
• whether worker state remains unchanged when prediction fails;
• whether unrelated surviving requests can continue;
• whether the error identifies the shape and supported domain.

The proposed first-version behavior is strict failure for missing coverage.
A fallback should be added only after PR2 validates it on held-out
out-of-domain cells and reports when it was used.

PR2 Calibration and Validation

Calibration and held-out workloads will be separated before fitting. Results
will not be merged across engine version, model revision, GPU, backend, or
server configuration.

For each predictor candidate, report:

• held-out step-duration absolute-error p50/p95/p99;
• held-out relative-error p50/p95/p99;
• in-domain coverage and missing-coverage rate;
• mixed-workload error separately from pure prefill and pure decode;
• model size and number of required calibration points;
• request-level TTFT, TPOT, E2E latency, throughput, and queueing-knee error.

No accuracy threshold will be fixed before the first measurements. The final
choice between a fitted function, a mixed correction, and the 3D grid will be
based on those results.