add RTX 4090 profile bundle + validation (Llama-3.1-8B, bf16, tp1) - #59
add RTX 4090 profile bundle + validation (Llama-3.1-8B, bf16, tp1)#59Arifuzzamanjoy wants to merge 4 commits into
Conversation
- fix npu_mem.mem_size 128 -> 24 in rtx4090_single_instance.json - revert unrelated uv pip change in scripts/compile.sh - remove redundant commented MODEL line in profile.sh - enrich host_metadata.txt with full profiling environment - accurate commit history (bench/validation not yet included) Refs casys-kaist#58
- real vLLM v0.19.0 serving benchmark: Llama-3.1-8B, TP=1, 300 requests - validation comparison: real vLLM vs LLMServingSim simulation - workload: sharegpt-llama-3.1-8b-300-sps10.jsonl Refs casys-kaist#58
|
Thanks for the bundle and the validation run. The profile data checks out. Replaying your bundle against the per-tick schedule It is tied to #40 instead — the scheduler's KV eviction behaviour under memory To fix it properly we are replacing the radix tree with a port of vLLM's blocked We will report back once the RTX 4090 numbers land in a sensible range, and then |
…ll-ISL gate Naming: --gpu-memory-utilization / gpu_memory_utilization becomes --npu-memory-utilization / npu_mem.mem_util. It was the only surface in the repo that said GPU -- the cluster config says npu_mem 40 times and num_npus 18 -- and the simulator models accelerators that are not GPUs. The flag help names vLLM's --gpu-memory-utilization so the correspondence is not lost. Placement: the per-instance override now lives inside npu_mem as mem_util, beside the mem_size it multiplies and matching its mem_* siblings. Validated at load: a non-numeric value or one outside (0, 1] is refused with a message saying it is a fraction, since mistyping 0.9 as 90 would silently size the cache 100x. Admission gate: --reserve-full-isl (on by default, per-instance reserve_full_isl) admits a request only if its whole sequence fits, not merely its first chunk. Port of vLLM's scheduler_reserve_full_isl, True there too, documented as preventing "over-admission and KV cache thrashing with chunked prefill". allocate_slots' block arithmetic is extracted into _num_blocks_to_allocate so the gate and the allocation cannot drift apart -- two estimates disagreeing was the radix tree's central defect. Startup output: a KV Cache Initialization section between the input config and the run loop, listing each instance's derived capacity: • Instance [0] : 54400 tokens / 3400 blocks (6.64 GiB/rank at util 0.90) The utilization fraction alone does not tell you where memory pressure will land; the token count does. It is per instance and only known once the schedulers exist, so it cannot be a row in the input-config block. Dropped that block's global utilization row, which would have lied as soon as one instance overrode it. New logger helper print_heading() centres a section title without drawing a second rule. "▶ Starting simulation..." moves to just before the loop. It used to print before config_builder, the prefix pools, the schedulers, the controller and the router were built. Docs: AGENTS.md gains the gate and npu_mem.mem_util; the KV-cache and memory page is rewritten around the block pool, the capacity formula and the gate; CONTRIBUTORS.md picks up #51, #53, #56, #57 and credits the two reports that drove this work (#40's scheduler analysis, and #58/#59, whose 24 GB profiling run was the first time the simulator saw real KV pressure); the CHANGELOG says explicitly what replaced the radix tree and that prefix caching's user-visible flags are unchanged. bench/examples/configs/*.json state mem_util: 0.9 explicitly. At 96 GiB those configs have 458k-617k tokens of KV, far past what the workload wants, so the value is inert there and all three examples reproduce their previous clocks exactly.
2 participants
refs #58
adds compute-only kernel latency bundle for RTX 4090 and an end-to-end
vLLM vs simulation comparison, as discussed in the issue thread with @JaehongCS20.
what's included
profiler/perf/RTX4090/meta-llama/Llama-3.1-8B/bf16/tp1/
host details (vast.ai)
validation results
errors are large. TTFT is massively underestimated by the simulator,
TPOT is overestimated. end-to-end latency is closer (~15%) because
the errors partially cancel. this matches what you described in the
issue thread — attention kernels on newer architectures producing
bigger gaps than the earlier RTXPRO6000 experiments.
all profile metadata and logs are retained in the PR. happy to dig
into which kernel category drives the TTFT gap if that's useful.
deferred