[codex] tune prefill settings for trace round1

[DngBack]

What changed

• Tuned docker-compose-air_0705-0940.yml for trace-round1 long-context shape.
• Exposed batching/prefill knobs as env overrides:
  • MAX_NUM_BATCHED_TOKENS
  • MAX_NUM_SEQS
  • MAX_CUDAGRAPH_CAPTURE_SIZE
  • MAX_NUM_PARTIAL_PREFILLS
  • MAX_LONG_PARTIAL_PREFILLS
  • LONG_PREFILL_TOKEN_THRESHOLD
• Defaults changed from max_long_partial_prefills=1 / long_prefill_token_threshold=8192 to 2 / 24576.
• Lowered default MAX_NUM_BATCHED_TOKENS from 8192 to 4096 to favor TBT/ITL once TTFT has enough margin.
• Fixed the concurrent prefill patch to account for skipped_waiting requests when calculating per-slot chunk budget.

Trace analysis

config_yml_aira/phase1_info/trace-round1.jsonl contains 120 requests in 6 waves of 20 requests. Requests inside a wave arrive every 25 ms; waves are separated by 4.525 s. Conversation history grows by 2 messages each wave.

Using the cached Qwen tokenizer as an estimate:

• prompt tokens min/p50/p95/max: 12924 / 18724 / 27348 / 27364
• all 120 requests are above the old 8192 long-prefill threshold
• final wave, requests 100-119, is around 27.3k prompt tokens each

With the previous settings, every request became a long prefill but only one long partial prefill could run at a time. Worse, when all-long requests were skipped due to quota, they moved into skipped_waiting; the patch only planned against waiting, so the next step could calculate an oversized one-request chunk and lose concurrency again. The patch now plans against both skipped_waiting and waiting, so all-long tail waves keep the intended two chunk slots.

For TBT/ITL, max_num_batched_tokens=8192 still allowed roughly 4096 prefill tokens per partial slot. That is good for TTFT but can make decode wait behind large prefill kernels. Defaulting to 4096 makes each two-slot prefill step closer to ~2048 tokens per slot, trading some TTFT slack for steadier decode interleaving.

Validation

• python3 -m py_compile config_yml_aira/0705-0940/patch_vllm_concurrent_prefill_v024.py
• docker build --build-arg BASE_IMAGE=vllm/vllm-openai:v0.24.0-ubuntu2404 -f config_yml_aira/0705-0940/Dockerfile.vllm-concurrent-prefill -t dngback/vllm-v0.24.0-ubuntu2404-concurrent-prefill:latest config_yml_aira/0705-0940
• docker run --rm --entrypoint python3 dngback/vllm-v0.24.0-ubuntu2404-concurrent-prefill:latest /opt/dng-vllm-patches/patch_vllm_concurrent_prefill_v024.py --check
• inspected patched scheduler in image for waiting_queues = (self.skipped_waiting, self.waiting)
• restarted compose with rebuilt image and tuned config
• vLLM logs confirmed max_num_batched_tokens=4096, max_num_partial_prefills=2, max_long_partial_prefills=2, long_prefill_token_threshold=24576
• curl http://127.0.0.1:8000/v1/models
• curl http://127.0.0.1:8000/v1/chat/completions returned OK.