Skip to content

feat: release and resume inference runtime - #56

Open
0z5a wants to merge 4 commits into
ViperEkura:mainfrom
0z5a:codex/inference-release-resume
Open

feat: release and resume inference runtime#56
0z5a wants to merge 4 commits into
ViperEkura:mainfrom
0z5a:codex/inference-release-resume

Conversation

@0z5a

@0z5a 0z5a commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a reversible lifecycle for scheduler-owned inference runtime state
  • release KV pages, task caches, executor workspaces, and graph state while retaining model weights and policy version
  • reconstruct the runtime through resume() and restart an async loop only when it was running before release
  • expose the lifecycle through InferenceEngine, RolloutGenerator, and RolloutRunner, including rollout tensor-cache cleanup
  • reject release for caller-owned caches instead of silently discarding external state

NVIDIA L20 benchmark

AstrAI 1B, BF16, batch 4, prompt 128, four greedy decode tokens, five complete release/resume cycles per cell:

Max context Runtime footprint Reclaimed Reclaimed % Release median Resume median Greedy parity
2,048 200.97 MiB 192.85 MiB 95.96% 105.62 ms 2.01 ms 5/5
8,192 777.14 MiB 769.01 MiB 98.95% 90.16 ms 5.09 ms 5/5
32,768 3,081.81 MiB 3,073.68 MiB 99.74% 89.27 ms 5.10 ms 5/5

The benchmark is bound to implementation commit 5900c786322b522162af0bef0674464806f1628a; raw per-cycle JSON and the reproduction tool are checked in. CUDA Graph was disabled to isolate scheduler-owned KV/workspace memory. The source-mounted image did not expose the optional attention, rotary, and GEMV extensions, so it used the established fallbacks; this PR makes a runtime-allocation lifecycle claim, not a kernel-throughput claim.

Validation

  • local: Ruff format/import checks passed; 628 passed, 103 skipped
  • NVIDIA L20 GPU5: focused scheduler/engine/rollout suite 71 passed in 2.27s
  • InfraSWE commit 811bc775ed5b3a6ec853219245f3469f78818020: comparison cell validated and 53 Draft/system-path tests passed
  • visible-evidence diagnostic ProjectFit: 92.29/100
  • BenchmarkTrust: 97.40/100

The InfraSWE score is explicitly diagnostic and unsealed. Official ProjectFit remains unresolved because the Draft seal, system-trace evidence, hidden probes, and verified evidence manifest are absent.

0z5a added 4 commits September 2, 2026 21:42
Add a reversible scheduler lifecycle that drops scheduler-owned KV storage, decode workspace, and CUDA graph state while retaining shared model weights and policy version. Expose the lifecycle through inference and rollout APIs, clear cached rollout tensors, and add single-GPU memory/parity benchmarking.
Capture five release/resume cycles for the AstrAI 1B BF16 preset at 2K, 8K, and 32K context bounds, including raw memory, latency, and greedy-parity data from the implementation commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant