Sibling fanout forks alternate anchor/root below ~64K tokens: default private catalog and state slots too small

[alphastorm]

Problem

On the shipped RTX 5090 profile (v0.4.6 container, qwen38-5090-v0.4.6, binary afb691db…), sibling forks of one stored template alternate between anchor reuse and a full re-prefill when the template is below roughly 64K tokens. Measured with omp-ninfer/scripts/fleet_probe.py (stored base, warm edit, explicit checkpoint, four stored forks from the base id):

template	forks (wall s)	server reuse path
57,853 tokens	1.38 / 22.50 / 1.26 / 22.71	anchor / root / anchor / root
67,681 tokens	1.39 / 1.25 / 1.30 / 1.32	anchor ×4

The retained v0.4.4 container (546bb6a8…) reproduces the alternation on the 57.9K template, so this is not a regression of the v0.4.6 image; the v0.4.3/v0.4.4 fanout receipts happened to use a 67.7K template.

Mechanism (measured)

• /v1/ninfer/status during the sequence: private_catalog capacity 2, occupied 2, device_state capacity 2, occupied 2, host_state capacity 8, host_kv occupied 0, and private_evictions increments by exactly one per completed stored fork.
• The cold forks' materialization diagnostics: predicted_now_ns 28.1 s (root), predicted_total_ns 477 s, model_optimal false; the hot forks select the anchor plan at predicted_now_ns 0.14 s with predicted_total_ns 337 s. With the private catalog at its default (2 = 2 × concurrency), reusing the anchor requires evicting the owner that holds it, and the future-loss accounting makes root cheaper on paper.

What did not fix it

• Raising the planner's search cap from 5 ms to 50 ms: the cold forks then stop on value_of_next_expansion after 354 targets with the same 477 s incumbent (search budget is not the limiter).
• Not marking a stored continuation's own pre-generation frontier (only the inherited one): the alternation keeps period 2 and only shifts phase (cold / hot / cold / hot).
• --max-private-continuations 8 alone: 57.9K goes 4/4, but in the same container a following 67.7K template goes 0/4 with host_state 8/8 saturated and last_materialization_predicted_nanoseconds ≈ 600 s.

What did fix it (configuration only, unchanged binary)

--max-private-continuations 8 --device-state-slots 4 --host-state-slots 24 on the shipped v0.4.6 image, one container, three templates in sequence: 57.9K → 4/4 (1.32 / 1.22 / 1.24 / 1.20 s), 67.7K → 4/4 (5.29 / 1.27 / 1.34 / 1.33 s), 57.9K again against the loaded catalog → 4/4 (1.39 / 1.26 / 1.46 / 1.58 s); last_materialization_predicted_nanoseconds 0.15 s, private_catalog 5/8, host_state 6/24, device_state 5/5. Cost: startup slack 3.56 → 3.13 GiB (headroom 1.00 GiB kept), device memory after twelve forks 28,632 MiB vs 28,125–28,195 MiB on the shipped profile; automatic KV capacity unchanged at 131,072.

Requested outcome

Either make the context-cache defaults scale for same-lane fanout (a private catalog of 2 × concurrency with 2 device-state slots cannot hold a base anchor plus one stored sibling without evicting the anchor), or document the sizing rule so a fanout profile can be derived: private continuations ≥ expected sibling forks + 2, with device-state and host-state slots scaled with it, gated at both a sub-64K and a ≥64K template in one process. The omp-ninfer product side will carry the larger catalog as a v0.4.8 RTX 5090 candidate profile with the probe above as its acceptance receipt.

References

• omp-ninfer docs/measurements/2026-09-04-template-fork-rtx5090*.json and the EXP-012 ledger entry (docs/PERFORMANCE.md).
• Related: Checkpoint export blocks the engine: stage under lock, write outside it (fanout branches pay 14s queue_wait) #34 (checkpoint export blocking fanout branches).

[alphastorm]

Clarification on the 67.7K line in the sweep table above: the shipped default (private catalog 2, device-state slots 2) already serves four hot forks at 67,681 tokens — 1.39 / 1.25 / 1.30 / 1.32 s — so the 5.29 s first fork under --max-private-continuations 8 --device-state-slots 4 --host-state-slots 24 was an anchor hit paying a 4.27 s state materialization, not a re-prefill, and it is a cost of the larger configuration at that size, not a gain. The trade is what the omp-ninfer ledger records (EXP-013): the configuration removes the two 22 s re-prefills per four forks at 57.9K and costs about 3.9 s once at 67.7K in a loaded catalog.

Requalified 2026-09-05 on the shipped image through the lifecycle tool (configuration 95765a38, receipt in omp-ninfer releases/v0.4.8/qualification/rtx5090.json): 4/4 anchor hits at 57.9K (1.36 / 1.22 / 1.22 / 1.20 s) and 4/4 at 67.7K (1.43 / 1.30 / 1.37 / 1.39 s) in one process. One more data point for the derivation question: after a docker restart, the first sibling fork of a restored template re-prefills once (22.2 s, root) before its siblings run hot — a restored session carries its endpoint but not the base anchor.