RFC / Not for merge: Experimental fixes for agent workloads (+10% to dflash2 agent)

[pkochubey]

Sorry for my sheet in PR and all this mess, I check Issues in project and fixed some issues, what related for me nvfp4+dflash2+agent coding long context+hermes.

Bellow AI bullshit-summary by PR:

Problem and scope

Related Issues:

• Private continuation cache can become permanently saturated across independent conversations #177 — private continuation cache saturation across independent conversations
• Context-cache planner charges private_transition_loss for checkpoints already proven unreachable by a key mismatch #178 — unreachable checkpoints contributing recovery cost
• Physically infeasible shared captures cannot reclaim victims without pressure_evidence #179 — shared-prefix capture unable to reclaim victims without pressure evidence
• Intermittent cudaErrorIllegalAddress with Qwen3.8-27B NVFP4 + MTP on RTX 5090 #208 — long-context split-KV attention page/split safety
• Identical duplicate parameters: the name-only check rejects a call with no value ambiguity #244 — duplicate identical tool-call parameters rejected
• Prefix reuse stops once the checkpoint budget is exercised, and only an engine restart restores it (no LRU eviction observed) #251 — prefix reuse stopping after repeated context-cache turnover
• NVFP4 fused SwiGLU: its TMA route still needs whole M tiles, so ragged widths take the composition #263 — ragged NVFP4 fused SwiGLU execution
• Default shared-prefix catalog (max_shared_prefixes) is smaller than one request's own candidate ceiling, silently killing prefix-cache hits at low concurrency #270 — shared-prefix catalog capacity too small for one request's candidate set
• Task list tool consistently fails after Claude Code Plan creation #276 — Claude Code tool-call XML variants not being recognized

This PR fixes several context-cache, long-context CUDA, NVFP4 and tool-calling edge cases that showed up in long-running agent workloads with Qwen models.

The main user-visible problems were:

• private continuation slots could saturate without selecting an old continuation for replacement;
• shared-prefix reuse could degrade after repeated catalog turnover;
• planner accounting could penalize checkpoints that were no longer reachable;
• physically infeasible shared-prefix captures could fail to explore a reclaim path;
• the default shared-prefix catalog could be smaller than the maximum number of candidates produced by one request;
• quantized small-T attention could select a split geometry that exceeded the page staging capacity at long context;
• fused NVFP4 SwiGLU did not handle arbitrary ragged token counts through the TMA path;
• the Qwen tool-call parser accepted only the original narrow XML representation and rejected XML forms emitted by Claude Code / agent harnesses;
• identical duplicate tool parameters were treated as an error;
• OpenAI Chat Completions and Anthropic Messages could not use a final assistant message as an assistant-prefill continuation.

The PR does not implement the suspended-generation capability proposed in #172. Assistant continuation remains a new request that can reuse the existing context-cache/private-continuation state rather than an opaque resumable generation handle.

Implementation

Context-cache planning and retention

Private continuation entries now carry their publication_order through the materialization policy and planning cost.

When the private continuation catalog is full and an eviction is required, the planner can select the oldest eligible publication instead of failing to make progress.

publication_order is also included in pressure-target ordering so replacement decisions remain deterministic when higher-priority retention metrics are equal.

Unreachable checkpoints are explicitly represented in the checkpoint policy and excluded from portfolio recovery-loss accounting.

Shared-capture planning can explore replacement targets when the requested materialization is physically infeasible, even when the previous pressure-evidence condition is absent.

The default shared-prefix capacity is increased to cover the maximum prepared-prompt candidate count produced by one request, preventing candidate discovery from exhausting the catalog immediately.

Long-context small-T attention

The special quantized attention path that forced the maximum split count for tokens == 1 at large windows was removed.

Quantized attention now uses the normal split calculation, including the page-count-derived lower bound.

Page staging computes the actual required page count and asserts that it fits the compile-time PageIds capacity instead of silently truncating the page range.

This keeps split geometry and paged-KV staging consistent for long-context decode.

Ragged NVFP4 fused SwiGLU

The NVFP4 A4 fused SwiGLU TMA path now supports ragged token counts rather than requiring exact tile multiples.

Workspace/TMA scheduling uses the padded execution extent while output stores remain guarded by the real token count.

This allows the fused path to be used for arbitrary M values without writing the padded tail.

Tool-call parsing

The Qwen tool-call parser is extended to recognize the XML forms observed from Claude Code and other agent harnesses, including:

<tool_call>
  <function name="...">
    <parameter name="...">...</parameter>
  </function>
</tool_call>

<function_calls>
  <invoke name="...">
    <parameter name="...">...</parameter>
  </invoke>
</function_calls>

<invoke name="...">
  <param name="...">...</param>
</invoke>

Tool and parameter names can be supplied either by the original =<name> representation or through a name= attribute.

Attribute parsing uses token boundaries so unrelated attributes such as filename= do not satisfy the name lookup.

Opening and closing XML forms must match (function/function, invoke/invoke, parameter/parameter, param/param).

Tool output may contain normal assistant text before the structured call; that prefix is preserved as assistant content.

Repeated parameters with the same name and the same value are coalesced. Repeated parameters with conflicting values still fail with DuplicateParameter.

Declared tool names and parameter schemas continue to be enforced after the XML representation has been normalized.

Assistant continuation / prefill

OpenAI Chat Completions and Anthropic Messages now treat a request whose final message is an assistant message as:

PromptContinuationMode::ContinueFinalAssistant

The chat template sets continue_final_message and does not add a new assistant generation prompt.

This enables agent clients to send an output-limited partial assistant response back to NInfer and continue the same assistant turn.

The request still goes through normal admission/planning. Existing private continuation/context-cache state can be reused when the token prefix matches; this is not the suspended-generation/resume-capability design proposed in #172.

Verification

Regression coverage was added or expanded for the affected components.

Context cache

Added coverage for private continuation saturation to verify that, with a full private catalog, the older publication_order is selected for eviction.

Added a continuous shared-catalog turnover regression for #251 to exercise repeated capture/replacement cycles instead of testing only a single planning operation.

The planner tests also cover the updated unreachable-checkpoint and replacement behavior.

NVFP4 SwiGLU

The NVFP4 A4 correctness matrix now includes ragged and tile-boundary values including:

2, 4, 5, 16, 56, 64, 65, 96, 97, 112,
128, 129,
255, 256, 257,
300,
511, 512, 513,
1023, 1024, 1025,
1408

These cases exercise both sides of the relevant 128/256/512/1024 tile boundaries and compare the fused implementation with the existing mathematical oracle.

Tool-call parser

Regression tests cover:

• Claude Code function name="..." syntax;
• invoke syntax;
• function_calls containers;
• standalone tool calls following ordinary assistant/plan text;
• <param> and <parameter> forms;
• attribute-name boundary handling;
• rejection of mismatched closing tags;
• coalescing identical duplicate parameters;
• rejection of conflicting duplicate parameters;
• preservation of embedded parameter-like markup in string values;
• a Plan + TaskCreate agent-response reproduction.

Assistant continuation

Schema tests verify that:

[system, user, assistant, user]

remains a normal NewAssistantTurn, while:

[system, user, assistant]

selects ContinueFinalAssistant, and that this mode reaches the final PromptInput.

Commands

A full verification run should include the normal project test suite:

cmake --build build -j
ctest --test-dir build --output-on-failure

For the CUDA changes, the affected long-context attention and NVFP4 SwiGLU tests should additionally be run under CUDA Compute Sanitizer, in particular memcheck and synccheck.

At the time of this report, the branch HEAD has no published GitHub CI/status checks, so a clean full build, complete registered test run and sanitizer run are not claimed by this PR description unless attached separately.

The following remain outside the scope of this change:

• opaque/stateful resumable generation as described in Support resumable output-limit generations without re-decoding accepted tokens #172;
• preserving a suspended frontend/parser/sampler session across OutputLimit;
• an end-to-end real-model stress test proving cache reuse over prolonged GPU/host cache turnover.