fix: re-feed persistent prefill inputs on cached-graph recompute in QwenCausalDecodeRuntime - #331
Open
drzsdrtfg wants to merge 1 commit into
Open
Conversation
The prefill graphs of QwenCausalDecodeRuntime upload positions and the attention mask once at graph build time, but the graph allocator only exempts OUTPUT-flagged tensors from lifetime-based memory reuse. After the last read of the mask (the final layer), its memory can be handed to later intermediates, so recomputing a cached prefill graph reads a clobbered mask and produces NaN from the first attention layer onward. Keep host-side copies of both inputs and re-upload them before every run_prefill()/run_batched_prefill() compute, mirroring what the decode step already does for its inputs. First-run outputs are unchanged; recomputes (warmup, repeated same-length prompts) now match them.
drzsdrtfg
added a commit
to drzsdrtfg/audio.cpp
that referenced
this pull request
Aug 29, 2026
- converter emits spec-matched combined.safetensors (backbone + folded decoder incl. ISTFT window); loader requires the window strictly - register soprano_warm_bench target; document ENGINE_BUILD_WARMBENCH - parse eos_bias from request options - streaming follows NeuTTS pattern: run_mode reports task mode, options parsed once in start_stream and reused by next_stream_event - accept bare spec-declared session/load option names (text_chunk_size, backbone_weight_type, decoder_weight_type) - add soprano warm bench and document measured CPU/Vulkan performance including F16/Q8_0 backbone storage types Note: the soprano warm bench --warmup/--iterations paths depend on the QwenCausalDecodeRuntime prefill re-feed fix (0xShug0#331).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Validation (native CPU + Vulkan; no NVIDIA GPU available)
NeuTTS (
prefill_tokens, token-prompt, persistent runtime - recompute reachable)Two identical utterances in one session (batch-text-file x2):
NeuTTS applies an intentional per-chunk seed offset for utterance 2, so the two
lines legitimately differ from each other; the point is the pre-fix second line
is 12.2s of garbage while post-fix it is healthy and reproducible.
DotTTS (
prefill_embeddings, persistent runtime)Two identical lines,
template_name=tts, fixed seed. Pre-fix and post-fix arebyte-identical and healthy in both lines (278A7D17...): the DotTTS path does
not trigger the gallocr reuse, confirming the fix is a behavioral no-op where
the corruption cannot occur.
Soprano (surfaced the bug)
PR #323 branch (which carried this fix until the split):
--warmup 1 --iterations 3passes on CPU and Vulkan; three identical seeded generationsare bit-identical (previously fatal
no finite logits). Soprano is not inupstream yet, so it was validated on the PR branch carrying the same fix code.
Compile matrix
Custom build including all runtime consumers (neutts, dots_tts, fireredtts3,
midashenglm_gen, minimax_music3, qwen3_asr) links cleanly - the header is
unchanged, so no API breakage.
Consumers and why the rest are low-risk
mentions QwenCausalDecodeRuntime in a comment. Removed from the affected list.
creates a fresh runtime (graph built once, not recomputed) - the re-feed is a
no-op for them. miniMax-H3 needs 15GB+ for a runtime test (skipped).
decode step already does every token; it is backend-agnostic.
Caveats
loop pre-fix (graph branch reused); NeuTTS two-line and Soprano
warmup/iterations are the discriminating repeat proofs.
ggml_backend_tensor_setbefore compute, so no backend-specific surprisesare expected, but CUDA CI coverage would be valuable.