qwen35: sampled-verify — speculative decoding with an active sampler

[Rhonstin]

What

Adds a sampled-verify mode to the qwen35 chain spec-decode path, enabled with DFLASH_SAMPLED_VERIFY=1. It implements the "active sampler during verification" item from the README's future additions: speculative decoding that works at any temperature while preserving the exact target sampling distribution.

Today the chain verify commits the target's argmax at each position, so spec decode is only distribution-correct at temperature 0 — sampling requests fall back to AR decode. On a 27B Q4_K_M / RTX 3090 that means 16 tok/s for any agent traffic that samples.

How it works

Sample-and-match: at each chain position the verifier draws a token from the target's full sampler chain (temperature / top-k / top-p / penalties) over that position's verify logits, and accepts the draft token iff the sample matches it. On mismatch, the sampled token itself becomes the bonus token. Every committed token is therefore an exact sample from the target distribution — output distribution identical to AR sampling, no acceptance-rate correction terms needed.

Three spots must sample rather than take argmax — each one, when wrong, corrupts generation in a distinctive way that we hit and fixed during validation:

1. First token after prefill — was the prefill argmax; injects one greedy token per request (at temp 1, 150/150 generations opened with the same word). Now sampled from the prefill logits.
2. Per-step seed — the next draft step is seeded with the replay's last token, which the next verify commits as-is. Argmax here injects one greedy token per step (~1/16 of output) and locks long generations into repetition loops. Now sampled from the replay's last verify logits.
3. The acceptance walk — over per-position verify logits, exposed via a new DFlashTarget::read_verify_logits() hook (virtual, default false; the qwen35 target reads the step graph's logits tensor, so targets that don't implement it are unaffected).

fa-window interaction

Sampled-verify is gated to fa_window == 0. With a finite --fa-window, the windowed verify pass starves the logit tail at long context: argmax survives, but sampling from the poisoned tail degrades quality. Decisive experiment on a 24K-token agent prompt: tool-call success 0/12 with a finite window vs 12/12 with full-attention verify. Prefill and AR decode always run full attention, so they're unaffected — this only constrains the verify pass.

Results (RTX 3090, Qwen3.6-27B Q4_K_M + DFlash draft)

Workload	AR sampling	sampled-verify
short prompts, temp 0.7–1.0	16 tok/s	60–70 tok/s
24K agent prompt w/ tools, temp 1.0	16 tok/s	~31 tok/s
tool-call success (12 runs, temp 1.0)	12/12	12/12

• Greedy path (DFLASH_SAMPLED_VERIFY unset) is bit-exact unchanged.
• Token histograms over 150 sampled generations match AR within noise.
• No degeneration over 1200-token continuous generations.
• DFLASH_SV_DEBUG=1 traces the acceptance walk per position.

Default is off, so existing deployments see zero behavior change.

[cubic-dev-ai]

2 issues found across 4 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Rhonstin]

Amended after self-review: (1) the DFLASH_SAMPLED_VERIFY gate is now correctly opt-in (=1) as documented — the earlier revision had it opt-out; (2) the acceptance walk now seeds its penalty history with the step's seed token (draft_tok[0]), which is committed by the replay but not yet in out_tokens at walk time — without it the sampled distribution drifted from AR by one token whenever repetition penalties are active.