Enhance FireRedTTS3 with streaming, batching, and server integration - #439
Open
duj12 wants to merge 12 commits into
Open
Enhance FireRedTTS3 with streaming, batching, and server integration#439duj12 wants to merge 12 commits into
duj12 wants to merge 12 commits into
Conversation
- app/server/firered_server.json: model entry for FireRedTTS3-Base q8_0 GGUF, task=clon, mode=offline, CUDA device 0, reference_cache_slots=8. Co-Authored-By: Claude <noreply@anthropic.com>
…al mode) - istft_graph: incremental overlap-add iSTFT (append_incremental/finish_incremental) for streaming audio output. - redae_codec_runtime: decode_incremental (chunked decoder Qwen with KV chaining via start_decode_embeddings/decode_embedding) + flush_incremental. - fireredtts3 redae wrapper: expose decode_reset/decode_incremental/flush_incremental. - fireredtts3 pipeline: begin_streaming -> FireRedTTS3StreamSession pull-based generator (AR loop emits one chunk per next_chunk(); chunk_patches config e.g. 3,12,12 -> first chunk ~0.5s, then ~2s). - fireredtts3 session: implement IStreamingVoiceTaskSession (PullEvents); parse fireredtts3.chunk_sizes session option; allow RunMode::Streaming. - model_specs/fireredtts3.json: add streaming mode + chunk_sizes session option. - app/server/firered_server.json: mode=streaming, chunk_sizes, model_spec_override. Verified: stream=True first chunk ~0.3s latency, then 2s chunks at ~0.62s intervals (RTF ~0.3); stream=False still returns whole utterance. Co-Authored-By: Claude <noreply@anthropic.com>
- kv_cache: TransformerBatchedKVState gains current_ends; TransformerBatchedKVCache
supports per-member ends (member_end/set_member_end/advance_member).
- qwen_causal_decode_runtime: run_batched_decode_step uses per-member positions
+ cache slots; positions tensor is {batch_size}; fixed max_batch graph reuse.
- qwen_causal_decoder: batched cached-step mask writer takes per-member ends+slots.
- qwen_decoder: batched RoPE applies per-member positions via permute (batch <-> token
axis) so different-prefix-length sequences batch correctly (llama.cpp-style).
Single-path regression passes (rms 0.048). Enables FireRedTTS3 batch scheduler.
Co-Authored-By: Claude <noreply@anthropic.com>
- FireRedArRuntime: expose start_decode_embeddings_batched / decode_embeddings_batched / export_batched_decode_state (thin wrappers over the Qwen backbone) so the scheduler can batch the AR decode. - RedAE codec: incremental decode state (decoder KV + ISTFT) hoisted into RedAeCodecRuntime::DecodeState, passed explicitly to decode_reset / decode_incremental / flush_incremental, enabling interleaved multi-slot streaming. FireRedRedAeRuntime threads the state; StreamSession owns a per-slot DecodeState. Co-Authored-By: Claude <noreply@anthropic.com>
…4, P2a-5) 真·batch 并发推理：多 session 共享一个 FireRedTTS3BatchScheduler， 并发请求各占一个 slot，单调度线程每轮把活跃 slot 的 AR decode 合并成 一个 batch（per-member position/mask/cache-slot），实现共享 GPU batch。 - batch_scheduler: llama.cpp update_slots 的模拟。固定 max_batch graph （kMaxDecodeCacheSteps=700 不重建），wave batching prefill（规避 CUDA pool 逆序 free 约束），step-0 AR 不 consume decode，stop/flush/RedAE 增量与 pipeline.cpp 一致。ggml CUDA pool 逆序约束 -> 唯一 GPU decode 线程（调度线程）。 - padded prefill: prefill_embeddings_padded 固定 kMaxPrefillSteps graph （grow-only），避免 prefill graph 重建破坏 pool；padding 行保持因果 mask（非 -inf，避免 softmax NaN 污染 KV）。 - per-member RoPE 修复: q 布局 [batch,steps,heads,dim]，permute(0,1,3,2) 把 batch 移到 token 轴（原 permute(1,0,2,3) heads/dim 交换导致 flash 输出 shape mismatch）。 - write_qwen_batched_cached_step_mask 修复: current_slot 是绝对位置， mask 是 batch-major 布局，需 offset + pos（原 offset + current_slot 越界）。 - server 集成: instance_count 注入 fireredtts3.max_batch，session 池共享 scheduler（session.cpp 注册表）。runtime.cpp 多实例池保留，每 session 变 slot shim。 - session 流式加锁 + free_slots_ 防重复归还，修复并发竞态崩溃。 验证: 单请求/连续请求/2/3 并发全部 HTTP 200，并发请求与单请求音频 逐位一致（MD5 相同）。正确性门通过。 Co-Authored-By: Claude <noreply@anthropic.com>
… true incremental streaming 调度线程在 next_chunk 的 cv_.wait 让锁期间持锁持续 tick，一次性生成完整个 slot， 导致所有 chunk 攒齐才返回，SSE 流式退化成离线（chunk 同时到达）。 修复：调度线程每 tick 后检测是否产出了新 chunk，有则 cv_wake_.wait_for(1ms) 短暂 让锁，让 next_chunk 抢到锁取走 chunk，实现真增量流式。 验证：44s 长文本流式 16 个 chunk 逐块到达（间隔 ~0.6s），每 chunk RTF≈0.32； 2 并发下两路 chunk 交错增量到达，整体 RTF 1.12。 Co-Authored-By: Claude <noreply@anthropic.com>
- istft_graph: finish_incremental 只输出未输出尾部，不再从 0 重放整段 （append 已输出 [pad,total-pad)，此前重放导致流式 2× 时长/末句重复） - batch_scheduler: finish_slot 的 tail decode 与 flush 改为 if/else 二选一， 避免 decode+flush 双输出；冻结非活跃 batch 行解码位置（set_batched_member_end）， 避免空行 mask 污染活跃行致 stop 判定错乱/内容重复 - qwen_decode_runtime/ar: 新增 set_batched_member_end 贯穿实现 - redae_codec_runtime: encoder/downsample 关 bf16_autocast（配合 per-slot 状态） - runtime/server: [REQ]/[LAUNCH]/[FINISH]/[REBUILD]/[SPLICE] 诊断日志； build_speech_request 透传 reference_text - pipeline.cpp: 清理调试残留（空块/孤立空行） 验证: 单路流式末句 20.35s（原 26.62s 2×），ASR 末句仅一遍；离线/流式对齐 16.32s Co-Authored-By: Claude <noreply@anthropic.com>
根因：finish_slot_locked 在 slot 变 Dead 时立即 push 回 free_slots_，而所属
session 尚未 drain 完尾块；新请求 launch 随即复用同 slot（slot=Slot{} 重置），
旧 session 的 next_chunk 读到新请求状态（串音），Slot{} 析构与 drain 交错
导致 free(): double free。
修复：
- Slot 增加 epoch；launch 返回 SlotHandle{id, epoch}；next_chunk 首行校验
epoch，stale 句柄立即返回空（不读新请求状态，防串音）
- finish_slot 不再 push free_slots_（slot 留保留态等 owner），owner 在
next_chunk 见空/reset/generate drain 完时显式 release_slot 归还
- 新增 abort（Active→Dead，reset/异常清理用）
- session scheduler_slot_ 句柄化；next_stream_event 见空即 release
- streaming run_stream catch 补 session.reset()（中断请求遗留 slot 立即回收）
验证：3 路并发无 double-free，slot 生命周期严格
LAUNCH(epoch)→FINISH→DONE_WAIT_RELEASE→RELEASED→LAUNCH。
注：并发串音（GPU decode 层）独立于本崩溃，另定位。
Co-Authored-By: Claude <noreply@anthropic.com>
两个独立修复，消除并发冷启动的串音与崩溃根因：
1. server session 池加载竞态（ensure_model_loaded_locked）：
- 旧逻辑仅当 max_loaded_models/min_free_memory_mb 配置时才持 model_load_mutex_
- 两者都没设时并发首请求同时进入，各自 clear()+push 同一 session 池 →
free_sessions 重复索引，所有请求借到同一 session（串音/截断的根因）
- 改为无条件持锁 + 以 model.loaded 为门，只建一个干净 session 池
2. 每 model 共享单一 ExecutionContext/backend（对齐 llama.cpp 单 context 多 slot）：
- 旧实现每 session 在 RuntimeSessionBase 自建 ExecutionContext（自建 CUDA
backend），instance_count=3 → 3 个 device-0 context；scheduler 借用第一个
session 的引用（session 析构即悬垂）→ 冷并发 "CUDA error: invalid device context"
- RuntimeSessionBase 增加"借外部 context"构造（nullptr=自建原行为，其余 ~46
个 session 不受影响）；共享注册表 assets-keyed 持 context，所有权随 session
- scheduler-mode（Base + max_batch>1）session 借用共享 context；Instruct/单
Base 保持各自独立 context（非单调度线程，共享会撞后端线程安全）
验证：单路回归 WER 0.14 不回归；并发冷启动不再 crash invalid device context。
注：并发请求在共享 scheduler 串行轮转下的内容残留/截断（同输入串行两请求
44 vs 3 patches）是独立 pre-existing bug，另定位。
Co-Authored-By: Claude <noreply@anthropic.com>
并发串音调查：scheduler KV 层确有 stale 行泄漏（slot 完成后 GPU 行带完整 K/V 不被 清零，Idle 行 freeze→advance 振荡且 mask 暴露 stale pos0），但非串音根因 —— 彻底重写后单路逐位一致、多路 decode 行互不污染，串音仍复现于 server session 并发 层（double-free + 并发 [REQ] 同 session_idx，另查）。 重写（round 制，删除 export/splice/rebuild/dirty/freeze 整机）： - 一轮并发请求 = 一次全量干净 import（只写本轮 prefill KV，其余行全零） - 每 tick 一次 batched decode 真并发推进所有活跃行 - runtime batched decode 新增 active_mask：非活跃行整行 -inf 且不 advance， 即使 cache 段残留上一请求内容也不 attend（inert 冻结） - 保留 AR/flow/RedAE 单路径语义（与 pipeline.cpp 逐位一致） - 保留 [ROUND]/[NAN]/[FINISH] 诊断日志 + batched_member_ends 只读 accessor 验证：单路流式 audio=7.04s / WER 0.1429，与重写前逐位一致（不回归）。 Co-Authored-By: Claude <noreply@anthropic.com>
根因是 SessionPoolLock 的双 index 字段 bug： - 头里 public `size_t index = 0` 与 private `size_t index_ = 0` 并存； - 构造函数只初始化 private index_，而所有调用处（sessions[pool.index]、 [REQ] 打印）都读 public index → 恒为 0； - 于是每个并发请求都借到 session 0 并绑 scheduler slot 0：新请求的 start_stream→reset() 会 abort 掉上一个还在跑的 slot → 截断在 chunk 边界、 内容互串、间歇 double-free。 修法：删掉重复的 private index_，让 public index 成为唯一字段 （构造/move/赋值/release 全用它）。 另：ensure_model_loaded_locked 是经典双重检查锁，但拿到 load_mutex_ 后 缺少二次 loaded 复查 → 并发首请求会各自完整加载一遍并重建 session 池。 补上锁内复查，确保只有首个请求真正加载。 验证：conc3 三路并发（不同文本/音色、同 seed）的输出与各自顺序 solo 逐位一致（MD5 全同）；单路不回归（audio=7.04s / WER 0.1429）。 scheduler 本身无需改动 —— 串音不在它，在 server 的 session 池借还。 Co-Authored-By: Claude <noreply@anthropic.com>
Owner
|
Hi @duj12 Thank you for the PR. Please split the PR into smaller, single-responsibility PRs to kepp PRs focused, and explain what a PR does in detail. We recently added a policy limiting each user to 3 concurrent PRs, including draft PRs. Please keep only keep three PRs you think I should review first. When changing shared framework code, explain which model families or routes were checked. Please provide runtime regression tests on the affected models (outputs, performance, memory, etc). Static analysis or reasoning is not enough. One important principle is that framework-level code shouldn’t be shaped just to benefit a single model. I’d suggest keeping the changes scoped to the model directory for now, and we can decide later which parts are general enough to be promoted into the framework. |
动机：profile 显示单路生成 flow(DiT) 占 56% 耗时，而多路并发只 batch 了 AR(23%)， flow 每路每 patch 串行 → 聚合吞吐上不去。 改动（batch_scheduler.cpp）： - decode_round_step 改两阶段：阶段A per-slot AR 后处理(stop/backbone/dit_cond3)； 阶段B 存活 slot 同步 flow denoise —— 同一 denoise 步把各 slot 的 x_in 拼成一次 flow.run(batch=2N)，按行拆回各自 CFG 合并；阶段C per-slot patch_encode/chunk/redae。 DiT 输出每行独立(flash batch 轴)，与各 slot 单独跑逐位一致。异构 steps 回退逐 slot。 - scheduler_loop 开轮收集窗口：pending 非空且无 round participant 时 wait 8ms 聚合同刻 请求再开轮，消除 round 拆轮（首请求抢先单独 prefill → flow batch 永不触发）。 - [PROF] 计时：AR decode / flow / redae 分项累计，finish 时打印。 验证（105 实测）： - 2-slot 同轮(members=[0,2])每路输出与各自 solo MD5 逐位一致（T1 1a419f52/T2 9d7d871d）。 - 吞吐 2 文本顺序 2.54 → 2-slot 同轮 3.11（+22%）；6 路因 HTTP 到达散布仍部分拆轮， aggregate RTF 升到 ~2.6。 - flow 图按需重建(batch 变 rebuild)；试过 fixed-capacity+零填充复用但破坏 DiT 输出，弃。 config：firered_server.json 默认 num_inference_steps=5（default_request_options）。 Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.