feat: Supporting Audio8_TTS models by jasonchen31 · Pull Request #333 · 0xShug0/audio.cpp

jasonchen31 · 2026-08-29T13:36:42Z

Audio8 TTS Preview — Native DualAR port

Summary

Native audio.cpp port of Audio8 TTS Preview as community family audio8_tts (DualAR: slow semantic → fast codebook → 44.1 kHz codec), derived from Fish Audio S2 Pro patterns. Ships as standalone GGUFs with no Python dependency.

0.6B Qwen (slow_backbone=qwen, 24L, dim 896, 14H/2KV, RoPE 1e6, packed QKV+bias) — fully native and CPU-verified via SenseVoice ASR round-trip.
0.1B Falcon-H1 (slow_backbone=falcon_h1, dim 512, d_inner 768=32×24, d_state 64, d_conv 4, dt_rank 24, n_group 1, GQA 8/2, RoPE 1e11, embedding_multiplier 0.1088/lm_head 0.0781) — weight-complete and builds natively (GGUF slow.embed_tokens + 24× mamba/attention + semantic_output 4097), slow AR currently a documented stub pending full Mamba2 port (see docs/FALCON_H1_0.1B_PORT_PLAN.md).

All 0.6B generation paths use ggml-native graphs; 0.1B not yet supported.

What’s in this PR

1. Community family `audio8_tts`

Spec: model_specs/audio8_tts.json (schema-v1, languages yue/zh/nl/en/fr/de/it/ja/ko/pl/es/auto, tasks tts/clon, packages audio8_tts_preview_0_6b_q8_0/audio8_tts_preview_0_1b_q8_0)
Docs: docs/community_models/audio8_tts.md (usage, architecture, GGUF packaging, validation notes), docs/community_models/models.md update, docs/FALCON_H1_0.1B_PORT_PLAN.md (M0–M4, 3–3.5d plan)
Tools: tools/community_models/convert_audio8_tts_codec.py (zipfile/pickle codec.pth → codec.safetensors, no torch) + tools/community_models/convert_audio8_tts.py (snapshot → standalone GGUF, embedded config/tokenizer/spec)

2. Runtime (`src/community_models/audio8_tts/`)

types.h / assets.cpp — flat ArkTTS config parsing including Falcon multipliers (embedding/lm_head/ssm/attn/mlp)
prompt_builder.* — ArkttsProcessor._prompt_segments chat template, reference-code placement, clean_text
ar.cpp — slow prefill/step + fast codebook graphs, embeddings (plain sum, no 1/sqrt(n)), RAS/top-k/top-p/Gumbel sampling
- 0.6B path: TransformerKVCache + QwenCausalDecoder (prefill writes into step cache, BF16 activation casts)
- 0.1B path: native BackendWeightStore load (slow.embed_tokens 69633×512, in_proj 1688×512, conv1d 896×1×4, dt_bias/A_log/D, out_proj, q/k/v/o, layernorms, semantic_output 4097) + stub falcon_forward_stateless:861 (TODO(Falcon-H1) — RMSNorm + in_proj split gate 768/xBC 896 + conv bias SiLU + gated out_proj + FFN, attn_out=0, no ggml_ssm_conv/B/C/dt/A/D/ggml_ssm_scan nor recurrent conv[3,896]/ssm[64,32,24] state, O(N²) full recompute, expand_compact 1024+EOS→4097). Full Mamba2 tracked as mamba-base.cpp:151/falcon-h1.cpp:132 reuse of external/ggml SSM.
codec.* / generator.* / session.* / tokenizer_text.* — window-transformer codec (encode/decode), generation loop, session options (weight_type, codec_weight_type, mem_saver, reference_cache_slots)

3. WebUI

webui/configs/models_catalog.json + model_params.json — audio8-tts / audio8-tts-0.5b entries (GGUF Q8, multilingual hints)
webui/native/dist/index.html — merged upstream shug0/main via checkout --theirs on conflict, then rebuilt (webui/native: pnpm run build, vite 7.3.6, dist/index.html 599K) to include upstream Echo-TTS/VoxCPM/etc. + Audio8 entries.

Verification

0.6B Qwen — PASS (linux-cpu-release `audiocpp_cli`)

audiocpp_cli --model /workspace/models/Audio8-TTS-Preview-0.6b-GGUF/audio8-tts-preview-0.6b-q8_0.gguf \
  --family audio8_tts --task tts --threads 2 --text "<text>" --out /tmp/x.wav
ffmpeg -y -i /tmp/x.wav -ar 16000 -ac 1 -c:a pcm_s16le /tmp/tmp16k.wav
curl -s -X POST http://192.168.1.2:11533/v1/audio/transcriptions \
  -F file=@/tmp/tmp16k.wav -F model=sensevoice-small

Build: linux-cpu-release audiocpp_cli 100% (no regressions in engine_model_audio8_tts).

0.1B Falcon-H1 — documented stub

Builds and runs: 2.36s 44100Hz via falcon_forward_stateless stub → STT like. vs target (prompt-invariant logits, expected). No /tmp writes or Python dependency (cleaned in 04476ef).
Full correctness requires M2/M3 (ggml_ssm_conv/scan + state + hybrid attention) — plan docs/FALCON_H1_0.1B_PORT_PLAN.md estimates 3–3.5d; external/ggml kernels already present (cpu/cuda/metal/vulkan).

Files changed (highlights)

src/community_models/audio8_tts/*, include/engine/community_models/audio8_tts/*, model_specs/audio8_tts.json, CMakeLists.txt (audiocpp_add_model(audio8_tts))
tools/community_models/convert_audio8_tts*.py, docs/community_models/audio8_tts.md, docs/FALCON_H1_0.1B_PORT_PLAN.md
webui/configs/models_catalog.json, webui/native/dist/index.html (rebuilt), plus upstream merge delta (CMake, external/ggml CUDA, app/server model memory, docs/models/*, echo_tts/voxcpm1/granite5asr)

GGUF Creation — 0.6B & 0.1B (and where to download)

1. Download the HF source snapshots (PyTorch preview checkpoints)

The port consumes the official PyTorch preview checkpoints from the Audio8 Hugging Face org — not ONNX, not pre-quantized GGUF. Each snapshot contains config.json / model.safetensors / codec.pth / tokenizer.json / modeling_arktts.py.

# via huggingface-cli (recommended, resume-capable)
pip install -U huggingface_hub
hf download Audio8/Audio8-TTS-Preview-0.6b --local-dir /models/Audio8-TTS-Preview-0.6b --local-dir-use-symlinks False
hf download Audio8/Audio8-TTS-Preview-0.1b --local-dir /models/Audio8-TTS-Preview-0.1b --local-dir-use-symlinks False

# or via git-lfs
git lfs clone https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b
git lfs clone https://huggingface.co/Audio8/Audio8-TTS-Preview-0.1b

File list expected in each --model-dir: config.json, model.safetensors (0.6B: 1.2G, 0.1B: 324M), codec.pth (both 1.3G), tokenizer.json/tokenizer_config.json/special_tokens_map.json, modeling_arktts.py/modeling_arktts_codec.py/processing_arktts.py. License: Apache-2.0 — https://github.com/Audio8-AI/Audio8_TTS

If you already have a GGUF, skip to How to test — but the sources below are the canonical way to rebuild one.

2. Build the `audiocpp_gguf` converter

cmake -S . -B build --preset release  # or: cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target audiocpp_gguf -j"$(nproc)"
# binary at: build/bin/audiocpp_gguf

3. Two-step offline conversion (no Python `torch` required, no downloads)

Both models use the same two tools — the codec converter is torch-free (zipfile/pickle + numpy + safetensors), the packager embeds 681 tensors (226 AR + 455 codec) + config/tokenizer/spec into one standalone GGUF.

Step 1 — Convert `codec.pth` → `codec.safetensors` (once per snapshot)

# 0.6B
python3 tools/community_models/convert_audio8_tts_codec.py \
  /models/Audio8-TTS-Preview-0.6b/codec.pth \
  /models/Audio8-TTS-Preview-0.6b/codec.safetensors

# 0.1B — same tool, different dir
python3 tools/community_models/convert_audio8_tts_codec.py \
  /models/Audio8-TTS-Preview-0.1b/codec.pth \
  /models/Audio8-TTS-Preview-0.1b/codec.safetensors

The converter fuses *.parametrizations.weight.original{0,1} and *.weight_g/_v weight-norm pairs into plain *.weight keys and validates anchors from modeling_arktts_codec.py (quantizer.semantic_quantizer…, decoder.model.0.conv.weight, etc.).

Step 2 — Package one self-contained GGUF per precision

# 0.6B — Q8_0 (default shipping precision, AR q8_0 + codec q8_0, 1.4G)
python3 tools/community_models/convert_audio8_tts.py \
  --model-dir /models/Audio8-TTS-Preview-0.6b \
  --converter build/bin/audiocpp_gguf --type q8_0 \
  --name audio8-tts-preview-0.6b --output-dir gguf-out

# 0.6B — BF16 reference (AR BF16, codec BF16, 1.8G)
python3 tools/community_models/convert_audio8_tts.py \
  --model-dir /models/Audio8-TTS-Preview-0.6b \
  --converter build/bin/audiocpp_gguf --type bf16 \
  --name audio8-tts-preview-0.6b --output-dir gguf-out

# 0.1B — Q8_0 (812M, AR q8_0 + codec q8_0)
python3 tools/community_models/convert_audio8_tts.py \
  --model-dir /models/Audio8-TTS-Preview-0.1b \
  --converter build/bin/audiocpp_gguf --type q8_0 \
  --name audio8-tts-preview-0.1b --output-dir gguf-out

# 0.1B — BF16 reference (975M)
python3 tools/community_models/convert_audio8_tts.py \
  --model-dir /models/Audio8-TTS-Preview-0.1b \
  --converter build/bin/audiocpp_gguf --type bf16 \
  --name audio8-tts-preview-0.1b --output-dir gguf-out

# If a Q8_0 package audibly drifts, keep the codec at 16-bit (conv stacks are quant-sensitive):
python3 tools/community_models/convert_audio8_tts.py --model-dir /models/Audio8-TTS-Preview-0.6b \
  --converter build/bin/audiocpp_gguf --type q8_0 --name audio8-tts-preview-0.6b-q8_0-codec-bf16
# (then pass mixed storage via --keep-type mapping in audiocpp_gguf — see tool --help)

Outputs:

gguf-out/audio8-tts-preview-0.6b-q8_0.gguf (1.4G) / -bf16.gguf (1.8G)
gguf-out/audio8-tts-preview-0.1b-q8_0.gguf (812M) / -bf16.gguf (975M)

Each GGUF embeds two tensor namespaces model_weights.* + codec_weights.* plus embedded config.json/tokenizer.json/model_specs/audio8_tts.json, so audiocpp_cli/audiocpp_server load it standalone with --family audio8_tts.

4. Load the GGUF

# 0.6B Q8
./build/bin/audiocpp_cli --family audio8_tts --task tts \
  --model gguf-out/audio8-tts-preview-0.6b-q8_0.gguf \
  --text "The quick brown fox jumps over the lazy dog." --out tts.wav --metrics

# 0.1B Q8 (build-only check until Mamba2 lands)
./build/bin/audiocpp_cli --family audio8_tts --task tts \
  --model gguf-out/audio8-tts-preview-0.1b-q8_0.gguf \
  --text "Hello world" --out tts01.wav

TODO (follow-up PRs)

Streaming session path
Complete 0.1B Mamba2 port (ggml_ssm_conv/B/C/dt/A/D/scan + ring conv/ssm + hybrid attention K=1) and SenseVoice-verify out/*0.1b.wav

References

Audio8 TTS: https://github.com/Audio8-AI/Audio8_TTS — modeling_arktts.py / modeling_arktts_codec.py / processing_arktts.py are the source of truth
Llama SSM reference: porting/llama.cpp/src/models/falcon-h1.cpp + mamba-base.cpp:151 build_mamba2_layer

Torch-free zipfile/pickle reader that converts codec.pth into codec.safetensors with arktts tensor names, fusing new-style parametrization and legacy weight-norm pairs into plain conv/in_proj keys. Shape anchors and fusion math asserted against the checkpoint.

Packages the 226 AR tensors and 455 codec tensors into one-file GGUFs (bf16 and q8_0) with the schema-v1 model spec embedded.

Port of Audio8 TTS Preview 0.6B (model_type arktts) reusing the fish_audio DualAR implementation: slow semantic AR with RAS sampling, fast codebook AR, and window-transformer codec decode/encode. Plain embedding addition at semantic begin/end tokens (no semantic_scale), packed wqkv with bias on slow layers, flat config, and a schema-v1 model spec driving the spec-backed loader.

Covers local GGUF/safetensors usage, voice cloning with a reference clip, request/session options, and the conversion tools.

…sidual cont) Prompt now mirrors Audio8_TTS/*.py clean_text (CJK-aware whitespace, control-strip, speaker:0) so len(prefix) and code placement match processing_arktts.py. Codec l2_normalize_last and residual now ggml_cont the Transpose view before Div/Sub, fixing 9.6 residual and 89.7% code mismatch -> 0.7% on ana.wav 150f. Session now accepts VoiceCloning task.

Add audio8-tts (family audio8_tts, GGUF Q8) to models_catalog.json and audio8_tts params to model_params.json so it appears in the Models Tab and Studio model dropdown (task tts, clone via reference voice + transcript). Rebuild webui + audiocpp_server.

Convert 0.1B codec.pth -> codec.safetensors (455 tensors) and model.safetensors + codec -> GGUF bf16/q8_0 (874 tensors, 975M/812M) via convert_audio8_tts*.py; add packages for 0.1B and placeholder 1.0B in model_specs, catalog entries for WebUI, and Falcon-H1/Mamba detection in types/assets/ar (slow_backbone, mamba_*). Qwen 0.6B/1.0B remains fully functional; 0.1B/1.0B-Mamba now loads and reports clear 'not yet implemented' TODO instead of missing embeddings. Docs: docs/audio.cpp/2026-08-27_0819_audio8_tts_0_1b_1_0b_support.md

No 1.0B checkpoint exists — catalog/spec now expose only 0.6B (Qwen) and 0.1B (Falcon-H1/Mamba). Rebuilt webui (pnpm) + server.

Drop audio8_tts_preview_0_6b_bf16 and 0_1b_bf16 from spec; q8_0 remains default for both 0.6B and 0.1B (WebUI already q8_0-only).

Stub native Mamba graph and route Falcon-H1 (0.1B) inference via HF ArkttsModel fallback (falcon_bridge.py) so 0.1B is STT-verifiable until ggml_ssm_conv/scan hybrid is landed (llama.cpp mamba-base.cpp).

Remove falcon_bridge.py/torch_bridge and restore native load path; load Falcon-H1 Mamba tensors (in_proj/conv1d/dt/A/D/out + attn q/k/v/o) from HF safetensors via llama.cpp falcon-h1.cpp/mamba-base.cpp:149. 0.6B Qwen still native; 0.1B now loads weights and fails with clear native-graph TODO (ssm_conv/scan hybrid) instead of python.

- Parse Falcon multipliers (embedding 0.1088, lm_head 0.078, ssm/attn/key/mlp) from config.json into Audio8TtsTextConfig - Load Falcon-H1 hybrid weights (slow.embed_tokens, 24x mamba.in_proj/conv/out + q/k/v/o + layernorms + semantic_output 4097) via BackendWeightStore - Implement stateless FalconH1 forward via raw ggml: RMSNorm + Mamba in_proj split gate/xBC + conv bias SiLU + gated y + out_proj + zero-attn stub + FFN, final RMSNorm + compact 4097 logits * lm_head_multiplier + hidden slice - Wire 0.1B generate path: build_falcon_embeddings * embedding_multiplier, falcon_forward_stateless per step O(n^2) recomputing full history, expand compact logits (0..4095 -> semantic 65537..69632, 4096 -> eos 228) for RAS/top-p sampling, drive fast 10-codebook AR via existing fast_graph, maintain full_matrix [11, steps] history - Keep 0.6B Qwen path unchanged (prefill/step KV cache); both 0.6B and 0.1B now build and generate 44.1kHz audio (0.6B ASR 'The quick brown fox...', 0.1B 2.28s RMS 0.038, clone 4.37s RMS 0.031) without torch dependency - References: modeling_arktts.py FalconH1Model, mamba-base.cpp:149 build_mamba2_layer, falcon-h1.cpp hybrid

…wback - Add detailed 0.1B port plan docs/FALCON_H1_0.1B_PORT_PLAN.md (M0-M4, 3.5d) reusing vendored external/ggml ssm_conv/scan (cpu/cuda/metal/vulkan) no ggml fork; scope 0.1B only K=1; hybrid Mamba2+attn via raw ggml - Document current native drawback stub falcon_forward_stateless:864: - missing ggml_ssm_conv/B/C/dt/A/D/scan, zero-attn (scale 0), no recurrent conv[3,896]/ssm[64,32,24] ring + KV, O(N^2) full recompute, only ssm_out/lm_head multipliers -> identical logits md5 7426eed2 STT fail 还过没./系。 vs HF modeling_arktts.py FalconH1Model - temp fallback to Python HF delegate via /tmp + system() (/workspace/.torch_venv/bin/python /tmp/gen_01b_for_cpp.py --text-file /tmp/falcon_prompt_*.txt -> /tmp/falcon_codes_*.bin) hard-coded /tmp writes flagged for removal Refs: porting/llama.cpp/src/models/falcon-h1.cpp + mamba-base.cpp:151, external/ggml ssm backends already built, next: hybrid layer + state

Remove hard-coded Python delegate (hard-coded /workspace/.torch_venv + /tmp/gen_01b_for_cpp.py + /tmp/falcon_prompt_*.txt/.bin + system()) and restore native ggml-only path. - ar.cpp: drop <cstdio>/<fstream>/<functional> includes, keep native falcon_forward_stateless (RMSNorm + Mamba in_proj split + conv bias SiLU + gated out_proj + FFN) with TODO stub note for missing ggml_ssm_conv/B/C/dt/A/D/ggml_ssm_scan/recurrent conv/ssm state + hybrid attention (currently attn_out=0, full recompute O(N^2), only ssm_out/lm_head multipliers). Documented in docs/FALCON_H1_0.1B_PORT_PLAN.md M2/M3 and references mamba-base.cpp:151 / falcon-h1.cpp:132; reuses vendored external/ggml ssm backends (cpu/cuda/metal/vulkan) without fork. - 0.1B generates 0.55s audio via stub (STT "I.") until full Mamba2 port lands; 0.6B Qwen path unchanged. No /tmp writes, no Python dependency, no hard-coded paths — upstream-ready. Build: linux-cpu-release audiocpp_cli OK

… stub - Mark 0.6B Qwen fully native CPU-validated via SenseVoice ASR (3.02s/2.32s/2.97s examples) with GGUF q8_0/bf16, family audio8_tts - Document 0.1B Falcon-H1 hybrid Mamba2 status: weight-complete but slow AR is a stub (ar.cpp:861 TODO, attn_out=0, no ssm_scan/state, O(N^2) recompute) pending M2/M3 in docs/FALCON_H1_0.1B_PORT_PLAN.md (reuses vendored external/ggml SSM backends, no fork/hard-coded /tmp) - Update architecture section to differentiate backbones and multipliers, fix TODOs (streaming, clone validation, Mamba2 completion)

# Conflicts: # webui/native/dist/index.html

Remove audio8_tts_preview_0_1b_q8_0 package from model_specs and audio8-tts-0.1b entry from webui catalog (hidden until Falcon-H1 Mamba2 port lands). Keep 0.6B as sole visible package. Rebuild webui/native/dist/index.html.

Remove docs/FALCON_H1_0.1B_PORT_PLAN.md from index (was added in e5d76cf) and ignore it via .gitignore. File stays on disk for local reference (M2/M3 Mamba2 plan) but is not part of upstream history.

jasonchen31 added 19 commits August 25, 2026 18:12

Add Audio8 TTS GGUF packaging tool

55e44ec

Packages the 226 AR tensors and 455 codec tensors into one-file GGUFs (bf16 and q8_0) with the schema-v1 model spec embedded.

Document Audio8 TTS community model

cb0bdec

Covers local GGUF/safetensors usage, voice cloning with a reference clip, request/session options, and the conversion tools.

fix(webui+spec): drop audio8-tts-1.0b, keep 0.6b/0.1b only

a1af573

No 1.0B checkpoint exists — catalog/spec now expose only 0.6B (Qwen) and 0.1B (Falcon-H1/Mamba). Rebuilt webui (pnpm) + server.

chore(audio8_tts): remove bf16 packages, keep q8_0 only

efa582f

Drop audio8_tts_preview_0_6b_bf16 and 0_1b_bf16 from spec; q8_0 remains default for both 0.6B and 0.1B (WebUI already q8_0-only).

Update webui index.html

3b2907b

feat(audio8_tts): add Falcon-H1 torch bridge for 0.1B

f082ff0

Stub native Mamba graph and route Falcon-H1 (0.1B) inference via HF ArkttsModel fallback (falcon_bridge.py) so 0.1B is STT-verifiable until ggml_ssm_conv/scan hybrid is landed (llama.cpp mamba-base.cpp).

Merge remote-tracking branch 'shug0/main' into dev-audio8_tts

a0525f2

# Conflicts: # webui/native/dist/index.html

chore(audio8): hide 0.1B model from spec and webui

5e3833b

Remove audio8_tts_preview_0_1b_q8_0 package from model_specs and audio8-tts-0.1b entry from webui catalog (hidden until Falcon-H1 Mamba2 port lands). Keep 0.6B as sole visible package. Rebuild webui/native/dist/index.html.

chore: untrack Falcon-H1 0.1B port plan, keep local only

b71adee

Remove docs/FALCON_H1_0.1B_PORT_PLAN.md from index (was added in e5d76cf) and ignore it via .gitignore. File stays on disk for local reference (M2/M3 Mamba2 plan) but is not part of upstream history.

gqf2008 mentioned this pull request Aug 29, 2026

🔄 [处理中][wt-audio8-asr] feat: Add Audio8-ASR-0.1B community model port (audio8_asr) #336

Open

8 tasks

0xShug0 added the new model Request for new model support label Aug 29, 2026

Navigation Menu

feat: Supporting Audio8_TTS models - #333

feat: Supporting Audio8_TTS models#333
jasonchen31 wants to merge 19 commits into
0xShug0:mainfrom
jasonchen31:dev-audio8_tts

jasonchen31 commented Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jasonchen31 commented Aug 29, 2026

Audio8 TTS Preview — Native DualAR port

Summary

What’s in this PR

1. Community family audio8_tts

2. Runtime (src/community_models/audio8_tts/)

3. WebUI

Verification

0.6B Qwen — PASS (linux-cpu-release audiocpp_cli)

0.1B Falcon-H1 — documented stub

Files changed (highlights)

GGUF Creation — 0.6B & 0.1B (and where to download)

1. Download the HF source snapshots (PyTorch preview checkpoints)

2. Build the audiocpp_gguf converter

3. Two-step offline conversion (no Python torch required, no downloads)

Step 1 — Convert codec.pth → codec.safetensors (once per snapshot)

Step 2 — Package one self-contained GGUF per precision

4. Load the GGUF

TODO (follow-up PRs)

References

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

1. Community family `audio8_tts`

2. Runtime (`src/community_models/audio8_tts/`)

0.6B Qwen — PASS (linux-cpu-release `audiocpp_cli`)

2. Build the `audiocpp_gguf` converter

3. Two-step offline conversion (no Python `torch` required, no downloads)

Step 1 — Convert `codec.pth` → `codec.safetensors` (once per snapshot)