Skip to content

docs: add VRAM requirements + note SDPA attention dispatch gap - #45

Open
moduvoice wants to merge 1 commit into
Soul-AILab:mainfrom
moduvoice:docs/t4-turing-notes
Open

docs: add VRAM requirements + note SDPA attention dispatch gap#45
moduvoice wants to merge 1 commit into
Soul-AILab:mainfrom
moduvoice:docs/t4-turing-notes

Conversation

@moduvoice

Copy link
Copy Markdown

Motivation

While verifying this project's inference pipeline on a real NVIDIA Tesla T4 (16GB, Turing/sm_75 — a
common budget/free-tier GPU, e.g. Google Colab's free tier), I found some hardware-specific
behavior and a documentation gap that aren't currently covered. This PR adds that documentation
only — no code or behavior changes.

Changes

  • Add a "Hardware Requirements" section (previously absent): checkpoint download size (model.pt + model-svc.pt = 5.61GB; separate Preprocess bundle 6.92GB), measured peak inference VRAM (1.6–2.7GB), and measured real-time factor (RTF ≈ 0.66) — both SVS and SVC comfortably fit a 16GB T4.
  • Note in the Quick Start section that cli.inference has no --lyrics/--text argument — it only accepts pre-built, frame-aligned metadata JSON via --prompt_metadata_path/--target_metadata_path, so synthesizing custom lyrics currently requires running the full preprocess pipeline (vocal separation + ASR + MIDI transcription, ~6.92GB extra checkpoints) first.
  • Document a measured attention-backend config/reality mismatch: DiffLlama's top-level config._attn_implementation reports "sdpa", but all 22 transformer layers actually instantiate eager LlamaAttention at runtime (each layer builds its own LlamaConfig(...) in soulxsinger/models/modules/llama.py, which never passes through PreTrainedModel._autoset_attn_implementation(), so it silently falls back to the "eager" class default). Forcing the same weights through real SDPA (self_attn.__class__ swapped to LlamaSdpaAttention, no weight/architecture change) measured 2.31x faster inference and ~40% lower peak VRAM with no measurable output-quality regression. This looks unintentional and may be worth fixing in the attention dispatch code — flagging it here as a candidate issue/PR rather than fixing it in this docs-only change.

Testing

  • Measured on a real Tesla T4 16GB (Turing, sm_75), driver 550.163.01, torch 2.2.0+cu121, transformers 4.41.2.
  • Checkpoint sizes confirmed via the HF Hub API (model.pt 2,818,092,278 bytes, model-svc.pt 2,793,965,154 bytes; Preprocess bundle 6,917,669,445 bytes across 8 files).
  • VRAM/RTF measured running the README's own bash example/infer.sh verbatim (fp16, n_steps=32, cfg=3, control=score): 51.24s of audio synthesized in 33.77s wall time (RTF ≈ 0.659).
  • Attention finding verified by (1) runtime inspection — self_attn.__class__.__name__ across all 22 layers is LlamaAttention, not LlamaSdpaAttention, despite config._attn_implementation == "sdpa" — and (2) a controlled before/after benchmark (2 runs each, torch.cuda.synchronize() timing + torch.cuda.max_memory_allocated()): eager 34.05s/34.61s (avg 34.33s, peak VRAM 2.71GB) vs. SDPA-patched 14.81s/14.92s (avg 14.87s, peak VRAM 1.63GB) — 2.31x speedup, ~40% VRAM reduction, output magnitude within normal run-to-run variance.

Adds a missing Hardware Requirements section (checkpoint download
size, peak VRAM, RTF measured on a real Tesla T4 16GB), documents that
custom lyrics currently require the full preprocess pipeline since
cli.inference has no --lyrics/--text flag, and notes a measured
attention backend config/reality mismatch (config reports "sdpa" but
all 22 layers run eager LlamaAttention) worth fixing separately.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant