model(wav2vec2): support facebook mms 1b asr

[fuliucansheng]

Summary

Adds support for facebook/mms-1b-all by routing Wav2Vec2 CTC ASR checkpoints through AutoModelForCTC instead of the ambiguous speech-seq2seq default, and adds a verified CPU fp32 automatic-speech-recognition recipe. Highest verified outcome on this host is L1 for CPU fp32: build, structural validation, analyze, and CPU perf all ran successfully.

Model metadata

What the model does

facebook/mms-1b-all is a Wav2Vec2 CTC automatic speech recognition model. It consumes raw audio samples (input_values) and emits CTC logits over a 154-token vocabulary for speech transcription/language-token decoding.

Primary user stories

• A user supplies a short waveform to obtain token logits for automatic speech recognition.
• A developer exports a Wav2Vec2 CTC checkpoint to ONNX and runs WinML CPU inference for ASR smoke testing.

Supported tasks

Task	Support surface	Evidence	Confidence
automatic-speech-recognition	checkpoint, Transformers, Optimum ONNX, WinML recipe	architectures=["Wav2Vec2ForCTC"]; winml inspect resolves ASR; fp32 recipe builds and perfs	verified

Model architecture

Wav2Vec2ForCTC
├── Wav2Vec2Model
│   ├── Convolutional feature encoder
│   ├── Feature projection (hidden size 1280)
│   └── Encoder stack x 48
│       ├── Self-attention (16 heads)
│       ├── Feed-forward / GELU
│       └── Stable LayerNorm + residual paths
└── CTC lm_head (1280 -> vocab 154)

Source/confidence: pinned checkpoint config and exported hierarchy metadata (verified).

Validation and support evidence

Baseline

• Base commit: 26a4a82b638f2169b5a2f75758abb31a60b1774e
• WinML version: 0.2.0
• Baseline inspect/config on main detected task automatic-speech-recognition but selected loader AutoModelForSpeechSeq2Seq; this is incorrect for architectures=["Wav2Vec2ForCTC"].
• Starting winml config -m facebook/mms-1b-all emitted loader.model_class: AutoModelForSpeechSeq2Seq, quant.mode: static, weight_type: uint8, activation_type: uint16.

Goal

• Effort: L1, class-wide code fix plus recipe.
• Goal ceiling attempted: L1 on reachable CPU fp32; CPU fp16 was attempted and failed in the fp16 conversion stage.
• Outcome: CPU fp32 recipe shipped; CPU fp16 recipe omitted because the exact tuple did not pass.

Outcome

• Code paths: src/winml/modelkit/models/hf/wav2vec2.py, src/winml/modelkit/models/hf/__init__.py.
• Recipe path: examples/recipes/facebook_mms-1b-all/cpu/cpu/automatic-speech-recognition_fp32_config.json.
• Tests: uv run pytest tests/unit/models/test_wav2vec2_mapping.py --tb=short --no-cov -m "not e2e and not npu and not gpu" -> 4 passed.
• Focused lint: uv run ruff check src/winml/modelkit/models/hf/wav2vec2.py tests/unit/models/test_wav2vec2_mapping.py -> all checks passed.
• Methodology friction observed: CPU fp16 for this >2GB model fails in ONNX shape inference serialization; no fp16 support claim is shipped.

Per-EP/device/precision results

Tier	EP / Device	Precision	Verdict	Mean	P50	Throughput	RAM delta	Task metric
L0	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-	-
L1	CPUExecutionProvider / cpu	fp32	PASS	119.41 ms	112.46 ms	8.37 samples/s	+3756.9 MB total	-
L0	CPUExecutionProvider / cpu	fp16	FAIL	-	-	-	-	fp16 conversion failed before artifact

CPU fp32 structural validation: ONNX IR 8, opset 17, input input_values [1, 16000], output logits [1, 49, 154]. External data layout is valid: model.onnx and model.onnx.data are in the same output directory.

CPU fp16 failure evidence: convert_float_to_float16 called ONNX shape inference and failed with google.protobuf.message.EncodeError: Failed to serialize proto on the 3.7GB model. The fp16 candidate recipe was removed from the final diff.

Delta

Recipe-vs-auto-config changes:

JSON pointer	Baseline value	Shipped value	Reason
/loader/model_class	AutoModelForSpeechSeq2Seq	AutoModelForCTC	CTC ASR checkpoint must load via the CTC head
/quant	static uint8/uint16	null	CPU fp32 recipe, no quantization
/export/dynamo	true	false	Matches existing checked-in CPU recipe convention

Code delta:

• Adds MODEL_CLASS_MAPPING for Wav2Vec2 automatic-speech-recognition -> AutoModelForCTC.
• Adds MODEL_CLASS_MAPPING for Wav2Vec2 audio-classification -> AutoModelForAudioClassification.
• Imports the mapping into the aggregate HF model-class registry.

examples/recipes/README.md is unchanged.

Analyze summary

Static rule analysis completed with usable results and exited nonzero only because one requested EP had no rule data.

Component-level summary:

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	conv feature encoder; 48x Wav2Vec2 encoder attention/FFN; CTC head	2510/2510 nodes tagged during export	QNN partial: Add, Div, Erf, Mul

Op-level summary:

Artifact	Graph	Dominant ops	EP roll-up
fp32	2047 ops / 13 types	Add 587; MatMul 482; Transpose 207; Mul 208; Reshape 192	OpenVINO NPU supported 2047/2047; QNN NPU 1991 supported / 56 partial / 0 unsupported

Reproduce commands

$OUT = "temp/repro_facebook_mms-1b-all_fp32"
uv run winml inspect -m facebook/mms-1b-all --format json
uv run winml build -m facebook/mms-1b-all -c examples/recipes/facebook_mms-1b-all/cpu/cpu/automatic-speech-recognition_fp32_config.json -o $OUT --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild
uv run python -c "import onnx; m=onnx.load('$OUT/model.onnx', load_external_data=False); print(m.ir_version, [(i.name, [d.dim_value or d.dim_param for d in i.type.tensor_type.shape.dim]) for i in m.graph.input], [(o.name, [d.dim_value or d.dim_param for d in o.type.tensor_type.shape.dim]) for o in m.graph.output])"
uv run winml perf -m $OUT/model.onnx --ep cpu --device cpu
uv run winml analyze --model $OUT/model.onnx --ep all --output $OUT/analyze_all.json
uv run pytest tests/unit/models/test_wav2vec2_mapping.py --tb=short --no-cov -m "not e2e and not npu and not gpu"
uv run ruff check src/winml/modelkit/models/hf/wav2vec2.py tests/unit/models/test_wav2vec2_mapping.py