Skip to content

feat(eval): support audio classification#1212

Draft
ssss141414 wants to merge 2 commits into
microsoft:mainfrom
ssss141414:shzhen/add-audio-classification-eval-support
Draft

feat(eval): support audio classification#1212
ssss141414 wants to merge 2 commits into
microsoft:mainfrom
ssss141414:shzhen/add-audio-classification-eval-support

Conversation

@ssss141414

Copy link
Copy Markdown
Contributor

Summary

Adds generalized audio-classification evaluation support and closes #1131.

Before this change, winml eval --task audio-classification failed before dataset loading with Task 'audio-classification' is not supported. This change registers a class-wide evaluator and schema without model-ID or dataset-specific branching.

The evaluator:

  • requires an explicit semantically compatible labeled dataset;
  • validates authoritative checkpoint/dataset label alignment;
  • decodes remote/archive audio through the dataset opener plus SoundFile without requiring TorchCodec;
  • supports static rank-2 waveform inputs through deterministic windowing and utterance mean-logit aggregation;
  • preserves and validates static rank-3+ extracted feature tensors;
  • reports accuracy, macro-F1, confusion matrix, and complete requested/eligible/selected/processed/rejected accounting.

Unsupported to supported

  • Baseline: audio-classification Eval exited before dataset loading as unsupported.
  • Candidate: direct pinned google/fleurs test rows execute successfully for both existing fp32 and fp16 CPU artifacts.
  • No model ID or FLEURS literal is present in shared evaluator source.

Dataset and metrics

Checkpoint: onecxi/open-vakgyata@f2754058e485dfc65cc62589b8d0e21c1d328399

Dataset: google/fleurs@70bb2e84b976b7e960aa89f1c648e09c59f894dd, test split, streaming, no shuffle, first 10 deterministic rows from each exact locale config:

bn_in, hi_in, or_in, ta_in, te_in, kn_in, ml_in, mr_in, gu_in

en_us is deliberately excluded because US English is not semantically equivalent to checkpoint class en-IN.

Precision Requested / processed / rejected Correct Accuracy 9-class macro-F1
fp32 90 / 90 / 0 52 0.5777777778 0.4714699307
fp16 90 / 90 / 0 52 0.5777777778 0.4714699307

The fp32/fp16 confusion matrices are identical; both metric deltas are zero. Macro-F1 is recomputed from the merged nine-class confusion matrix, not averaged across per-locale macro-F1 values.

Accuracy measures total utterance correctness. Macro-F1 gives each eligible locale equal weight and exposes class collapse (or-IN, te-IN, and ml-IN have zero recall in this bounded smoke sample). These are meaningful balanced smoke metrics, not a population FLEURS benchmark.

Performance

CPUExecutionProvider, 100 measured iterations after 10 warmups:

Precision Mean P50 P90 Throughput
fp32 44.948 ms 47.373 ms 49.018 ms 22.25 samples/s
fp16 47.749 ms 45.324 ms 54.581 ms 20.94 samples/s

Numerical parity

Same pinned checkpoint, feature extractor, named 16 kHz one-second waveform, and logits interpretation for PyTorch and ONNX:

Precision Cosine Max abs Top-1
fp32 0.99999999967 0.0003737211 ta-IN preserved
fp16 0.99999986842 0.0055872202 ta-IN preserved

Component analysis

Semantic regions: convolutional feature encoder; feature projection; Wav2Vec2 Transformer encoder; two stable-LayerNorm Transformer layers; temporal mean pooling; classification projector; ten-language classifier.

  • fp32: 129 mapped, 0 partial, 0 unmapped nodes.
  • fp16: 129 mapped semantic nodes plus two topology-level Cast nodes without a unique component tag; no actionable EP finding.

Operator-level analysis

  • fp32: 129 operators, 14 unique types.
  • fp16: 131 operators, 15 unique types; the delta is two Cast nodes.
  • Dominant counts for both: Reshape=32, Transpose=23, Gemm=19, LayerNormalization=15, Gelu=10, Conv=8, Add=7, MatMul=4.

Static rules classify all operator types as supported for NvTensorRTRTX GPU, QNN NPU/GPU, and OpenVINO NPU/GPU/CPU. CUDA, MIGraphX, DML, CPU, and VitisAI rows are unknown where the pinned rule set has no classification; this is static rule evidence, not runtime coverage.

Validation

  • Direct pinned FLEURS: 18/18 locale/precision result files valid; 180/180 processed; zero rejected.
  • Focused evaluator tests: 22 passed.
  • mypy -p winml.modelkit: success, 411 source files.
  • ruff check src tests: passed.
  • git diff --check: passed.

This PR is intentionally kept in Draft state pending final independent review and CI verification.

@ssss141414 ssss141414 added the model-scale-by-skill Model support PR created or maintained by the adding-model-support skill label Jul 26, 2026
Comment thread tests/unit/eval/test_audio_classification_evaluator.py Fixed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model-scale-by-skill Model support PR created or maintained by the adding-model-support skill

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants