feat(eval): support audio classification#1212
Draft
ssss141414 wants to merge 2 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds generalized
audio-classificationevaluation support and closes #1131.Before this change,
winml eval --task audio-classificationfailed before dataset loading withTask 'audio-classification' is not supported. This change registers a class-wide evaluator and schema without model-ID or dataset-specific branching.The evaluator:
Unsupported to supported
google/fleurstest rows execute successfully for both existing fp32 and fp16 CPU artifacts.Dataset and metrics
Checkpoint:
onecxi/open-vakgyata@f2754058e485dfc65cc62589b8d0e21c1d328399Dataset:
google/fleurs@70bb2e84b976b7e960aa89f1c648e09c59f894dd, test split, streaming, no shuffle, first 10 deterministic rows from each exact locale config:bn_in,hi_in,or_in,ta_in,te_in,kn_in,ml_in,mr_in,gu_inen_usis deliberately excluded because US English is not semantically equivalent to checkpoint classen-IN.The fp32/fp16 confusion matrices are identical; both metric deltas are zero. Macro-F1 is recomputed from the merged nine-class confusion matrix, not averaged across per-locale macro-F1 values.
Accuracy measures total utterance correctness. Macro-F1 gives each eligible locale equal weight and exposes class collapse (
or-IN,te-IN, andml-INhave zero recall in this bounded smoke sample). These are meaningful balanced smoke metrics, not a population FLEURS benchmark.Performance
CPUExecutionProvider, 100 measured iterations after 10 warmups:
Numerical parity
Same pinned checkpoint, feature extractor, named 16 kHz one-second waveform, and logits interpretation for PyTorch and ONNX:
ta-INpreservedta-INpreservedComponent analysis
Semantic regions: convolutional feature encoder; feature projection; Wav2Vec2 Transformer encoder; two stable-LayerNorm Transformer layers; temporal mean pooling; classification projector; ten-language classifier.
Operator-level analysis
Static rules classify all operator types as supported for NvTensorRTRTX GPU, QNN NPU/GPU, and OpenVINO NPU/GPU/CPU. CUDA, MIGraphX, DML, CPU, and VitisAI rows are unknown where the pinned rule set has no classification; this is static rule evidence, not runtime coverage.
Validation
mypy -p winml.modelkit: success, 411 source files.ruff check src tests: passed.git diff --check: passed.This PR is intentionally kept in Draft state pending final independent review and CI verification.