feat(eval): add translation quality evaluation

[ssss141414]

Summary

This follow-up to merged PR #1115 turns translation evaluation from an explicit CLI blocker into a supported, task-meaningful path for WinML encoder-decoder models. It adds a generalized translation dataset adapter, deterministic static-shape generation, corpus SacreBLEU/chrF2 metrics, precise sample accounting, and regression coverage without branching on a checkpoint name. CPU fp32 and fp16 both complete a pinned, shuffled 100-row WMT14 French-to-English evaluation with identical quality scores.

Model metadata

What the model does

Helsinki-NLP/opus-mt-fr-en is a text-to-text Marian encoder-decoder checkpoint that translates French input text into English output text. The encoder produces contextual hidden states; the autoregressive decoder produces vocabulary logits and updated key/value-cache tensors. Evidence: pinned checkpoint revision c4aed37b318c763fd177aa449b44e3b783cc6c02, checkpoint configuration, and exported encoder/decoder interfaces. Confidence: verified.

Primary user stories

• Supply French text and obtain English translation for cross-language communication or downstream processing. Evidence: checkpoint identity and declared translation pipeline task. Confidence: verified.

Supported tasks

• translation: checkpoint, Transformers, and WinML composite inference. Confidence: verified.
• text2text-generation: Transformers, Optimum ONNX, and the decoder component. Confidence: verified.
• feature-extraction: Optimum ONNX and the encoder component. Confidence: verified.

Model architecture

MarianMTModel (hidden width 512, 8 attention heads)
Γö£ΓöÇΓöÇ Encoder graph (static batch 1, sequence length 512)
Γöé   Γö£ΓöÇΓöÇ Token + sinusoidal position embeddings
│   └── Encoder layer × 6
Γöé       Γö£ΓöÇΓöÇ Self-attention
Γöé       Γö£ΓöÇΓöÇ Feed-forward network
Γöé       ΓööΓöÇΓöÇ Residual + LayerNorm
ΓööΓöÇΓöÇ Cached autoregressive decoder graph (capacity 512)
    Γö£ΓöÇΓöÇ Token + sinusoidal position embeddings
    ├── Decoder layer × 6
    Γöé   Γö£ΓöÇΓöÇ Self-attention
    Γöé   Γö£ΓöÇΓöÇ Encoder cross-attention
    Γöé   Γö£ΓöÇΓöÇ Feed-forward network
    Γöé   ΓööΓöÇΓöÇ Residual + LayerNorm
    ├── Key/value cache (6 key/value pairs, 8 heads × 512 × 64)
    └── LM head (512 → 59,514 vocabulary logits)

• Source/confidence: pinned checkpoint configuration, MarianMTModel/WinML wrapper source, traced module hierarchy, and ONNX graph interfaces (mapped).

Validation and support evidence

Baseline

• Base: current origin/main commit 67169d45c40ed8326065e87ed29d19623b79cbb4.
• The merged recipe(opus-mt-fr-en): add CPU fp32 and fp16 recipes #1115 recipes already build fp32/fp16 encoder and decoder artifacts and support component perf/parity, but winml eval --schema --task translation exits 2 because translation is absent from the evaluator registry.
• Existing recipe evidence was produced from semantically identical build configurations: all four winml_build_config.json payloads equal the current checked-in recipes after removing the build-generated auto: false field.
• Baseline model build: PASS; encoder and decoder artifacts emitted. Baseline component perf was approximately 71.773 ms encoder and 21.897 ms decoder on CPU fp32.
• Optimum/WinML already resolve Marian component export and composite inference; the unsupported surface was task evaluation, not model export.

Goal

• Effort: shared task-family evaluator support.
• Goal: meaningful L3 translation quality evaluation while retaining existing L0 build, L1 perf, and L2 encoder parity evidence.
• Outcome: add generalized translation registry/schema/adapter/generation/metric support, not checkpoint-specific behavior.
• Success requires the real dataset ΓåÆ preprocessing ΓåÆ static ONNX encoder ΓåÆ cached decoder generation ΓåÆ corpus metric path to complete for both fp32 and fp16, with inference errors propagated rather than hidden as skipped samples.

Outcome

• translation changed from CLI-unsupported to supported.
• Flat source/reference columns and WMT-style nested translation dictionaries are supported.
• Dataset language keys, multilingual-tokenizer language identifiers, and prompt prefixes are independently configurable.
• Static encoder inputs truncate to ONNX capacity. Decoder generation is greedy (num_beams=1) and bounded by the graph's static KV-cache capacity.
• Metrics are computed once at corpus level: SacreBLEU with 13a tokenization and standard character-only chrF2 (n_word_order=0, beta=2), both reported on the conventional 0ΓÇô100 scale.
• Runtime, tokenizer, and inference failures propagate. Only malformed dataset rows or malformed generated outputs are skipped and counted; an all-rejected run fails closed.
• CPU coverage is complete for the existing fp32/fp16 composite recipes. No accelerator runtime claim is made by this CPU evaluation.

Per-EP/device/precision results ΓÇö including perf and eval data

Goal ladder

Tier	Scope	Verdict	Evidence
L0	CPU fp32/fp16 encoder + decoder	PASS	Four ONNX graphs load with named static interfaces; fp16 artifacts contain FLOAT16 initializers.
L1	CPU component perf	PASS	Concrete latency, throughput, and RSS deltas below.
L2	Encoder ONNX vs PyTorch	PASS	fp32 cosine 0.9999999999995187, max abs 3.337860107421875e-06; fp16 cosine 0.999999184734255, max abs 0.004638791084289551. Decoder parity is not claimed because the harness does not reconstruct identical dynamic-cache state.
L3	CPU composite translation	PASS	Pinned WMT14 fr-en/test, deterministic shuffled 100-row subset, 100/100 evaluated for both precisions.

Perf

These are per-component CPU measurements from the semantically identical #1115 recipe artifacts (10 measured iterations after 3 warmups). Composite autoregressive wall time depends on generated sequence length, so component figures are retained rather than combined into a misleading constant latency.

EP / device	Precision	Component	Mean	p50	Throughput	RSS Δ	VRAM Δ
CPUExecutionProvider / cpu	fp32	encoder	70.032 ms	70.415 ms	14.28 samples/s	140.57 MB	0 MB
CPUExecutionProvider / cpu	fp32	decoder step	23.607 ms	23.677 ms	42.36 samples/s	243.33 MB	0 MB
CPUExecutionProvider / cpu	fp16	encoder	92.095 ms	92.482 ms	10.86 samples/s	291.08 MB	0 MB
CPUExecutionProvider / cpu	fp16	decoder step	26.441 ms	26.450 ms	37.82 samples/s	252.65 MB	0 MB

Eval

Dataset: wmt/wmt14, config fr-en, split test, revision b199e406369ec1b7634206d3ded5ba45de2fe696. The row schema is a nested translation mapping; source_lang=fr and target_lang=en match the checkpoint direction. The 3,003-row test split is document-grouped, so this evidence uses streaming seeded shuffle (seed=42) before taking 100 rows rather than biased first-N sampling. It is a reproducible engineering subset, not a claim of full-test benchmark equivalence. Hugging Face reports the dataset license as unknown; this PR makes no permissive-license claim.

EP / device	Precision	Selection	Accounting	SacreBLEU 13a	chrF2
CPUExecutionProvider / cpu	fp32	shuffled seed 42, 100 rows	attempted 100, evaluated 100, skipped 0	39.0807	63.2021
CPUExecutionProvider / cpu	fp16	shuffled seed 42, 100 rows	attempted 100, evaluated 100, skipped 0	39.0807	63.2021

Precision deltas are 0.0000 SacreBLEU and 0.0000 chrF2. The scale is consistent with the checkpoint card's full WMT14 report (BLEU 37.8 and chrF approximately 63.3), while preserving the subset qualification above.

On this Windows host the native process can terminate during provider teardown after the complete result JSON is written and is parseable. Both final runs were accepted only after validating JSON, exact 100/100/0 accounting, and zero INVALID_ARGUMENT, Inference failed, or Evaluation failed log matches; this is not represented as an inference failure.

Delta

No recipe changes are included; the production recipe README is untouched.

• Add WinMLTranslationEvaluator and lazy/registry wiring for translation.
• Add TranslationMetric with corpus SacreBLEU 13a and true chrF2, rather than averaging sentence metrics or mislabeled chrF++.
• Add translation task schema with composite encoder/decoder roles and explicit source/target/tokenizer-language/prefix controls.
• Expose max_encoder_length and max_decode_length from generic ONNX metadata on WinMLEncoderDecoderModel; reject a non-positive/dynamic decoder KV capacity where static cached generation cannot be safe.
• Add regression coverage for nested and flat data, multiple references, language direction, tokenizer IDs, prompt prefixes, empty/malformed rows and outputs, runtime-error propagation, fail-closed behavior, registry/schema, corpus metrics, and tokenizer-free pipelines.

Bug fix explanation:

1. Symptom/trigger: recipe(opus-mt-fr-en): add CPU fp32 and fp16 recipes #1115 explicitly records winml eval --schema --task translation as unsupported; attempting Marian's checkpoint-default 4-beam generation against a static batch-1 decoder also expands decoder inputs to batch 4 and fails ORT dimension validation.
2. Root cause: no translation evaluator/metric/schema existed, and generic Transformers generation defaults did not honor the static ONNX encoder and KV-cache capacities.
3. Changed mechanism: register a task evaluator, adapt flat/nested datasets, use deterministic greedy generation, truncate the encoder to its metadata-derived capacity, and cap generated tokens from the decoder cache shape.
4. General rule: behavior derives from task schema, user-provided language semantics, and ONNX I/O metadata. There is no Helsinki-NLP/opus-mt-fr-en or model-name branch.
5. Compatibility/blast radius: existing evaluator defaults and task registrations are unchanged. New encoder/decoder properties are read-only. Existing static cached models already require a positive concrete cache dimension; malformed metadata now fails early with a clear error. Pipelines without a tokenizer remain supported.
6. Regression evidence: final commands/evaluator partition 2400 passed, 8 skipped, 2 deselected; Marian regression 25 passed; focused translation/shared-model suite 81 passed; final translation suite 19 passed; Ruff passed; Mypy passed for 412 source files; git diff --check passed. Two real 100-row fp32/fp16 CLI evaluations completed as reported above.

Analyze summary ΓÇö component level and op level

Static rule analysis completed with parseable data for all requested EP rule groups; command exit 1 was caused by unavailable host runtime/plugin registration, so status is ANALYZE-PARTIAL-SUCCESS. This is static compatibility analysis, not runtime execution on those accelerators.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32 encoder	embeddings/input; 6× self-attention, FFN, residual/normalization	204 mapped, 0 unmapped	No partial/unsupported op type in TensorRT, QNN, or OpenVINO rule sets.
fp16 encoder	same encoder regions	205 mapped, 0 unmapped	No partial/unsupported op type in TensorRT, QNN, or OpenVINO rule sets.
fp32 decoder	embeddings/input; 6× self-attention, cross-attention, FFN, residual/normalization; KV update; LM head	392 mapped, 0 unmapped	QNN GPU partial: Expand, Cast, Concat; ScatterND remains unknown in several accelerator rule sets.
fp16 decoder	same decoder regions	418 mapped, 0 unmapped	QNN NPU partial: Expand; QNN GPU partial: Expand, Concat, unsupported: Cast; ScatterND remains unknown.

Functional MLP and residual/normalization regions are mapped from graph scopes/topology rather than separate traced module boundaries. The model-breakdown topology used fp32 graphs; fp16 topology was checked separately against the converted artifacts.

Op-level summary

Artifact	Graph	Dominant ops	Static EP roll-up
fp32 encoder	204 ops / 16 types	Reshape 61; Gemm 36; Transpose 24; Add/Mul 19 each	TensorRT/QNN/OpenVINO rule sets classify all 16 types supported.
fp16 encoder	205 ops / 16 types	Reshape 61; Gemm 36; Transpose 24; Add/Mul 19 each	Same supported roll-up; one additional Cast node.
fp32 decoder	392 ops / 20 types	Reshape 112; Gemm 60; Transpose 54; Add 33; Mul 32	QNN GPU partial as above; ruleful groups otherwise support known types, with ScatterND unknown.
fp16 decoder	418 ops / 20 types	Reshape 112; Gemm 60; Transpose 54; Add 33; Mul 32; Cast 30	QNN NPU/GPU findings as above; ScatterND unknown.

Rule-less CPU, CUDA, MIGraphX, DML, and VitisAI rows report all op types unknown and are not runtime support claims.

Reproduce commands

$MODEL = 'Helsinki-NLP/opus-mt-fr-en'
$OUT = 'temp/opus-mt-fr-en-translation-eval'
$RECIPES = 'examples/recipes/Helsinki-NLP_opus-mt-fr-en/cpu/cpu'

winml build -c $RECIPES/translation_fp32_encoder_config.json -m $MODEL -o $OUT/fp32-encoder
winml build -c $RECIPES/translation_fp32_decoder_config.json -m $MODEL -o $OUT/fp32-decoder
winml build -c $RECIPES/translation_fp16_encoder_config.json -m $MODEL -o $OUT/fp16-encoder
winml build -c $RECIPES/translation_fp16_decoder_config.json -m $MODEL -o $OUT/fp16-decoder

winml perf -m $OUT/fp32-encoder/model.onnx --task feature-extraction --ep cpu --device cpu
winml perf -m $OUT/fp32-decoder/model.onnx --task text2text-generation --ep cpu --device cpu
winml analyze --model $OUT/fp32-encoder/model.onnx --ep all --device all --output $OUT/analyze-fp32-encoder.json
winml analyze --model $OUT/fp32-decoder/model.onnx --ep all --device all --output $OUT/analyze-fp32-decoder.json

winml eval `
  -m encoder=$OUT/fp32-encoder/model.onnx `
  -m decoder=$OUT/fp32-decoder/model.onnx `
  --model-id $MODEL --task translation --ep cpu --device cpu `
  --dataset wmt/wmt14 --dataset-name fr-en `
  --dataset-revision b199e406369ec1b7634206d3ded5ba45de2fe696 `
  --split test --samples 100 --streaming --shuffle `
  --column source_lang=fr --column target_lang=en `
  --format json --output $OUT/eval-fp32.json --overwrite --no-color