feat(owlv2): add CPU fp32/fp16 recipes and Einsum op support

[codykk]

Summary

Add verified CPU execution provider support for google/owlv2-large-patch14-ensemble with fp32 and fp16 recipes. Register the Einsum operator in the runtime checker with a conservative exact-match strategy to eliminate false OpUnsupportedError during static EP analysis. Highest outcome: L1 (perf PASS on both precisions).

Model metadata

What the model does: OWLv2 is an open-vocabulary object detection model that localizes objects in images given free-form text queries, without requiring predefined categories.

Primary user stories: Zero-shot object detection — detect arbitrary objects described by natural language prompts in any image, without retraining or fine-tuning.

Supported tasks: zero-shot-object-detection

Model architecture:

Owlv2ForObjectDetection (google/owlv2-large-patch14-ensemble)
├── Vision Encoder (ViT-Large/14)
│   ├── Patch embedding (14x14, dim=1024)
│   ├── Transformer encoder x 24
│   │   ├── Self-attention (16 heads, dim=1024)
│   │   ├── Feed-forward (1024 -> 4096 -> 1024, GELU)
│   │   └── Residual + LayerNorm
│   └── Post-layernorm
├── Text Encoder
│   ├── Token embedding (vocab x 768)
│   ├── Transformer encoder x 12
│   └── Final layernorm + projection
├── Class Head
│   ├── Dense projections (vision/text -> shared dim)
│   └── Einsum (equation: ...pd,...qd->...pq)  ← query-key inner product
├── Box Head (MLP: 1024 -> 1024 -> 4)
└── Objectness Head (MLP: 1024 -> 1024 -> 1)

• Source/confidence: pinned checkpoint google/owlv2-large-patch14-ensemble config + Owlv2ForObjectDetection source class (verified).

Validation and support evidence

Baseline

On main, winml build succeeds for OWLv2 but winml analyze reports OpUnsupportedError for the /class_head/Einsum node — no OpInputGenerator is registered for the Einsum op type, so static EP analysis cannot evaluate it.

Goal

• Effort: L1-light (recipe + single new InputGenerator)
• Goal ceiling: L1 (build + perf benchmark on CPU EP)
• Outcome: L1 (recipes + code fix + unit tests + findings)

Outcome

L1 PASS on CPU EP for both fp32 and fp16. EinsumInputGenerator registered with 12 equation combinations covering the OWLv2 class head's ...pd,...qd->...pq pattern. Conservative exact-match approach adopted per reviewer feedback.

Per-EP/device/precision results

Tier	EP	Device	Precision	Verdict	P50 Latency	Throughput	RAM Δ
L0	CPUExecutionProvider	cpu	fp32	PASS (build 131s)	—	—	—
L1	CPUExecutionProvider	cpu	fp32	PASS	15.1s	0.07 samples/s	+3.8 GB
L0	CPUExecutionProvider	cpu	fp16	PASS (build 151s)	—	—	—
L1	CPUExecutionProvider	cpu	fp16	PASS	14.1s	0.06 samples/s	+9.5 GB

Delta

Code changes (from baseline where Einsum had no InputGenerator):

• New: src/winml/modelkit/pattern/op_input_gen/einsum_input_generator.py — EinsumInputGenerator class with:
  • get_input_and_infinite_attribute_combinations(): 12 equation patterns (transpose, identity, matmul, dot, outer, element-wise, batched matmul variants, OWLv2 pattern)
  • derive_properties(): extracts 7 semantic features from equation string (has_ellipsis, has_explicit_output, is_full_reduction, has_repeated_labels, has_contraction, output_num_labels, inputs_share_all_labels)
  • get_infinite_property_names(): returns ["Inputs_shape", "Inputs_value", "Inputs_is_constant"] — attr_equation deliberately excluded (conservative exact-match)
• New: Registration in src/winml/modelkit/pattern/op_input_gen/__init__.py
• New: Unit tests in tests/unit/analyze/test_input_generators.py — TestEinsumDeriveProperties (8 parametrized tests) + TestEinsumEquationExecution (12 ORT execution tests)
• New: Recipes under examples/recipes/google_owlv2-large-patch14-ensemble/cpu/cpu/ (fp32 + fp16 configs, opset 17, batch_size=1, gelu_fusion + matmul_add_fusion)

Production recipe README remains untouched.

Analyze summary — /class_head/Einsum

Before (main branch)

/class_head/Einsum:
Status: OpUnsupportedError
Reason: No OpInputGenerator registered for op type "Einsum"

No runtime rule is generated for Einsum → any Einsum node is reported as unsupported.

After (this PR)

EinsumInputGenerator registered with 12 input combinations. attr_equation is a finite (exact-match) property — each distinct equation string generates its own rule, avoiding false generalizations across different contraction patterns (conservative approach per review feedback).

Rule matching key (finite properties for /class_head/Einsum):
attr_equation: ...pd,...qd->...pq
has_contraction: True
has_ellipsis: True
has_explicit_output: True
has_repeated_labels: False
inputs_share_all_labels: False
is_full_reduction: False
num_inputs: 2
output_num_labels: 2
Inputs_dim: 3

Result: rule keyed on exact equation → /class_head/Einsum matches → supported

ORT execution verification

Equation validated independently via ONNX Runtime (shapes from model graph):

OWLv2 /class_head/Einsum (equation: ...pd,...qd->...pq)
Input shapes: (1, 576, 768), (1, 2, 768)
Expected output shape: (1, 576, 2)
EP: CPUExecutionProvider
Status: PASS (TestEinsumEquationExecution covers this case)

Why supported and not rules_not_found

1. ...pd,...qd->...pq is explicitly listed in get_input_and_infinite_attribute_combinations()
2. attr_equation is a finite (exact-match) key in get_infinite_property_names()
3. Runtime checker generates a rule keyed on the exact equation string + derived semantic properties
4. When analyzing /class_head/Einsum, the node's properties match the rule exactly → supported

Reproduce commands

# Build fp32
winml build --config examples/recipes/google_owlv2-large-patch14-ensemble/cpu/cpu/zero-shot-object-detection_fp32_config.json --model google/owlv2-large-patch14-ensemble --output-dir $OUT/owlv2_fp32

# Build fp16
winml build --config examples/recipes/google_owlv2-large-patch14-ensemble/cpu/cpu/zero-shot-object-detection_fp16_config.json --model google/owlv2-large-patch14-ensemble --output-dir $OUT/owlv2_fp16

# Perf benchmark
winml perf -m $OUT/owlv2_fp32/model.onnx --ep cpu --iterations 5
winml perf -m $OUT/owlv2_fp16/model.onnx --ep cpu --iterations 5

# Unit tests (Einsum generator)
python -m pytest tests/unit/analyze/test_input_generators.py::TestEinsumDeriveProperties -v
python -m pytest tests/unit/analyze/test_input_generators.py::TestEinsumEquationExecution -v

[DingmaomaoBJTU]

Requesting changes for two issues in the Einsum runtime-checker coverage: equation semantics are currently excluded from matching, and the generic validation test skips at least one new equation. The OWLv2 recipes otherwise match existing conventions. Please also include a before/after analyze artifact for /class_head/Einsum; runtime-rule parquet files are external to this diff, so the artifact is needed to demonstrate that the shipped analysis result is supported rather than merely changing OpUnsupportedError to rules_not_found.

[DingmaomaoBJTU]

Could we avoid dropping attr_equation from the rule key until the equation is represented by derived semantic properties? derive_properties() currently keeps only num_inputs and the maximum rank, so equations with the same arity and rank but different contractions, ellipsis or broadcasting, repeated labels, reductions, or output permutations collapse to the same condition set. On an EP with equation-dependent support, that can either generalize a passing case to unsupported equations or make result processing report a conflict and omit the Einsum rule entirely. The conservative fix would be to match the exact equation, or derive a canonical semantic signature before treating the raw equation as infinite.

[codykk]

attr_equation is no longer in get_infinite_property_names(), so each distinct equation string participates in exact rule matching. Additionally, derive_properties() now extracts 7 semantic properties (has_ellipsis, has_contraction, has_repeated_labels, etc.) to further distinguish equations even if we relax exact matching in the future.

[DingmaomaoBJTU]

Could we add a dedicated parametrized test that executes every equation and checks its output shape? The generic validate_inputs() groups cases only by _validation_input_constraints and stops after one case in a group succeeds. This case has the same two input shapes as bij,bjk->bik above, so it is skipped after that case passes; changing this equation to an invalid one would still leave the current test green. A rule-signature test for same-rank, different-equation cases would also cover the matching issue.

[codykk]

Added TestEinsumEquationExecution — a dedicated parametrized test that builds a real ONNX Einsum node for each equation independently and verifies its output shape via ORT inference, bypassing the validate_inputs() grouping entirely. Also added test_same_rank_different_equations_produce_different_properties to assert that same-arity/same-rank equations produce distinct semantic signatures.