recipe(owlv2-large-patch14-finetuned): add CPU zero-shot detection configs

[fuliucansheng]

Summary

Adds CPU recipe coverage for google/owlv2-large-patch14-finetuned, an OWLv2 open-vocabulary zero-shot object detector exported through Optimum's OwlV2OnnxConfig and WinML's generic inference path. This is an L0 recipe-only contribution: current main can export and run the model, and this PR records verified CPU fp32/fp16 coverage with checked-in recipes.

Model metadata

What the model does: google/owlv2-large-patch14-finetuned detects objects described by text queries in an image. The checkpoint combines a CLIP-style text tower, a large ViT image tower, and object-detection heads that emit class logits, normalized boxes, and text/image embeddings.

Primary user stories:

Story	Confidence
A user supplies an image plus text prompts to obtain zero-shot detection logits and bounding boxes for open-vocabulary object localization.	verified from checkpoint config, processor metadata, and Owlv2ForObjectDetection architecture

Supported tasks:

Task	Surface	Confidence
zero-shot-object-detection	Transformers / Optimum ONNX / WinML export surface	verified

Model architecture:

Owlv2ForObjectDetection
├── Owlv2Model
│   ├── Text tower
│   │   ├── token embeddings (vocab 49408, max positions 16, hidden 768)
│   │   └── encoder stack x12 (self-attention + MLP)
│   └── Vision tower
│       ├── patch embedding (image 1008, patch 14, hidden 1024)
│       └── encoder stack x24 (self-attention + MLP)
├── Owlv2ClassPredictionHead -> logits [B, 5184, 1]
├── Owlv2BoxPredictionHead -> pred_boxes [B, 5184, 4]
└── Embedding outputs -> text_embeds, image_embeds

Source/confidence: pinned checkpoint config resolves model_type: owlv2, architectures: [Owlv2ForObjectDetection], image size 1008, patch size 14, vocab size 49408, text hidden size 768, vision hidden size 1024, and processor classes Owlv2Processor, CLIPTokenizer, and Owlv2ImageProcessor.

Validation and support evidence

Baseline

Baseline was run from current origin/main at ace130f20e36df44ebf7de6cf0356235c8ee79fd with winml, version 0.2.0.

Probe	Result
winml inspect	PASS: loader AutoModelForZeroShotObjectDetection, exporter OwlV2OnnxConfig, WinML class WinMLModelForGenericTask; task zero-shot-object-detection; default exported input order is displayed as pixel_values, input_ids, attention_mask.
Optimum coverage	VENDOR-ONLY for owlv2: vendor task includes zero-shot-object-detection; WinML adds no extra task override.
Recipe-free build	PASS: build complete in 141.5s; model loaded as Owlv2ForObjectDetection with 468 modules and 437.6M parameters; export tagged 1987/1987 nodes; final artifact emitted.
Recipe-free perf	PASS on CPU fp32, one-iteration smoke: avg/p50 6317.99 ms, throughput 0.16 samples/sec, RAM total +3622.3 MB.
L3 task metric	CLI-BLOCKED: winml eval --schema --task zero-shot-object-detection reports the task is not supported by the eval registry, so no task metric is available through the CLI today.

Baseline winml config -p fp32 --ep cpu --device cpu and winml config -p fp16 --ep cpu --device cpu are the sources for the checked-in recipes.

Goal

Effort: L0. Goal ceiling: L1. Outcome: L0 recipe-only.

Success means the checked-in CPU fp32/fp16 recipes build structurally valid ONNX artifacts and run winml perf on CPU. L2/L3 are not claimed for this PR: the eval CLI has no zero-shot-object-detection task, and no task metric is reported.

Outcome

Highest verified goal: L1 PASS on CPU for both fp32 and fp16. Coverage is full for the targeted EP set (cpu/cpu/fp32, cpu/cpu/fp16); no accelerator tuples are claimed.

Shipped recipes:

Recipe
examples/recipes/google_owlv2-large-patch14-finetuned/cpu/cpu/zero-shot-object-detection_fp32_config.json
examples/recipes/google_owlv2-large-patch14-finetuned/cpu/cpu/zero-shot-object-detection_fp16_config.json

Methodology friction observed: none. Model finding applied: OWLv2 checked-in recipes keep export.input_tensors in Owlv2ForObjectDetection.forward(input_ids, pixel_values, attention_mask) order so ONNX input names stay aligned when the recipe is used directly.

Per-EP/device/precision results

Tier	EP / Device	Precision	Verdict	Evidence
L0 build	CPUExecutionProvider / cpu	fp32	PASS	Recipe build complete in 115.5s; model is loadable with inputs input_ids [1,16] int32, pixel_values [1,3,1008,1008] float32, attention_mask [1,16] int32; outputs logits [1,5184,1], pred_boxes [1,5184,4], text_embeds [1,1,768], image_embeds [1,72,72,1024]; 1439 nodes; external data colocated with model.onnx.
L0 build	CPUExecutionProvider / cpu	fp16	PASS	Recipe build complete in 136.2s with --precision fp16; fp16 stage complete in 17.3s; FLOAT16 initializers present; model data is about 829.2 MB vs 1658.4 MB for fp32; 1444 nodes.
L1 perf	CPUExecutionProvider / cpu	fp32	PASS	One-iteration smoke: avg/p50 6152.99 ms, throughput 0.16 samples/sec, RAM total +3622.5 MB; perf reports model precision fp32.
L1 perf	CPUExecutionProvider / cpu	fp16	PASS	One-iteration smoke: avg/p50 about 12147.35 ms, throughput 0.08 samples/sec, RAM total +5623.4 MB; perf reports model precision fp16.
L3 eval	CPUExecutionProvider / cpu	fp32	CLI-BLOCKED	winml eval --schema --task zero-shot-object-detection reports unsupported task; supported task list does not include zero-shot-object-detection.
L3 eval	CPUExecutionProvider / cpu	fp16	CLI-BLOCKED	Same unsupported-task blocker as fp32; no task-metric value is reported.

Delta

Both recipes are generated from current winml config output with explicit CPU targeting, then refined to declare export.input_tensors in the model forward-signature order:

JSON pointer	Baseline value	Shipped value	Reason
/export/input_tensors/0/name	pixel_values	input_ids	Keep recipe input order aligned with Owlv2ForObjectDetection.forward(input_ids, pixel_values, attention_mask).
/export/input_tensors/1/name	input_ids	pixel_values	Same positional ONNX input-name guard.

The fp32 recipe preserves quant: null; the fp16 recipe preserves the generated fp16 quantization block and model metadata. No source code or tests are changed, and examples/recipes/README.md is untouched.

Reducibility: recipe-only is appropriate because WinML/Optimum already resolves the task, loader, exporter, and generic WinML inference class. This PR is a per-checkpoint verified CPU coverage declaration plus a checked-in forward-order recipe, not a class-wide source fix.

Analyze summary

Static rule analysis completed with partial-success exit semantics: QNN/OpenVINO rows were complete, while VitisAI has no rule data and is all-unknown. This is compatibility analysis, not runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	text tower x12; vision tower x24; class/box/objectness heads	1987 exported nodes tagged during HTP export; 1439 optimized ONNX ops analyzed	QNN NPU partial: GELU-related Div, Erf, Add, Mul; unknown Sqrt, Einsum, Cast. OpenVINO NPU: unknown Sqrt, Einsum, Cast. VitisAI: no rule data.
fp16	text tower x12; vision tower x24; class/box/objectness heads	1987 exported nodes tagged during HTP export; 1444 optimized ONNX ops analyzed	QNN NPU partial: Expand, GELU-related Div, Erf, Add, Mul; unknown Sqrt, Cast, Einsum. OpenVINO NPU: unknown Sqrt, Cast, Einsum. VitisAI: no rule data.

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp32	1439 ops / 27 types	Reshape 331; Add 327; MatMul 295; Transpose 181; Mul 117; LayerNormalization 76; Sigmoid 37; Softmax 36	QNN: 1432 supported, 2 partial, 0 unsupported, 5 unknown. OpenVINO: 1434 supported, 0 partial, 0 unsupported, 5 unknown. VitisAI: 1439 unknown.
fp16	1444 ops / 27 types	Reshape 331; Add 327; MatMul 295; Transpose 181; Mul 117; LayerNormalization 76; Sigmoid 37; Softmax 36	QNN: 1436 supported, 3 partial, 0 unsupported, 5 unknown. OpenVINO: 1439 supported, 0 partial, 0 unsupported, 5 unknown. VitisAI: 1444 unknown.

Reproduce commands

$MODEL = "google/owlv2-large-patch14-finetuned"
$OUT = "temp/repro_google_owlv2_large_patch14_finetuned"

uv run winml inspect -m $MODEL --format json
uv run winml config -m $MODEL -p fp32 --ep cpu --device cpu -o "$OUT/config_fp32.json"
uv run winml config -m $MODEL -p fp16 --ep cpu --device cpu -o "$OUT/config_fp16.json"
uv run winml build -c examples/recipes/google_owlv2-large-patch14-finetuned/cpu/cpu/zero-shot-object-detection_fp32_config.json -m $MODEL -o "$OUT/fp32" --ep cpu --device cpu --no-analyze --no-compile --rebuild
uv run winml build -c examples/recipes/google_owlv2-large-patch14-finetuned/cpu/cpu/zero-shot-object-detection_fp16_config.json -m $MODEL -o "$OUT/fp16" --ep cpu --device cpu --precision fp16 --no-analyze --no-compile --rebuild
uv run winml perf -m "$OUT/fp32/model.onnx" --ep cpu --device cpu --iterations 1 --warmup 0
uv run winml perf -m "$OUT/fp16/model.onnx" --ep cpu --device cpu --iterations 1 --warmup 0
uv run winml eval --schema --task zero-shot-object-detection
uv run winml analyze --model "$OUT/fp32/model.onnx" --ep all --output "$OUT/fp32/analyze_all.json"
uv run winml analyze --model "$OUT/fp16/model.onnx" --ep all --output "$OUT/fp16/analyze_all.json"