recipe(bert): add dell-research-harvard/lt-un-data-fine-fine-es (feature-extraction, sentence-similarity)

[DingmaomaoBJTU]

Summary

Adds curated CPU float recipes for dell-research-harvard/lt-un-data-fine-fine-es — a Spanish BertModel sentence embedder (BETO-based, from hiiamsid/sentence_similarity_spanish_es) used for record-linkage / semantic similarity (config model_type = bert, architectures = [BertModel], hidden 768, vocab 31002, type_vocab_size = 2 → has token_type_ids, pad_token_id = 1). Ships fp32 + fp16 for both feature-extraction and sentence-similarity under cpu/cpu/ (all quant: null; no CPU quantized variant per repo convention).

This is a catalog-style CPU float recipe, stated honestly: the default winml build (no recipe) builds successfully for this model and emits a w8a16 quantized artifact. The contribution's value is supplying the CPU float convention — fp32 + fp16 under cpu/cpu/ (consistent with #1084 / #1112 / #1170 / #1182) instead of the default w8a16 — plus an L2 numeric + retrieval-ranking parity proof that the trained embedding head is preserved end-to-end. Claimed tiers: Effort L0 · Goal ceiling L2 · Outcome L0.

Contrast with the sibling e5-small recipe (#1182): that model's XLMRobertaTokenizer emits no token_type_ids, so its baseline build crashes at quantize. This model's classic WordPiece BertTokenizer does emit token_type_ids, so its baseline quantizes fine — the difference is tokenizer-driven, not a property of type_vocab_size = 2.

---

1. Recipe path(s)

• examples/recipes/dell-research-harvard_lt-un-data-fine-fine-es/cpu/cpu/feature-extraction_fp32_config.json
• examples/recipes/dell-research-harvard_lt-un-data-fine-fine-es/cpu/cpu/feature-extraction_fp16_config.json
• examples/recipes/dell-research-harvard_lt-un-data-fine-fine-es/cpu/cpu/sentence-similarity_fp32_config.json
• examples/recipes/dell-research-harvard_lt-un-data-fine-fine-es/cpu/cpu/sentence-similarity_fp16_config.json

Per task, fp32 and fp16 are byte-identical (git blob OID 659160eb… for feature-extraction, 6c1a3127… for sentence-similarity; quant: null float bucket — on CPU both realize as fp32, fp16 materializes on GPU/NPU). opset 17, batch 1, inputs input_ids[1,512] (range 0..31002) + attention_mask[1,512] + token_type_ids[1,512] (all int32), output last_hidden_state[1,512,768].

2. README row

None. Recipe-only, CPU-only — deliberately not added to the "passes fp16 eval on all 10 (EP, device) buckets" table, which would overclaim for a CPU-only recipe (consistent with #1084 / #1112 / #1170 / #1182).

3. Build output dir

temp/ltun_fe/ (feature-extraction) and temp/ltun_ss/ (sentence-similarity) — scratch, gitignored. model.onnx + model.onnx.data (416.9 MB float).

4. Build log

• Recipe FE: ✅ Build complete in 100.0s (Export 52.4 + Optimize 46.7; no quantize — quant: null).
• Recipe SS: ✅ Build complete in 92.3s (Export 47.1 + Optimize 44.4; same graph, output last_hidden_state[1,512,768]).
• ONNX IR 8, opset 17, external data co-located.

5. Appended findings

model_knowledge/bert.json → bert-002 (baseline quantize SUCCEEDS here → recipe supplies CPU float convention; refines/falsifies the over-broad reading of bert-001 — the e5 quantize crash is tokenizer-dependent). Lane A (skill repo); not part of this model PR's diff.

6. Optimum-coverage probe

bert feature-extraction is VENDOR-ONLY (added_by_winml = []; vendor onnx tasks [feature-extraction, fill-mask, multiple-choice, question-answering, text-classification, token-classification]). sentence-similarity is not a distinct Optimum onnx task — winml's task layer maps it onto the same AutoModel backbone export (the SS recipe builds the identical graph, confirmed). No winml-added exporter exists or is needed.

7. Claimed (Effort, Goal, Outcome)

L0 / L2 / L0. Baseline gate: winml build -m dell-research-harvard/lt-un-data-fine-fine-es (no -c) on main (origin/main = d9feefff, winml 0.2.0) — builds successfully: Export ✅ (35.3s, 416.9 MB) + Optimize ✅ (23.3s) + Quantize ✅ (45.6s) → model.onnx (127.8 MB, w8a16 uint8/uint16), Build complete 121.5s. The recipe does not fix a crash; it supplies the quant: null fp32 + fp16 float CPU variants per repo convention (the default w8a16 is not what we ship for CPU). Baseline HEAD == current origin/main (git rev-list --count HEAD..origin/main = 0).

8. Goal-ladder verdict table

Tier	Verdict	Evidence
L0 (build)	PASS	recipe-driven float build exit 0 (FE 100.0 s / SS 92.3 s, no quantize); ONNX IR 8 opset 17, inputs input_ids/attention_mask/token_type_ids [1,512] int32, output last_hidden_state[1,512,768], external data co-located
L1 (perf)	PASS	CPU/fp32 avg 623.22 ms, P50 609.55 ms, min 444.80 ms (bs=1, 512 tok)
L2 (numeric vs PyTorch)	PASS	6 Spanish texts: raw last_hidden_state max-abs 5.860e-04; mean-pooled+L2-normalized sentence-embedding cosine 1.000000 (min = mean); retrieval ranking identical (PyTorch == ONNX) and semantically correct (paraphrase pair + QA pair rank top)

Ceiling L2 reached; no downgrade.

9. Methodology-evolution declaration

Charter re-tiered L0★ → L0 during testing (recorded, not silent). The planner initially assumed the e5-style baseline-quantize crash would recur; the Step 1b baseline gate proved it does not for a WordPiece-BertTokenizer model (baseline quantizes to w8a16 fine). Per the anti-drift rule this bounced back to the planner for a re-issued charter (L0, catalog-style float recipe). No SKILL.md / agents/*.md edits; the refinement lives in bert-002 model knowledge.

10. Perf & eval data

EP / Device	Precision	Verdict	Mean	P50	Min	Task metric
CPUExecutionProvider / cpu	fp32	PASS	623.22 ms	609.55 ms	444.80 ms	N/A (L3 not marched — L2 ceiling)

winml eval not run (Goal ceiling L2). Target EPs = [cpu]; coverage partial (GPU/NPU host-unreachable here).

11. Component / op-level data

winml analyze: 384 total operators, 15 unique types, all supported on the analyzed EP (384/0/0/0, "Ready to deploy").

12. Reproducible commands

# baseline (main): builds successfully, emits w8a16 (NOT the CPU float convention)
winml build -m dell-research-harvard/lt-un-data-fine-fine-es -o temp\ltun_baseline

# recipe-driven float builds (this PR)
winml build -m dell-research-harvard/lt-un-data-fine-fine-es -c examples\recipes\dell-research-harvard_lt-un-data-fine-fine-es\cpu\cpu\feature-extraction_fp32_config.json -o temp\ltun_fe
winml build -m dell-research-harvard/lt-un-data-fine-fine-es -c examples\recipes\dell-research-harvard_lt-un-data-fine-fine-es\cpu\cpu\sentence-similarity_fp32_config.json -o temp\ltun_ss

# op coverage
winml analyze -m temp\ltun_fe\model.onnx

# L2 embedding + ranking parity vs PyTorch + L1 CPU timing: temp\ltun_l2.py
#   (6 Spanish texts, pad to 512, int32 ids, token_type_ids from BertTokenizer,
#    mean-pool + L2 normalize, compare last_hidden_state + sentence-emb cosine + argsort)
python temp\ltun_l2.py

[DingmaomaoBJTU]

APPROVE (independent reviewer; comment-only approval because same GitHub identity cannot use gh pr review --approve)

Coverage: full for the declared CPU target (CPUExecutionProvider / cpu); deferred EPs: none for this CPU-only PR. Note: the PR body says "coverage partial" due GPU/NPU host-unreachable, but its own target EP set is [cpu], so I treat the verified coverage as full for the shipped scope.

Evidence summary:

1. Diff scope / base

• PR recipe(bert): add dell-research-harvard/lt-un-data-fine-fine-es (feature-extraction, sentence-similarity) #1185 head 37697b340d470f8ecbc496a1c6ff822bef172cb5; base main at d9feefffccecfe47f08b7d2cfffb11fe57eb94df; detached worktree merge-base = d9feefffccecfe47f08b7d2cfffb11fe57eb94df.
• git rev-list --count HEAD..origin/main = 0 (not behind).
• gh api .../pulls/1185/files --jq '.[].filename' and local git diff --name-only origin/main...HEAD both show exactly these four files and nothing else:
  • examples/recipes/dell-research-harvard_lt-un-data-fine-fine-es/cpu/cpu/feature-extraction_fp16_config.json
  • examples/recipes/dell-research-harvard_lt-un-data-fine-fine-es/cpu/cpu/feature-extraction_fp32_config.json
  • examples/recipes/dell-research-harvard_lt-un-data-fine-fine-es/cpu/cpu/sentence-similarity_fp16_config.json
  • examples/recipes/dell-research-harvard_lt-un-data-fine-fine-es/cpu/cpu/sentence-similarity_fp32_config.json

2. Recipe correctness

• All four recipe JSONs parse successfully.
• Confirmed: opset 17, batch 1, inputs input_ids, attention_mask, token_type_ids all int32 shape [1,512]; value ranges [0,31002], [0,2], [0,2]; output last_hidden_state; optim.clamp_constant_values=true; quant:null; compile:null; loader model_class=AutoModel, model_type=bert; task matches filename (feature-extraction or sentence-similarity).
• Blob OIDs from git ls-files -s:
  • FE fp16/fp32 both 659160eba98de1ec86bdb70a430b5927f2f91a3d
  • SS fp16/fp32 both 6c1a31273ef2d5a1c50683f4d5de26cde1e7e9b5
• git diff --no-index confirms each fp16/fp32 pair is byte-identical.

3. Baseline gate (central honesty claim)

• Ran no-recipe baseline: winml build -m dell-research-harvard/lt-un-data-fine-fine-es -o C:\Users\qiowu\source\repos\wcli-review-ltun\rev_baseline.
• Parsed stdout: ✅ Build complete in 112.4s.
• Quantize stage present: ✅ Quantize 40.4s, precision uint8/uint16 (weight/activation).
• Built artifact is w8a16: winml_build_config.json has weight_type: uint8, activation_type: uint16; final external-data artifact totals ~127.8 MB (model.onnx 0.4 MB + model.onnx.data 127.4 MB).
• Therefore the PR's L0-not-L0★ framing is correct: baseline succeeds; recipe does not fix a crash.

4. Recipe L0 builds and ONNX structure

• FE fp32 recipe build: ✅ Build complete in 92.9s; no Quantize stage; winml_build_config.json quant = null; model.onnx.data co-located (416.79 MB).
• SS fp32 recipe build: ✅ Build complete in 94.7s; no Quantize stage; winml_build_config.json quant = null; model.onnx.data co-located (416.79 MB).
• ONNX validation with onnx.load(..., load_external_data=False):
  • FE: IR 8, ai.onnx opset 17, inputs input_ids/attention_mask/token_type_ids elem_type 6 shape [1,512], output last_hidden_state elem_type 1 shape [1,512,768].
  • SS: IR 8, ai.onnx opset 17, same input/output names, dtypes, and shapes.

5. Independent L2 parity

• I wrote and ran my own script (rev_l2_parity.py) against the freshly-built FE model.onnx, not the producer's temp script.
• Sentences: 5 Spanish sentences, padded/truncated to max_length 512, tokenizer-provided token_type_ids, ONNX inputs cast to int32, ORT CPUExecutionProvider.
• Raw last_hidden_state max-abs per sentence:
  • 9.357929e-05
  • 3.908873e-04
  • 3.681183e-04
  • 5.857944e-04
  • 9.212494e-04
• Overall max-abs: 9.212494e-04 (< 1e-3).
• Mean-pooled + L2-normalized paired embedding cosine: [1.0, 1.0, 1.0, 0.9999999404, 1.0]; min = 0.9999999404 (>= 0.9999).
• PyTorch-vs-ONNX retrieval ranking identical: true.

6. L1 perf / op-level body audit

• Re-ran winml perf -m rev_fe\model.onnx --device cpu --ep cpu: avg 686.55 ms, P50 654.23 ms, min 408.95 ms, fp32, CPUExecutionProvider. This is within ±20% of the PR body's 623.22 ms / 609.55 ms / 444.80 ms.
• Re-ran winml analyze -m rev_fe\model.onnx: 384 total operators, 15 unique types, QNNExecutionProvider(NPU) 384/0/0/0, Ready to deploy — matches the PR body's component claim.

7. Optimum coverage probe

• Re-ran probe for bert:
  • vendor tasks: feature-extraction, fill-mask, multiple-choice, question-answering, text-classification, token-classification
  • after ensure_hf_models_registered(): same list
  • added_by_winml = []
• Result: VENDOR-ONLY, matching PR body item 6 and supporting Effort L0.

8. PR body audit

• Tiers are stated as Effort L0 · Goal ceiling L2 · Outcome L0, not L0★.
• Body explicitly says baseline succeeds and emits w8a16; it does not claim to fix a crash.
• Item 9 discloses the L0★ → L0 re-tiering.
• README row omission is explicitly justified as avoiding overclaim for the all-EP README table.
• Numeric claims checked against independent reruns are either matched or within expected tolerance; my independent L2 numbers pass the stated thresholds.

Concerns: no blocking concerns. Only non-blocking wording note: item 10's "coverage partial" is confusing because the declared target EPs are [cpu]; for this PR's CPU-only shipped scope, coverage is full with no deferred EPs.