fix(layoutlm): resolve invoices question answering support

[fuliucansheng]

Summary

This PR adds support for DmitrySpartak/layoutlm-invoices by fixing task resolution for *ForQuestionAnswering architecture heads when Optimum cannot infer a task, then adding validated CPU recipes for fp32 and fp16 LayoutLM invoice question answering. The shipped outcome is Outcome-L1: a class-derived resolver fix, regression coverage, and exact-evidence CPU recipes. L0 build, L1 perf, and L2 PyTorch-vs-ONNX comparison pass for both precisions; L3 task-metric eval is CLI-blocked today because the generic SQuAD question-answering evaluator does not supply LayoutLM bbox and token_type_ids inputs.

Model metadata

What the model does

DmitrySpartak/layoutlm-invoices is a LayoutLM document question-answering checkpoint for extracting answer spans from invoice text with OCR/layout coordinates. The checkpoint is cataloged locally as document-question-answering, while the export/runtime surface is canonical span question-answering with start_logits and end_logits.

Primary user stories

• A user supplies an invoice question plus tokenized invoice text and bounding boxes to obtain the answer span in the document.
• A document-processing workflow supplies OCR words and layout coordinates to extract invoice fields such as totals, dates, vendor names, or identifiers.

Supported tasks

Task	Support surface	Evidence	Confidence
document-question-answering	checkpoint/catalog	local catalog entry for DmitrySpartak/layoutlm-invoices	verified
question-answering	Transformers/Optimum/WinML export	LayoutLMForQuestionAnswering; LayoutLMQAIOConfig; current winml inspect resolves task question-answering	verified

Model architecture

LayoutLMForQuestionAnswering
|-- LayoutLMModel backbone
|   |-- Text/token embeddings
|   |-- 1D position embeddings
|   |-- 2D layout embeddings from bbox coordinates
|   |-- Token type embeddings
|   `-- Transformer encoder stack x 12
|       |-- Multi-head self-attention (12 heads)
|       |-- Feed-forward / GELU
|       `-- Residual + LayerNorm
`-- QA span head
    |-- start_logits
    `-- end_logits

Source/confidence: checkpoint config and HTP export hierarchy for LayoutLMForQuestionAnswering (verified).

Validation and support evidence

Baseline

• Base commit: 26a4a82b638f2169b5a2f75758abb31a60b1774e (origin/main when this branch was cut).
• WinML version: winml, version 0.2.0.
• Baseline diagnosis: origin/main has LayoutLMQAIOConfig registered for layoutlm/question-answering, but Optimum's TasksManager.infer_task_from_model(LayoutLMForQuestionAnswering) does not infer the task. Before this fix, auto-resolution fell through to the layoutlm family default (next-sentence-prediction), producing the wrong task/output contract (logits) for this checkpoint.
• Current branch inspection: winml inspect -m DmitrySpartak/layoutlm-invoices --format json resolves model_type=layoutlm, architectures=[LayoutLMForQuestionAnswering], task question-answering, exporter LayoutLMQAIOConfig, inputs input_ids, bbox, attention_mask, token_type_ids, and outputs start_logits, end_logits.

Goal

• Effort: L1, because the correct task is derivable from the published architectures[] suffix and generalizes to question-answering heads without naming this model.
• Goal ceiling: L2 on CPU, with L3 attempted and blocked by the evaluator/input-schema gap.
• Outcome: L1, with source resolver change, unit regression test, and CPU fp32/fp16 recipes.

Outcome

• Shipped code:
  • src/winml/modelkit/loader/resolution.py
  • tests/unit/loader/test_resolve_task.py
• Shipped recipes:
  • examples/recipes/DmitrySpartak_layoutlm-invoices/cpu/cpu/question-answering_fp32_config.json
  • examples/recipes/DmitrySpartak_layoutlm-invoices/cpu/cpu/question-answering_fp16_config.json
• Coverage: CPUExecutionProvider / cpu / fp32 and fp16 verified. No accelerator EP recipe is claimed.
• Production recipe README: unchanged.
• Methodology friction: no methodology changes are included in this model PR.

Per-EP/device/precision results

Tier	EP / Device	Precision	Verdict	Evidence
L0 build	CPUExecutionProvider / cpu	fp32	PASS	winml build ... --no-analyze --no-optimize --no-quant --no-compile --rebuild; build complete in 31.1s; final artifact model.onnx; IR 8 / opset 17; inputs [1,512], bbox [1,512,4]; outputs start_logits, end_logits; external data beside model (model.onnx.data 508,803,072 bytes).
L0 build	CPUExecutionProvider / cpu	fp16	PASS	winml build ... --precision fp16 --no-analyze --no-optimize --no-compile --rebuild; build complete in 30.2s; FP16 stage emitted 242.8 MB artifact; Model Precision: fp16; 208 FLOAT16 initializers and 12 INT64 initializers; external data beside model (model.onnx.data 254,401,536 bytes).
L1 perf	CPUExecutionProvider / cpu	fp32	PASS	winml perf ... --input-data temp/layoutlm_invoices_valid_inputs.npz --iterations 20 --warmup 3; Avg 121.32 ms, P50 122.92 ms, throughput 8.24 samples/sec, RAM total +454.9 MB.
L1 perf	CPUExecutionProvider / cpu	fp16	PASS	winml perf ... --input-data temp/layoutlm_invoices_valid_inputs.npz --iterations 20 --warmup 3; Avg 150.94 ms, P50 152.95 ms, throughput 6.63 samples/sec, RAM total +473.6 MB; perf printed Model Precision: fp16.
L2 compare	CPUExecutionProvider / cpu	fp32	PASS	LayoutLMForQuestionAnswering PyTorch reference vs ONNX Runtime on identical valid LayoutLM inputs: start_logits cosine 1.00000000 / max_abs 0.00002289; end_logits cosine 1.00000000 / max_abs 0.00001907.
L2 compare	CPUExecutionProvider / cpu	fp16	PASS	Same harness: start_logits cosine 0.99999998 / max_abs 0.00927544; end_logits cosine 0.99999999 / max_abs 0.00676346.
L3 eval	CPUExecutionProvider / cpu	fp32	CLI-BLOCKED	winml eval ... --task question-answering --samples 2 uses default rajpurkar/squad and fails with Missing required inputs: {'token_type_ids', 'bbox'}. Generic SQuAD QA schema has no LayoutLM bbox/OCR layout fields.
L3 eval	CPUExecutionProvider / cpu	fp16	CLI-BLOCKED	Same command and blocker: Missing required inputs: {'token_type_ids', 'bbox'}.

Note: winml perf on a raw ONNX path with random generated inputs can create invalid token_type_ids for this checkpoint (type_vocab_size=1). Perf evidence therefore uses a real .npz input file with valid monotonic bbox coordinates and all-zero token_type_ids.

Delta

• Code delta: add an architecture suffix fallback table for task heads when resolving config.architectures. The fallback currently maps ForQuestionAnswering to question-answering and only runs after resolving the architecture class but before falling back to broad family defaults. It does not branch on model id.
• Test delta: add a resolver regression for layoutlm + LayoutLMForQuestionAnswering, proving task and optimum task resolve to question-answering with TaskSource.TASKS_MANAGER.
• Recipe delta: add LayoutLM invoice QA recipes under the exact verified CPU bucket. The recipe pins the LayoutLM QA input order (input_ids, bbox, attention_mask, token_type_ids), uses bbox range [0,1000], keeps token_type_ids in the valid half-open [0,1] range for this checkpoint, and emits start_logits/end_logits.

Analyze summary - component level and op level

Static rule analysis completed with ANALYZE-PARTIAL-SUCCESS: winml analyze --ep all exited nonzero because VitisAI has no rule data, but the JSON is complete and parseable for QNN, OpenVINO, and VitisAI rule coverage.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	embeddings; 12x LayoutLM encoder attention/FFN/GELU blocks; QA span head	HTP tagged 638/638 export nodes; final optimized graph has 442 ops	QNN partial GELU-related Add/Div/Erf/Mul; OpenVINO supported; VitisAI no rule data
fp16	embeddings; 12x LayoutLM encoder attention/FFN/GELU blocks; QA span head	HTP tagged 638/638 export nodes; final FP16 graph has 444 ops	QNN partial GELU-related Add/Div/Erf/Mul; OpenVINO supported; VitisAI no rule data

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp32	442 ops / 16 types	Add 129; MatMul 97; Reshape 48; Transpose 48; Mul 37; LayerNormalization 25	OpenVINO: all 16 op types supported; QNN: 14 supported op types plus partial Add/Div/Erf/Mul; VitisAI: 16 unknown op types due missing rule data
fp16	444 ops / 16 types	Add 129; MatMul 97; Reshape 48; Transpose 48; Mul 37; LayerNormalization 25	OpenVINO: all 16 op types supported; QNN: 14 supported op types plus partial Add/Div/Erf/Mul; VitisAI: 16 unknown op types due missing rule data

Reproduce commands

$OUT32='temp/repro_layoutlm_invoices_fp32'
$OUT16='temp/repro_layoutlm_invoices_fp16'
uv run winml inspect -m DmitrySpartak/layoutlm-invoices --format json
uv run winml build -m DmitrySpartak/layoutlm-invoices -c examples/recipes/DmitrySpartak_layoutlm-invoices/cpu/cpu/question-answering_fp32_config.json -o $OUT32 --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild
uv run winml build -m DmitrySpartak/layoutlm-invoices -c examples/recipes/DmitrySpartak_layoutlm-invoices/cpu/cpu/question-answering_fp16_config.json -o $OUT16 --ep cpu --device cpu --precision fp16 --no-analyze --no-optimize --no-compile --rebuild
uv run python -c "import numpy as np; rng=np.random.default_rng(1); n=512; x0=rng.integers(0,900,size=(1,n),dtype=np.int32); y0=rng.integers(0,900,size=(1,n),dtype=np.int32); w=rng.integers(1,101,size=(1,n),dtype=np.int32); h=rng.integers(1,101,size=(1,n),dtype=np.int32); bbox=np.stack([x0,y0,np.minimum(x0+w,1000),np.minimum(y0+h,1000)],axis=-1).astype(np.int32); np.savez('temp/layoutlm_invoices_valid_inputs.npz', input_ids=rng.integers(0,50265,size=(1,n),dtype=np.int32), bbox=bbox, attention_mask=np.ones((1,n),dtype=np.int32), token_type_ids=np.zeros((1,n),dtype=np.int32))"
uv run winml perf -m $OUT32/model.onnx --input-data temp/layoutlm_invoices_valid_inputs.npz --ep cpu --device cpu --iterations 20 --warmup 3
uv run winml perf -m $OUT16/model.onnx --input-data temp/layoutlm_invoices_valid_inputs.npz --ep cpu --device cpu --iterations 20 --warmup 3
uv run winml analyze --model $OUT32/model.onnx --ep all --output $OUT32/analyze_all.json
uv run winml analyze --model $OUT16/model.onnx --ep all --output $OUT16/analyze_all.json
uv run winml eval --schema --task question-answering
uv run winml eval -m $OUT32/model.onnx --model-id DmitrySpartak/layoutlm-invoices --task question-answering --samples 2 --ep cpu --device cpu -o $OUT32/eval_squad.json --overwrite --format json

Quality gates run locally:

uv run ruff check src/ tests/
uv run mypy -p winml.modelkit
uv run pytest tests/unit/loader/test_resolve_task.py --tb=short --no-cov -m "not e2e and not npu and not gpu"
uv run pytest tests/unit/models tests/unit/loader tests/unit/datasets tests/unit/export --tb=short --no-cov -m "not e2e and not npu and not gpu"
uv run pytest tests/unit/commands tests/unit/config tests/unit/build tests/unit/compiler tests/unit/session tests/unit/eval --tb=short --no-cov -m "not e2e and not npu and not gpu"

Results: ruff passed, mypy passed, focused loader tests passed (21 passed), models/loader/datasets/export partition passed (1375 passed, 6 skipped, 1 xfailed), and commands/config/build/compiler/session/eval partition passed (2383 passed, 6 skipped, 2 deselected).