feat: EP registration + monitoring — subprocess isolation, structured failures, universal op-tracing dispatch, lazy CLI startup#1019
Open
tezheng wants to merge 43 commits into
Open
Conversation
Collaborator
|
Let’s exclude the .md files from this pr. |
Contributor
|
please
Thanks |
tezheng
force-pushed
the
feat/ep-registration-and-monitoring
branch
from
July 6, 2026 03:53
d242cd6 to
cc7a665
Compare
Collaborator
Author
E2E verification checklistRunnable verification protocol for this branch — see Verified on Intel/OpenVINO NPU host (author). Other platforms — please pick up the delta. Minimum smoke (~5 min with warm cache)uv run winml sys --format json
uv run winml perf -m microsoft/resnet-50 --device auto --ep auto --monitor
uv run winml perf -m ./convnext/model_opt_qdq.onnx --ep qnn --device npu --op-tracing basic
uv run winml perf -m t5-small --device cpuPlatforms wanted
How to classify failures
Anti-pattern grepEvery regression signature has a grep pattern in the doc's grep table. If you find Author's results (two runs, Intel host)
Raw command outputs under If you're picking up a platform
|
Collaborator
Author
PR #1019 E2E Verification PlanBranch: Context / why this doc existsPR #1019's own "Test plan" checklist references a three-commit chain A prior manual check on High-level goalIndependently re-verify every item in PR #1019's Test plan checklist High-level passing criteriaAll of the following must be true to close this verification:
TasksEach task is independent and can be picked up in any order, but Task 1 Task 1 — Full unit test suite baselineGoal: establish the true current pass/fail/skip counts on Command: Pass criteria:
Verdict recorded as: PASS if all failures fall into a known Task 2 — OpenVINO monitor unit testsGoal: confirm the second op-tracing monitor implementation Command: Pass criteria: all tests pass (PR claims 27). If the count differs, Task 3 — Perf/op-tracing dispatch unit testsGoal: confirm Command: Pass criteria: all tests pass (PR claims 45, 7 new for dispatch). Task 4 — CLI cold-start performanceGoal: confirm Commands: then verify Pass criteria: Task 5 —
|
| Vendor | EP name | Typical hardware | Directory to point WINMLCLI_EP_PATH at |
|---|---|---|---|
| Qualcomm | QNNExecutionProvider |
Snapdragon/Copilot+ NPU (arm64/arm64ec) | A QNN EP build dir, e.g. the onnxruntime-qnn pip package's Lib\site-packages\onnxruntime_qnn\libs\<arch>\ or a custom QNN EP build's x64\Release-equivalent |
| Intel | OpenVINOExecutionProvider |
Intel NPU/GPU/CPU (Core Ultra etc.) | A compiled OpenVINO EP build directory, <path>\x64\Release |
| AMD | VitisAIExecutionProvider |
AMD Ryzen AI NPU | A compiled VitisAI EP build directory, <path>\x64\Release |
Command (adjust path to an actual available EP build directory for
the vendor being tested — ask the user for the correct
WINMLCLI_EP_PATH target if none is on hand; do not fabricate a path):
$env:WINMLCLI_EP_PATH="<path-to-vendor-EP-build-dir>"; uv run winml sys
Pass criteria (apply per vendor tested): N truthful <vendor> row
versions surface (N = however many EP DLL variants exist in that
directory); Architecture/Driver columns render for NPU/GPU rows; any
[failed] rows show a compact Win32 reason string and a PE-read
fallback version rather than a blank/opaque failure — for whichever
vendor(s) were actually exercised.
Known gap to check for while running this task: winml sys's
top-level backend-readiness summary (_check_qnn_sdk() /
_check_openvino() in src/winml/modelkit/commands/sys.py) only
reports install/readiness status for QNN and OpenVINO — there is no
equivalent _check_vitisai() / AMD readiness check, even though
VitisAIExecutionProvider is a first-class entry in EP_DEVICE_SPECS.
On an AMD test machine, confirm whether this is an intentional scope
limit (VitisAI readiness reported some other way) or a genuine gap in
the sys command's backend summary that should be filed separately —
don't assume it's covered just because directory-sourced EP discovery
itself works for VitisAI.
Note: this task requires actual hardware/EP-build access for the
vendor(s) being tested. If no suitable directory is available for a
given vendor, record that vendor's row as BLOCKED (no test fixture),
not FAIL — do not skip it silently from the final report. It's fine for
different sessions/machines to cover different vendor rows over time
rather than requiring all three in one run.
Task 6 — winml perf baseline (no monitor)
Goal: confirm the EP registration/dispatch changes haven't
regressed baseline inference performance or exit codes.
Command (device/EP auto-detected by default; optionally force a
specific vendor with --ep qnn, --ep openvino, or --ep vitisai to
explicitly cover a particular combination rather than relying on
whatever the test machine auto-picks — useful when a machine has more
than one EP installed):
uv run winml perf -m microsoft/resnet-50
Pass criteria: exit code 0; throughput > 300 samples/sec. The
~2.5 ms latency figure is a reference point measured on one specific
device (Qualcomm/QNN NPU) — treat it as vendor-relative, not an
absolute target: record which EP/device combination was actually
exercised (the "Device" panel in the output states EP + version), and
flag as DEVIATION (not FAIL) if latency is in a plausible order of
magnitude for that vendor's NPU/GPU class but doesn't match the
~2.5 ms QNN reference number — different vendors (Intel/OpenVINO,
AMD/VitisAI, Qualcomm/QNN) are expected to have materially different
absolute latencies for the same model.
Task 7 — winml perf --monitor (hardware monitor)
Goal: confirm the HW monitor path still renders correctly and
doesn't throw monitor/EP errors after the registration changes.
Command:
uv run winml perf -m microsoft/resnet-50 --monitor
Pass criteria: HW monitor output renders (CPU/RAM/NPU utilization
etc.); no monitor- or EP-related errors or tracebacks in output.
Task 8 — winml perf --help wording check
Goal: confirm the --op-tracing help text was generalized away
from vendor-specific (QNN-only) language now that OpenVINO is a second
supported monitor.
Command:
uv run winml perf --help
Pass criteria: help text for --op-tracing mentions "Auto-selects
the monitor for the chosen EP" (or equivalent EP-agnostic phrasing) —
no leftover QNN-specific or vendor-specific wording implying only one
EP is supported.
Task 9 — Reconcile PR description with actual state
Goal: not a test — a documentation fix. After Tasks 1–8 are done,
summarize the discrepancies between the PR body's stated commit list /
baseline-failure claims and what's actually true on cc7a6650.
Pass criteria: a clear written summary (in the final report, not
necessarily a PR edit unless the user asks for one) of what changed
between what the PR says and what's actually on the branch now,
so a reviewer isn't misled.
Final report format
For each task: task number, verdict (PASS / FAIL / DEVIATION / BLOCKED),
one-line evidence summary, and a pointer to full command output if
retained. Close with the high-level goal's three passing criteria,
explicitly confirmed or not.
tezheng
pushed a commit
that referenced
this pull request
Jul 7, 2026
Re-verifying PR #1019's Test Plan on top of cc7a665 surfaced six independent regressions and stale tests. This closes all six. - Complete the session/ep_device facade migration: expose VALID_SOURCE_TAGS (public), _SHORT_TO_FULL and _format_bytes (facade-only, not in __all__) so the 10 direct-import bypass sites in src/ and tests/ route through the session facade. Satisfies test_ep_device_import_rule.py's architecture fitness function. - Update test_static_analyzer_cli.py to short-form EP names: the CLI --ep option now accepts only short names (qnn, openvino, vitisai) and canonicalizes device via Click Choice(case_sensitive=False), so a previously-unreachable assertion (device == "GPU") is now reached and corrected to "gpu". - Fix stale 2-tuple mock: _run_onnx_benchmark returns tuple[BenchmarkResult, Any]; the test mocked a bare MagicMock. - Remove the stale ONNX+op-tracing parse-time-rejection test: direct-ONNX op-tracing is now implemented, so the rejection the test asserted no longer exists. - Pass ep_device in TestFromOnnxDictDispatch: from_onnx() gained a required keyword-only ep_device arg in cc7a665's refactor. - Reconcile OpenVINO op-tracing tests with the shipped CLI-refusal stub: _resolve_ep_monitor unconditionally refuses OpenVINO op-tracing (Intel wheel lacks the CSV-dump mechanism) and has no NPU/GPU auto-fallback. Five tests written against an unshipped fallback design (sibling branch e6be765) are rewritten to assert the actual refusal behavior. Also on this branch: - onnx/__init__.py: make the lazy __getattr__ data-driven via a _LAZY_IMPORTS map + importlib, replacing the hardcoded name check. - qnn_monitor.py: fix a stale docstring reference to a nonexistent self._onnx_path attribute (the profiling session is built from the model handed to benchmarking, not a monitor-owned path). Incidental: pre-commit ruff-format normalized pre-existing format drift in several touched test files, and three non-behavioral lint fixes (TYPE_CHECKING import move, drop MagicMock alias, docstring wraps) were required to pass the whole-file lint gate. AMD/VitisAI hardware work and the winml sys backend-readiness gap remain out of scope.
tezheng
pushed a commit
that referenced
this pull request
Jul 9, 2026
…tale-test alignment Follow-up fixes on PR #1019 (branch feat/ep-registration-and-monitoring), found while re-verifying the Test Plan on top of cc7a665 / 5f101e0. Source hardening (commands/sys.py, session/ep_device.py): - WinMLEPRegistrationFailed gains a raw_error kwarg: the isolated-register path passes the child's last non-empty stderr line so a [failed] EP row shows the real exception instead of a mangled wrapper fragment (observed live: "exited 1: d."). The Win32 loader code is searched in the FULL wrapper message so it survives post-traceback tail noise; str() keeps the full wrapper for logs. - isolated_ep_register no longer wraps `yield` in try/except JSONDecodeError, so a caller-side JSONDecodeError propagates unchanged instead of being mislabeled "subprocess produced invalid JSON". - Escape ORT/plugin-supplied text before Rich rendering at every sys render site (EP error reason, device error, per-source device facts, device name) so bracketed spans render literally and never raise MarkupError. Stale-test alignment (tests/): - test_main.py: the _mock_hw_detection fixture mocked _gather_ep_info as a list (pre-cc7a665 contract); it now returns dict[str, {entries:[...]}]. Update the mock and the executionProviders isinstance assertion to dict (verified against live `winml sys --list-ep --format json`). - test_ep_path.py: gate the two PyPISource "installed distribution" tests on the optional onnxruntime-ep-openvino wheel (the [openvino] extra CI installs via `uv sync --all-extras`) so a base local sync skips cleanly. All fixes developed test-first. Full unit suite: 5162 passed, 0 failed, 77 skipped. E2E CLI matrix re-verified on AMD (VitisAI/MIGraphX/DML/CPU).
tezheng
added a commit
that referenced
this pull request
Jul 13, 2026
Consolidated src/ changes spanning cc7a665..fa9986f + fix: - cc7a665 feat(session): unified-source EP refactor + P0 fixes + simplification pass - 5f101e0 fix: close out PR #1019 regressions and stale tests - 057e00b fix(sys/ep): truthful isolated-register errors, hardened rendering, stale-test alignment - (local) refactor(session): rename _ep_short_or_none -> ep_short_or_none; shrink session facade __all__ (drop SessionState, InferenceError — session-lifecycle types now sourced from session.session where tested). Kept in facade: EPDeviceSpec, UnknownListingPick, WinMLEPMonitorMismatch, default_device_for_ep, known_ep_short_names, lookup_device_spec — the test_ep_device_import_rule fitness function requires them there. Test updates and non-src changes (docs, pyproject.toml, CI workflow) land in follow-up commits.
tezheng
force-pushed
the
feat/ep-registration-and-monitoring
branch
from
July 13, 2026 02:31
057e00b to
e17db79
Compare
timenick
reviewed
Jul 14, 2026
DingmaomaoBJTU
left a comment
DingmaomaoBJTU
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed the EP-registration + monitoring changes. Most of it holds up well — the subprocess isolation in sys.py, the structured WinMLEPRegistrationFailed parsing, ep_path.py discovery/precedence, the QNN + op-metrics parsing, and the lazy transformers meta_path hook all look solid. A handful of things to flag before merge, the main one being a cache-key regression in from_onnx that can silently serve the wrong cached artifact. Details inline.
Collaborator
Author
E2E-verification run (
|
| # | Command | Verdict |
|---|---|---|
| 1 | uv run winml --version |
PASS |
| 2 | uv run winml --help |
PASS |
| 3 | uv run winml (no args) |
PASS |
| 4 | uv run winml sys --help |
PASS |
| 5 | uv run winml perf --help |
PASS |
| 6 | uv run winml sys |
PASS |
| 7 | uv run winml sys --format json |
PASS |
| 8 | uv run winml sys --format compact |
PASS |
| 9 | uv run winml sys --list-ep |
PASS |
| 10 | uv run winml sys --list-ep --format json |
PASS |
| 11 | uv run winml sys --list-device |
PASS |
| 12 | uv run winml sys --verbose |
PASS |
| 13 | uv run winml analyze -m microsoft/resnet-50 |
ENV — analyze -m is click.Path(exists=True); HF id rejected (pre-existing doc gap) |
| 14 | uv run winml analyze -m ./convnext/model_opt.onnx |
ENV — parquet rule fixtures not installed in this checkout |
| 15 | uv run winml analyze -m ./convnext/model_opt_qdq.onnx --device npu --ep qnn |
ENV — same missing parquet fixtures |
Group B — config + compile + build (10 PASS / 3 ENV)
| # | Command | Verdict |
|---|---|---|
| 16 | uv run winml config -m ./convnext/model_opt.onnx --device npu -o ./tmp/config-npu.json |
PASS |
| 17 | uv run winml config -m ./convnext/model_opt.onnx --device cpu --ep qnn -o ./tmp/config-cpu-qnn.json |
PASS |
| 18 | uv run winml config -m ./convnext/model_opt.onnx --device cpu --ep qnn@pypi -o ./tmp/config-reject.json |
PASS (source-tag pin cleanly rejected — intentional scaffolding) |
| 19 | uv run winml compile -m ./convnext/model_opt.onnx --ep qnn --device npu |
ENV — no Qualcomm NPU on host |
| 20 | uv run winml compile -m ./convnext/model_opt.onnx --ep openvino --device cpu |
PASS (compiled in 1.24s) |
| 21 | uv run winml compile --list --device npu |
PASS |
| 22 | uv run winml compile --list --device gpu |
PASS |
| 23 | uv run winml compile -m ./convnext/model_opt.onnx --ep qnn@pypi --device npu |
ENV — @pypi stripped correctly; no QNN NPU |
| 24 | uv run winml config -m ./convnext/model_opt.onnx --device cpu -o ./tmp/build.json |
PASS (prereq for build) |
| 25 | uv run winml build -c ./tmp/build.json -m microsoft/resnet-50 -o ./tmp/hf |
ENV — HF build path requires config.export (pre-existing assert) |
| 26 | uv run winml build -c ./tmp/build.json -m ./convnext/model_opt.onnx -o ./tmp/onnx |
PASS |
| 27 | uv run winml build -c ./tmp/build.json -m t5-small -o ./tmp/t5 |
PASS — composite (encoder + decoder) built end-to-end in 35.4s |
| 28 | uv run winml build -c ./tmp/build.json -m ./convnext/model_opt.onnx --ep qnn@pypi -o ./tmp/reject |
PASS (source-tag pin cleanly rejected) |
Group C — perf + composite + op-tracing + directory-EP (7 PASS / 5 ENV)
| # | Command | Verdict |
|---|---|---|
| 29 | uv run winml perf -m microsoft/resnet-50 --device auto --ep auto --monitor |
PASS (315 samp/s, 4-row monitor rendered) |
| 30 | uv run winml perf -m ./convnext/model_opt.onnx --device auto --ep auto |
PASS (102 samp/s) |
| 31 | uv run winml perf -m ./convnext/model_opt_qdq.onnx --device auto |
PASS (38 samp/s) |
| 32 | uv run winml perf -m microsoft/resnet-50 --ep qnn@pypi --device npu |
ENV — no Qualcomm NPU |
| 33 | uv run winml perf -m microsoft/resnet-50 --ep openvino@pypi --device cpu |
ENV — OpenVINO CPU plugin "Could not deserialize by device xml header" on this Intel |
| 34 | uv run winml perf -m ./convnext/model_opt_qdq.onnx --ep qnn --device npu --op-tracing basic |
ENV — no QNN NPU |
| 35 | uv run winml perf -m ./convnext/model_opt.onnx --ep openvino --op-tracing basic |
PASS (correctly refused with expected "OpenVINO wheels don't implement CSV-dump" message) |
| 36 | $env:WINMLCLI_EP_PATH='./x64/Release'; uv run winml perf -m microsoft/resnet-50 --device auto |
PASS (directory-EP resolves to openvino@directory, 409 samp/s) |
| 37 | uv run winml perf -m microsoft/resnet-50 --device gpu --monitor |
ENV — OpenVINO GPU "CompiledModel was not initialized" on Iris Xe |
| 38 | uv run winml perf -m microsoft/resnet-50 --iterations 100 --warmup 10 --batch-size 4 |
PASS (throughput knobs applied; static batch=1 fallback noted) |
| 39 | uv run winml perf -m t5-small --device cpu |
PASS — composite dispatch, 307 samp/s |
| 40 | uv run winml compile -m ./tmp/t5/encoder.onnx --ep openvino --device cpu |
ENV — doc says encoder.onnx, on-disk artifact is encoder_model.onnx (doc/fixture mismatch) |
ENV breakdown
| Category | Count | Rows |
|---|---|---|
| No Qualcomm NPU on host (need Snapdragon) | 4 | 19, 23, 32, 34 |
| OpenVINO on this Intel host (CPU deser / GPU init) | 2 | 33, 37 |
| Missing parquet rule fixtures | 2 | 14, 15 |
| Pre-existing doc/code gaps on this branch | 2 | 13 (analyze click.Path), 25 (HF build asserts config.export) |
| Doc/fixture path mismatch | 1 | 40 (encoder.onnx vs encoder_model.onnx) |
Not exercised
- Anything gated on
WINML_TEST_NPU=1on a Snapdragon host - CUDA / VitisAI / MIGraphX EP paths (SDKs not installed)
- Interactive TTY output (no PTY in agent runs)
Bottom line: no command that was expected to succeed on this hardware regressed. The 11 ENV failures are unchanged from prior runs on the same machine and none are caused by the merge or the post-merge cleanup.
tezheng
force-pushed
the
feat/ep-registration-and-monitoring
branch
from
July 15, 2026 03:19
100ec0c to
f1086f2
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
timenick
previously requested changes
Jul 16, 2026
tezheng
added a commit
that referenced
this pull request
Jul 16, 2026
Main advanced 9 commits (telemetry, recipes, --export-type #1104), leaving PR #1019 conflicting — which blocks the pull_request merge ref and therefore all CI. Re-merge to make the PR mergeable again. Conflicts, all in commands/build.py + its test (the genai-bundle path #1104 touched): - Keep origin's --export-type optimized restructure of _maybe_build_genai_bundle (want_optimized vs implicit-shortcut), but graft this branch's session.resolve_device(EPDeviceTarget) API into it and into the new _resolve_optimized_target helper (origin still called the removed sysinfo.resolve_device). - Keep this branch's defer-EP comment on build()'s ep-resolution (matches the actual resolve_device code; origin's resolve_check_device_ep comment did not). - test_build_genai_bundle.py: restore MagicMock import (origin's new _fake_ctx needs it), and repoint origin's new --export-type tests from the removed sysinfo.resolve_device / resolve_check_device_ep onto session.resolve_device. All 26 genai-bundle tests pass; build imports + ruff clean. Claude-Session: https://claude.ai/code/session_01CRVeL37EPZvGhMpQNj1hTF
timenick
dismissed
their stale review
July 17, 2026 02:05
Review state was submitted in error; retaining the inline comments as comments only.
… failures, universal op-tracing dispatch, lazy CLI startup Squashed single-commit form of PR #1019 (feat/ep-registration-and-monitoring), rebased onto current main. Highlights: - EP registration + monitoring: unified-source EP refactor, WinMLEPRegistry / EPDeviceTarget resolution (session.resolve_device), catalog-driven EP paths, structured EP failures, per-op QNN/OpenVINO/VitisAI monitoring. - Universal op-tracing dispatch relocated under session/monitor; lazy CLI startup (no eager transformers import on --help). - Integrated with mainline features carried through repeated main merges: WinMLManifest, dynamic export controls, genai-bundle build (winml build / perf), --export-type, pipeline_tag task fallback. Quality gates green: `ruff check src/ tests/` clean; `mypy -p winml.modelkit` = "Success: no issues found in 402 files" (all 108 type errors resolved with type-level fixes only). License headers present. Known-remaining: ~136 pre-existing unit-test failures (EP-refactor test drift, tracked separately) and two latent runtime bugs flagged for follow-up (genai_session._register_eps missing registry methods; perf --module ep-tuple). Claude-Session: https://claude.ai/code/session_01CRVeL37EPZvGhMpQNj1hTF
tezheng
force-pushed
the
feat/ep-registration-and-monitoring
branch
from
July 17, 2026 06:26
f28c9fc to
b9d13d8
Compare
xieofxie
reviewed
Jul 20, 2026
…typing Move to transformers>=5,<6 via a [tool.uv] override and neutralize the transitive standard onnxruntime (pulled by onnxruntime-qnn) with an always-false `sys_platform == "unobtainium"` marker so it never overwrites the onnxruntime-windowsml module on disk. Gate the Windows-only winml deps behind `sys_platform == "win32"` and relax onnxruntime-windowsml to a range (>=1.24.5,<1.25) to absorb the PEP 440 .post/date version mismatch. Fix all mypy errors surfaced by the transformers 4->5 API changes: - Cache.layers is now list[CacheLayerMixin | LinearAttentionCacheLayerMixin]; narrow to CacheLayerMixin (WinML caches build StaticLayer) via a _layer() helper / casts in kv_cache, decoder_wrapper, qwen, marian, mu2, t5, bart. - resolver: IMAGE_PROCESSOR_MAPPING_NAMES entries became dicts keyed by backend; extract the canonical processor name (real tf5 bug fix). - transformers_compat: pass the new q_offset arg to _ignore_causal_mask_sdpa (real tf5 bug fix); annotate monkey-patches. - sam/bart config-type narrowing; qwen_transformer_only ClassVar assign ignore.
Resolve conflicts preferring branch features, with three deliberate exceptions where main had the stronger change: - check_ops.py / op_utils.py: drop the dead `sys_info` writer parameter (main's cleanup; it was collected but never persisted per its own comment). - _perf_genai.py: adopt main's device-native EP preference (#1138) but return via the branch's `short_ep_name`/`EPNameOrAlias` idiom; cast str->EPName for the EP_SUPPORTED_DEVICES lookup. - check_patterns.py: keep the branch's single-source-of-truth re-export of EP checkers from check_ops (required by test_ep_checker_dedup) plus main's RuntimeCheckerRulesPrefilter body; include main's additive NvTensorRTRTX choice (a GPU EP, outside the branch's NPU carve-out). Also fix a pre-existing branch bug (fails identically on the pre-merge tip): the winml-genai perf path passed the raw EpAtSourceParamType `(ep, source)` tuple straight into GenaiPerfConfig.ep instead of the EP name. Unpack it like the ONNX path does. Restores test_explicit_ep_overrides_device / test_explicit_ep_without_device. mypy (405 files) and ruff (src/ tests/) are green. This merge introduces zero new test failures; the 20 remaining failures in analyze/build/perf CLI tests are pre-existing and identical on the pre-merge tip (stale EP/device-resolution mocks + one `tuple.device` bug in the build EP path).
…e key transformers>=5 dropped the lenient fallback that let AutoConfig.from_pretrained load a config lacking a `model_type` key (older Hub models such as prajjwal1/bert-tiny); it now raises "Unrecognized model". transformers 4 returned a base PretrainedConfig in that case. Add a shared `load_hf_config(auto_config, model_id, ...)` helper that tries AutoConfig.from_pretrained first and falls back to PretrainedConfig.from_pretrained only when the model omits model_type (re-raising the original AutoConfig error otherwise). No architecture-specific inference — the generic config carries the raw fields and downstream resolution keys off the task / model_class. The helper takes the caller's own AutoConfig reference as an argument so each call site's module-level `AutoConfig` stays monkeypatchable in tests. Routed the loader config-load paths (config.resolve_loader_config, resolution.resolve_composite_components / resolve_composite_load_task, hf.load_hf_model) through it. Restores the 8 TestExportPriority[task] tests (regressed by the transformers-5 upgrade); zero new failures vs the pre-fix tip.
## Summary Add an opt-in `static-split-to-slice` algebraic rewrite that replaces eligible ONNX `Split` nodes with equivalent input-form `Slice` nodes while preserving output tensor names. The rewrite only applies when the input shape, axis, and split boundaries are statically known. It supports explicit split sizes and equal splits, avoids malformed or dynamic cases, handles name collisions, preserves nested-subgraph captures, removes dead generated nodes and constants, and remains idempotent through the public optimization path. ## Why this is not a pattern rewrite The existing pattern framework is designed for fixed-topology replacements with compatible source and target schemas. Its match result exposes a single pattern output, and `PatternRewriter` constructs the replacement using that fixed output mapping. A `Split` node is variadic: one matched node may have any number of outputs, and the replacement must generate one `Slice` node plus four boundary initializers for each output. The replacement topology and constants therefore depend on the specific matched node rather than on a fixed target pattern. Representing this correctly as a pattern would first require broader framework support for variadic output mappings and match-parameterized replacement topology. For that reason, this transformation is implemented as a focused algebraic rewrite rather than adding model-specific behavior or forcing it into the current fixed-shape pattern abstraction. --------- Co-authored-by: hualxie <hualxie@microsoft.com>
…pes (#1148) Add verified CPU (fp32 + fp16) recipes for MIT/ast-finetuned-audioset-10-10-0.4593, the Audio Spectrogram Transformer fine-tuned on AudioSet for audio event classification. Effort L0 (recipe-only); Goal ceiling L2 reached — numeric parity with PyTorch confirmed (cosine = 1.0). L3 is CLI-BLOCKED because `winml eval` does not yet support the `audio-classification` task. ## Model metadata ### What the model does Audio Spectrogram Transformer (AST) fine-tuned on AudioSet-10-10 for multi-label audio event classification across 527 categories. Processes mel-spectrogram patches (1024 x 128) through a ViT-style encoder and outputs per-class logits. - Evidence: HuggingFace model card, `config.json` (`num_labels: 527`, `architectures: ["ASTForAudioClassification"]`) - Confidence: verified ### Primary user stories - A developer supplies a raw audio spectrogram to obtain class probabilities for 527 AudioSet event categories for audio tagging / content analysis. - Evidence: checkpoint task (`audio-classification`), model card - Confidence: verified ### Supported tasks | Task | Support surfaces | Evidence | Confidence | |---|---|---|---| | audio-classification | checkpoint, transformers, optimum-onnx, winml | Optimum vendor-native; `winml inspect` confirms default support | verified | ### Model architecture ```text ASTForAudioClassification ├── Embeddings │ ├── Patch embedding (Conv2d 16x16, stride 10x10 → 768-d) │ ├── CLS + distillation tokens │ └── Positional embedding (interpolated, 1024+2 positions) ├── Encoder x 12 │ ├── Self-attention (12 heads, 768-d) │ ├── Feed-forward (768 → 3072 → 768, GELU) │ └── Residual + LayerNorm └── Classification head (768 → 527 logits, mean of CLS + distill) ``` - Source/confidence: `ASTForAudioClassification` source and pinned checkpoint config (`verified`). ## Validation and support evidence ### Baseline - **origin/main**: `38767add6f91c7b10b6394fae3af6f437e02effd` - **winml version**: 0.2.0 - **Optimum probe**: VENDOR-ONLY — `audio-classification` and `feature-extraction` registered by Optimum vendor for `audio-spectrogram-transformer`; winml adds nothing. - **Build**: PASS (22.5s, 557 ONNX nodes, 100% tagged) - **Perf**: PASS (367ms avg, 2.72 samples/s, RAM +540MB) - **Eval**: CLI-BLOCKED (`audio-classification` not in `winml eval` supported tasks) - **Goal floor**: L1 (baseline builds and perfs) - **Starting recipe**: auto-config output identical to shipped recipe (no delta) ### Goal - **Effort**: L0 (recipe-only — Optimum vendor-native, auto-config correct, no code change) - **Goal ceiling**: L2 (numeric vs PyTorch — CPU-reachable; L3 CLI-BLOCKED) - **Outcome**: L0 (recipe + report) - **Success definition**: L0 = build PASS; L1 = perf runs without crash; L2 = cosine similarity >= 0.999 and max abs diff < 1e-3 vs PyTorch ### Outcome - **Highest reached**: L2 PASS on both (cpu, cpu, fp32) and (cpu, cpu, fp16) - **Coverage**: full (all target tuples passed or CLI-BLOCKED) - **Deferred tuples**: none - **Shipped recipes**: - `examples/recipes/MIT_ast-finetuned-audioset-10-10-0.4593/cpu/cpu/audio-classification_fp32_config.json` - `examples/recipes/MIT_ast-finetuned-audioset-10-10-0.4593/cpu/cpu/audio-classification_fp16_config.json` ### Per-EP/device/precision results | Tier | EP / Device | Precision | Verdict | Mean | p50 | Throughput | RAM delta | Task metric | |---|---|---|---|---|---|---|---|---| | L0 (build) | CPUExecutionProvider / cpu | fp32 | PASS | — | — | — | — | — | | L0 (build) | CPUExecutionProvider / cpu | fp16 | PASS | — | — | — | — | — | | L1 (perf) | CPUExecutionProvider / cpu | fp32 | PASS | 438.76 ms | 433.28 ms | 2.28 samples/s | +538.4 MB | — | | L1 (perf) | CPUExecutionProvider / cpu | fp16 | PASS | 409.30 ms | 411.60 ms | 2.44 samples/s | +539.8 MB | — | | L2 (numeric) | CPUExecutionProvider / cpu | fp32 | PASS | — | — | — | — | cosine=1.00000000, max_abs=1.03e-05 | | L2 (numeric) | CPUExecutionProvider / cpu | fp16 | PASS | — | — | — | — | cosine=1.00000000, max_abs=1.03e-05 | | L3 (eval) | CPUExecutionProvider / cpu | fp32 | CLI-BLOCKED | — | — | — | — | `audio-classification` not in eval task list | | L3 (eval) | CPUExecutionProvider / cpu | fp16 | CLI-BLOCKED | — | — | — | — | same as above | ### Delta Recipe is identical to `winml config` auto-config output — no field overrides. Filed for verified `cpu/cpu` coverage claim that the out-of-the-box auto-build does not formally establish. `examples/recipes/README.md` remains untouched. ### Analyze summary — component level and op level ANALYZE-UNAVAILABLE: `winml analyze` exited with error "No runtime rule parquet files were found" — rule parquet files are not present in this installation. This is an environment gap, not a model issue. Static rule analysis was not possible on this host. ### Reproduce commands ```powershell # L0 — build fp32 winml build -m MIT/ast-finetuned-audioset-10-10-0.4593 -c examples/recipes/MIT_ast-finetuned-audioset-10-10-0.4593/cpu/cpu/audio-classification_fp32_config.json -o $OUT/fp32 --ep cpu --device cpu --rebuild # L0 — build fp16 winml build -m MIT/ast-finetuned-audioset-10-10-0.4593 -c examples/recipes/MIT_ast-finetuned-audioset-10-10-0.4593/cpu/cpu/audio-classification_fp16_config.json -o $OUT/fp16 --ep cpu --device cpu --rebuild # L1 — perf fp32 winml perf -m $OUT/fp32/model.onnx --ep cpu --device cpu # L1 — perf fp16 winml perf -m $OUT/fp16/model.onnx --ep cpu --device cpu # L2 — numeric comparison vs PyTorch (Python, seed=42) python -c " import numpy as np, torch, onnxruntime as ort from transformers import ASTForAudioClassification model = ASTForAudioClassification.from_pretrained('MIT/ast-finetuned-audioset-10-10-0.4593'); model.eval() np.random.seed(42); x = np.random.uniform(-1, 1, (1, 1024, 128)).astype(np.float32) pt = model(torch.from_numpy(x)).logits.numpy() for tag, path in [('fp32','$OUT/fp32/model.onnx'), ('fp16','$OUT/fp16/model.onnx')]: ox = ort.InferenceSession(path, providers=['CPUExecutionProvider']).run(None, {'input_values': x})[0] cos = np.dot(pt.flatten(), ox.flatten()) / (np.linalg.norm(pt) * np.linalg.norm(ox)) print(f'{tag}: cosine={cos:.8f}, max_abs={np.max(np.abs(pt - ox)):.2e}') " ``` Co-authored-by: Yong Yue <yongyue@microsoft.com>
…t-qa (#1093) This PR adds `impira/layoutlm-document-qa` support to the LayoutLM export path in `winml-cli`, along with flat fp32 and w8a16 document-QA recipes. It ships at Outcome `L1` (`effort=L1-light`, `goal_ceiling=L2`) and the supplied tester verdicts reached `L2 PASS`. The main caveat is that raw `winml perf` with random dummy inputs crashes on this checkpoint because `token_type_ids` can contain `1` while `type_vocab_size=1`; the tester therefore used the valid-input fallback path for L1 measurements. Implementation summary: this adds `LayoutLMQAIOConfig` registration for (`layoutlm`, `question-answering`), a `ZeroTokenTypeLayoutLMTextInputGenerator` that forces all-zero `token_type_ids`, generic `bbox` ordering support in `src\winml\modelkit\onnx\io.py`, README indexing, fp32/w8a16 recipes, and unit coverage in `tests/unit/export/test_onnx_config_overrides.py` and `tests/unit/export/test_pytorch_export.py`. 1. **Recipe path(s)** - `examples/recipes/impira_layoutlm-document-qa/question-answering_fp32_config.json` - `examples/recipes/impira_layoutlm-document-qa/question-answering_w8a16_config.json` - Code paths in scope: `src/winml/modelkit/models/hf/layoutlm.py`, `src/winml/modelkit/models/hf/__init__.py`, `src/winml/modelkit/onnx/io.py`, `tests/unit/export/test_onnx_config_overrides.py`, `tests/unit/export/test_pytorch_export.py` 2. **README row** - Added row in `examples/recipes/README.md`: **yes**. 3. **Build output dir** - Refreshed isolated baseline against current `origin/main` `f102649b63e6bf013a180523d15e6cab75906dd9`. - Baseline output dir: `temp\impira_layoutlm-document-qa\baseline_offline` - fp32 output dir: `temp\impira_layoutlm-document-qa\fp32` - w8a16 output dir: `temp\impira_layoutlm-document-qa\w8a16` - Exact refreshed baseline command: `C:\Users\qiowu\source\repos\winml-cli\.venv\Scripts\winml.exe build -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\baseline_offline --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild` - Refreshed baseline result: `FAIL` with exact evidence `Error: Unrecognized configuration class <class 'transformers.models.layoutlm.configuration_layoutlm.LayoutLMConfig'> for this kind of AutoModel: AutoModelForNextSentencePrediction.` - Engineering delta claimed by this PR: current `origin/main` fails at the refreshed isolated baseline above, while the branch-specific LayoutLM QA registration/input-generation changes below produce successful fp32 and w8a16 builds. 4. **Build log** - fp32 build command: `./.venv/Scripts/winml.exe build -c examples/recipes/impira_layoutlm-document-qa/question-answering_fp32_config.json -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\fp32 --ep cpu --device cpu --rebuild` - Result: `✅ Build complete in 278.1s` - Artifact: `temp\impira_layoutlm-document-qa\fp32\model.onnx` - I/O: inputs `input_ids`, `bbox`, `attention_mask`, `token_type_ids` as INT32 with shapes `[1,512]`, `[1,512,4]`, `[1,512]`, `[1,512]`; outputs `start_logits`, `end_logits` as FLOAT `[1,512]` - Initializer histogram: `FLOAT=206, INT64=63` - Build quant: `null` - w8a16 build command: `./.venv/Scripts/winml.exe build -c examples/recipes/impira_layoutlm-document-qa/question-answering_w8a16_config.json -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\w8a16 --rebuild` - Result: `✅ Quantize 82.3s` / `Precision: uint8/uint16 (weight/activation)` / `✅ Build complete in 392.4s` - Artifact: `temp\impira_layoutlm-document-qa\w8a16\model.onnx` - I/O: same contract as fp32 - Initializer histogram: `FLOAT=411, INT32=196, INT64=63, UINT16=225, UINT8=196` - Build quant: `mode=static, weight_type=uint8, activation_type=uint16` 5. **Appended findings** - Lane A only (not staged in this PR): local skill-repo commit `26e1b7b1d83919d53c673c142854dbda16c6eb84` appended `layoutlm-001`, `layoutlm-002`, and `layoutlm-003` to `copilot-skills/dev_skill/adding-model-support/model_knowledge/layoutlm.json`. - Methodology finding `_meta-060` was also appended locally in the skill repo; it is declared here for traceability but intentionally excluded from this model PR diff. 6. **Optimum-coverage probe** - Vendor registrations: `['feature-extraction','fill-mask','text-classification','token-classification']` - After WinML changes: `['feature-extraction','fill-mask','text-classification','token-classification','question-answering']` - Added by WinML: `['question-answering']` - Verdict: branch state adds `question-answering` coverage for LayoutLM beyond the vendor baseline. 7. **Claimed (Effort, Goal, Outcome)** - `model_id`: `impira/layoutlm-document-qa` - `model_type`: `layoutlm` - `effort`: `L1-light` - `goal_ceiling`: `L2` - `outcome`: `L1` - `target_eps`: `cpu` - Planner facts carried into the implementation: `architectures=["LayoutLMForQuestionAnswering"]`, `model_type="layoutlm"`, `is_encoder_decoder=false`, `type_vocab_size=1`, `max_position_embeddings=514`, `max_2d_position_embeddings=1024`; the critical finding was that vendor dummy generation can emit `token_type_ids=1`, so this checkpoint required forcing all-zero `token_type_ids`. - The claimed delta is specifically from refreshed-current-main baseline `FAIL` (item 3) to branch-state recipe builds/L1/L2 `PASS` evidence (items 4, 8, 10, 11). 8. **Goal-ladder verdict table** Catalog gate summary: refreshed-current-main `baseline_build=FAIL`, `verdict=real-engineering`, `file_pr=true`. | Tier | Verdict | Evidence | |---|---|---| | L0 | PASS | The refreshed isolated baseline against current `origin/main` failed with `Error: Unrecognized configuration class <class 'transformers.models.layoutlm.configuration_layoutlm.LayoutLMConfig'> for this kind of AutoModel: AutoModelForNextSentencePrediction.` After the branch's LayoutLM QA export/input-generation changes, both recipe builds succeeded. fp32 produced `temp\impira_layoutlm-document-qa\fp32\model.onnx` with INT32 inputs (`input_ids`, `bbox`, `attention_mask`, `token_type_ids`) and FLOAT outputs (`start_logits`, `end_logits`), histogram `FLOAT=206, INT64=63`, and `build_quant=null`. w8a16 produced `temp\impira_layoutlm-document-qa\w8a16\model.onnx` with the same I/O contract, histogram `FLOAT=411, INT32=196, INT64=63, UINT16=225, UINT8=196`, and `build_quant={mode: static, weight_type: uint8, activation_type: uint16}`. | | L1 | PASS | Providers: `['AzureExecutionProvider', 'CPUExecutionProvider']`. Raw `winml perf` on both artifacts crashed with `Error: Benchmark failed: Inference failed | Context: error=[ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Gather node. Name:'/layoutlm/embeddings/token_type_embeddings/Gather' Status Message: indices element out of data bounds, idx=1 must be within the inclusive range [-1,0]'`. Tester then used the `_meta-017` valid-input fallback script; per-(EP,device) results are in item 10. | | L2 | PASS | fp32 vs PT: `start_logits` cosine `0.9999999999219783`, max_abs `0.0022459030151367188`, mean_abs `2.549774944782257e-05`; `end_logits` cosine `0.9999999999193658`, max_abs `0.0023813247680664062`, mean_abs `2.7379952371120453e-05`. w8a16 vs PT: `start_logits` cosine `0.9522225190737116`, max_abs `13.425172805786133`, mean_abs `2.4731171131134033`; `end_logits` cosine `0.9597506819201421`, max_abs `12.746326446533203`, mean_abs `2.3781232833862305`. | 9. **Methodology-evolution declaration** - `_meta-060` (`silent-failure`) was appended locally in the Lane A skill repo: explicit `winml build` overrides such as `--device cpu --ep cpu` can silently clear `config.quant`, yielding a successful build of the wrong precision for a quant recipe. - Reviewer/tester guidance was updated locally so quant recipes are validated from the preserved quant config and artifact, not from a superficially successful overridden build. 10. **Perf & eval data** Raw `winml perf` on both artifacts crashed on random dummy inputs with the Gather/token-type error quoted in item 8, so the tester used the valid-input fallback path for the L1 measurements below. | EP / Device | Precision | Verdict | Mean | p50 | Throughput | RAM Δ | Task metric | |---|---|---|---|---|---|---|---| | CPUExecutionProvider / cpu | fp32 | PASS | `5005.385395001213 ms` | `5076.764100005676 ms` | `0.19978481597015124 samples/s` | `+467062784 B total (+358121472 B session init)` | `N/A (goal ceiling L2)` | | CPUExecutionProvider / cpu | w8a16 | PASS | `5581.969174998085 ms` | `5745.492499998363 ms` | `0.1791482483420097 samples/s` | `+311275520 B total (+25878528 B session init)` | `N/A (goal ceiling L2)` | 11. **Component / op-level data** - Analyze command: `./.venv/Scripts/winml.exe analyze --model temp\impira_layoutlm-document-qa\fp32\model.onnx --ep all --output temp\impira_layoutlm-document-qa\analyze_fp32.json` - Artifact: `temp\impira_layoutlm-document-qa\analyze_fp32.json` - Model summary: `395` total operators across `16` unique operator types. - Per-EP classification: - `QNNExecutionProvider / NPU`: `16 supported / 0 partial / 0 unsupported / 0 unknown` (`395` total ops covered) - `OpenVINOExecutionProvider / NPU`: `15 supported / 1 partial / 0 unsupported / 0 unknown`; partial op: `OP/ai.onnx/Slice` - `VitisAIExecutionProvider / NPU`: `0 supported / 0 partial / 0 unsupported / 16 unknown` (no rule data) 12. **Reproducible commands** ```powershell C:\Users\qiowu\source\repos\winml-cli\.venv\Scripts\winml.exe build -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\baseline_offline --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild .\.venv\Scripts\winml.exe build -c examples\recipes\impira_layoutlm-document-qa\question-answering_fp32_config.json -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\fp32 --ep cpu --device cpu --rebuild python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py inspect --model temp\impira_layoutlm-document-qa\fp32\model.onnx --build-config temp\impira_layoutlm-document-qa\fp32\winml_build_config.json --label fp32 .\.venv\Scripts\winml.exe build -c examples\recipes\impira_layoutlm-document-qa\question-answering_w8a16_config.json -m impira/layoutlm-document-qa -o temp\impira_layoutlm-document-qa\w8a16 --rebuild python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py inspect --model temp\impira_layoutlm-document-qa\w8a16\model.onnx --build-config temp\impira_layoutlm-document-qa\w8a16\winml_build_config.json --label w8a16 .\.venv\Scripts\winml.exe perf --model temp\impira_layoutlm-document-qa\fp32\model.onnx --ep cpu --device cpu .\.venv\Scripts\winml.exe perf --model temp\impira_layoutlm-document-qa\w8a16\model.onnx --ep cpu --device cpu python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py perf --model temp\impira_layoutlm-document-qa\fp32\model.onnx --model-id impira/layoutlm-document-qa --label fp32 python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py perf --model temp\impira_layoutlm-document-qa\w8a16\model.onnx --model-id impira/layoutlm-document-qa --label w8a16 python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py compare --model temp\impira_layoutlm-document-qa\fp32\model.onnx --model-id impira/layoutlm-document-qa --label fp32 python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py compare --model temp\impira_layoutlm-document-qa\w8a16\model.onnx --model-id impira/layoutlm-document-qa --label w8a16 .\.venv\Scripts\winml.exe analyze --model temp\impira_layoutlm-document-qa\fp32\model.onnx --ep all --output temp\impira_layoutlm-document-qa\analyze_fp32.json python temp\impira_layoutlm-document-qa\layoutlm_qa_checks.py summarize-analyze --analyze-json temp\impira_layoutlm-document-qa\analyze_fp32.json ``` --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ing' into dingmaomaobjtu-fix-pr-1019-review # Conflicts: # tests/unit/session/test_winml_session.py
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR completes the EP registration, device-selection, session, and monitoring work while closing the CI and review regressions found during integration.
OrtEpDeviceclass before selecting an automatic EP/device pair.EPConfigimmutability, and canonicalcompile --listbehavior.Validation
uv run pytest tests\unit\ -q: 7451 passed, 77 skipped, 3 xfaileduv run ruff check src\ tests\: passeduv run mypy -p winml.modelkit: passed for 409 source filesuv run pytest tests\unit\sysinfo\ -q— 89 passed