Add vision genai inference path for multi-file VLM evaluation#2488

Merged

xiaoyu-work merged 18 commits into

mainfrom

jiafa/add-vision-genai-evaluator

Jun 2, 2026

Commits on Jun 2, 2026

Add vision genai inference path for multi-file VLM evaluation

jiafatom
and
Copilot
committed
Fix task_type_components_map to apply all component overrides

jiafatom
and
Copilot
committed
Address review comments on vision genai inference

jiafatom
and
Copilot
committed
Fix EP mapping: skip CPUExecutionProvider for genai

jiafatom
and
Copilot
committed
Fix lint: remove unused import, unused loop var, use .values()

jiafatom
and
Copilot
committed
Fix genai provider: use device field instead of ORT EP names

jiafatom
and
Copilot
committed
Cap max_length to 128 for vision VQA generation

jiafatom
and
Copilot
committed
Increase max_length cap to 4096 for vision genai inference

jiafatom
and
Copilot
committed
Address all review comments and fix lint errors

jiafatom
and
Copilot
committed
Add system_prompt support for vision VQA evaluation

jiafatom
and
Copilot
committed
Add options_col support and extract leading number from predictions

jiafatom
and
Copilot
committed
Address review comments: extract helper, remove debug code, fix lint

jiafatom
and
Copilot
committed
Add opt-in number extraction for multiple-choice VQA tasks

jiafatom
and
Copilot
committed
Re-trigger CI: flaky test_mnb_to_qdq failure unrelated to PR changes
jiafatom
committed
Address Copilot review: fix vision detection for empty dict, add unit tests
jiafatom
committed
Address review: add JSON error handling, guard PIL import, fix file handle leak
jiafatom
committed
Fix formatting: use parenthesized context managers
jiafatom
committed
Fix formatting: collapse ImportError raise to single line
jiafatom
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add vision genai inference path for multi-file VLM evaluation#2488

Add vision genai inference path for multi-file VLM evaluation#2488
xiaoyu-work merged 18 commits into
mainfrom
jiafa/add-vision-genai-evaluator

Commits on Jun 2, 2026

Add vision genai inference path for multi-file VLM evaluation

Fix task_type_components_map to apply all component overrides

Address review comments on vision genai inference

Fix EP mapping: skip CPUExecutionProvider for genai

Fix lint: remove unused import, unused loop var, use .values()

Fix genai provider: use device field instead of ORT EP names

Cap max_length to 128 for vision VQA generation

Increase max_length cap to 4096 for vision genai inference

Address all review comments and fix lint errors

Add system_prompt support for vision VQA evaluation

Add options_col support and extract leading number from predictions

Address review comments: extract helper, remove debug code, fix lint

Add opt-in number extraction for multiple-choice VQA tasks

Re-trigger CI: flaky test_mnb_to_qdq failure unrelated to PR changes

Address Copilot review: fix vision detection for empty dict, add unit tests

Address review: add JSON error handling, guard PIL import, fix file handle leak

Fix formatting: use parenthesized context managers

Fix formatting: collapse ImportError raise to single line