Skip to content

feat: model_converter support for models from training_extensions#594

Merged
piotrgrubicki merged 30 commits into
masterfrom
mgumowsk/550-training-extensions-integration
Jun 15, 2026
Merged

feat: model_converter support for models from training_extensions#594
piotrgrubicki merged 30 commits into
masterfrom
mgumowsk/550-training-extensions-integration

Conversation

@mgumowsk

@mgumowsk mgumowsk commented May 27, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Here's a summary of the most important features and changes in PR #594:

Architecture Refactor — Converter Registry

  • Monolithic cli.py was split into a converters/ package with a BaseConverter, a CONVERTER_REGISTRY, and dedicated subclasses: TorchvisionConverter, TimmConverter, YoloConverter, and the new GetituneConverter
  • The standalone yolo/yolo.py CLI entry point was removed and absorbed into the new architecture

Getitune (OpenVINO Training Extensions) Support

  • Added GetituneConverter to convert models exported by the getitune (formerly OTX) framework, covering classification, multi-label classification, detection, segmentation, and rotated detection tasks

Multiplie Datasets support

  • DatasetRegistry class loads a datasets.json config file mapping dataset type identifiers (e.g. imagenet-1k, coco-detection, ade20k) to local filesystem paths, decoupling model configs from environment-specific paths
  • Model configs now declare a dataset_type field; INT8 quantization is skipped if absent
  • datasets/ package with readers for: ClassFolderReader (ImageNet-style), CocoImagesReader (COCO detection/segmentation), and Ade20kReader (ADE20K segmentation)
  • reader_for() factory selects the correct reader by dataset_type

Task-Aware Metrics Module

  • metrics/ package with a Metric abstract base and concrete implementations:
  • TopOneAccuracy — classification top-1
  • MultilabelMAP — multi-label mean AP via torchmetrics
  • CocoDetectionMAP — object detection mAP via pycocotools
  • SemSegMIoU — semantic segmentation mIoU via confusion matrix
  • metric_for() factory dispatches to the right metric based on (dataset_type, model_type, task)

Conversion Summary Reporting

  • --report [PATH] CLI flag generates a console table + Markdown report with per-model accuracy (Original, FP32, FP16, INT8) and a status (OK, ACCURACY DROP >5%, FAILED: conversion, SKIPPED, etc.)
  • Rolling upsert_result() updates the report after each model so partial runs are preserved

Timm model fixes

  • TimmConverter now reads preprocessing parameters (mean, std, input size) directly from timm.data.resolve_data_config, overriding hand-maintained config values to avoid silent accuracy regressions
  • Pinned huggingface_revision is now required for reproducibility

Config & Presets

  • examples/config.json replaced by a comprehensive presets/config.json covering torchvision, timm, YOLO11, and getitune models, plus a presets/datasets.json template
  • New dataset_type field added to model config schema
  • Extensive new unit tests covering the dataset readers, all metric implementations, the dataset registry, the converter classes, the reporting module, and CLI integration

Example conversion summary report

Conversion Summary Report

Model Full Name Model Type Model Library Original URL Original Accuracy FP32 Accuracy FP16 Accuracy INT8 Accuracy Status
EfficientNet-B0 Classification torchvision https://download.pytorch.org/models/efficientnet_b0_rwightman-3dd342df.pth 81.60% 81.60% 81.60% 81.80% OK
EfficientNet-B1 Classification torchvision https://download.pytorch.org/models/efficientnet_b1_rwightman-bac287d4.pth 85.20% 85.20% 85.00% 83.60% OK
EfficientNet-B2 Classification torchvision https://download.pytorch.org/models/efficientnet_b2_rwightman-c35c1473.pth 86.80% 86.80% 86.80% 86.00% OK
EfficientNet-B3 Classification torchvision https://download.pytorch.org/models/efficientnet_b3_rwightman-b3899882.pth 89.40% 89.40% 89.40% 89.60% OK
EfficientNet-B4 Classification torchvision https://download.pytorch.org/models/efficientnet_b4_rwightman-23ab8bcd.pth 89.00% 89.00% 89.00% 87.60% OK
EfficientNet-B5 Classification torchvision https://download.pytorch.org/models/efficientnet_b5_lukemelas-1a07897c.pth 90.20% 90.20% 90.20% 89.40% OK
EfficientNet-B6 Classification torchvision https://download.pytorch.org/models/efficientnet_b6_lukemelas-24a108a5.pth 90.40% 90.40% 90.40% 90.20% OK
EfficientNet-B7 Classification torchvision https://download.pytorch.org/models/efficientnet_b7_lukemelas-c5b4e57e.pth 91.40% 91.40% 91.40% 91.20% OK
EfficientNetV2-S Classification torchvision https://download.pytorch.org/models/efficientnet_v2_s-dd5fe13b.pth 90.60% 90.60% 90.60% 91.20% OK
EfficientNetV2-M Classification torchvision https://download.pytorch.org/models/efficientnet_v2_m-dc08266a.pth 90.20% 90.20% 90.20% 89.80% OK
EfficientNetV2-L Classification torchvision https://download.pytorch.org/models/efficientnet_v2_l-59c71312.pth 12.80% 12.80% 12.80% 0.00% ACCURACY DROP >5%
MobileNetV2 Classification torchvision https://download.pytorch.org/models/mobilenet_v2-b0353104.pth 74.00% 74.00% 74.00% 73.40% OK
MobileNetV3-Small Classification torchvision https://download.pytorch.org/models/mobilenet_v3_small-047dcff4.pth 45.00% 45.00% 45.20% 39.80% ACCURACY DROP >5%
MobileNetV3-Large Classification torchvision https://download.pytorch.org/models/mobilenet_v3_large-8738ca79.pth 75.60% 75.60% 75.60% 75.00% OK
ResNet-18 Classification torchvision https://download.pytorch.org/models/resnet18-f37072fd.pth 71.60% 71.60% 71.60% 71.80% OK
ResNet-34 Classification torchvision https://download.pytorch.org/models/resnet34-b627a593.pth 76.40% 76.40% 76.40% 76.20% OK
ResNet-50 Classification torchvision https://download.pytorch.org/models/resnet50-0676ba61.pth 79.20% 79.20% 79.20% 79.60% OK
ResNet-101 Classification torchvision https://download.pytorch.org/models/resnet101-63fe2227.pth 81.80% 81.80% 81.80% 81.40% OK
ResNet-152 Classification torchvision https://download.pytorch.org/models/resnet152-394f9c45.pth 82.20% 82.20% 82.20% 82.20% OK
Mask R-CNN ResNet-50 FPN MaskRCNN torchvision https://download.pytorch.org/models/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth N/A 0.01% 0.01% 0.01% OK
MobileNetV2-0.5 LAMB ImageNet-1k Classification timm https://huggingface.co/timm/mobilenetv2_050.lamb_in1k 71.60% 71.60% 71.40% 68.00% OK
MobileNetV2-1.0 RandAugment ImageNet-1k Classification timm https://huggingface.co/timm/mobilenetv2_100.ra_in1k 81.00% 81.00% 81.00% 79.00% OK
MobileNetV2-1.1 RandAugment ImageNet-1k Classification timm https://huggingface.co/timm/mobilenetv2_110d.ra_in1k 81.40% 81.40% 81.40% 75.60% ACCURACY DROP >5%
MobileNetV2-1.2 RandAugment ImageNet-1k Classification timm https://huggingface.co/timm/mobilenetv2_120d.ra_in1k 83.40% 83.40% 83.60% 83.00% OK
MobileNetV2-1.4 RandAugment ImageNet-1k Classification timm https://huggingface.co/timm/mobilenetv2_140.ra_in1k 81.80% 81.80% 81.80% 81.20% OK
EfficientNet-B0 RandAugment ImageNet-1k Classification timm https://huggingface.co/timm/efficientnet_b0.ra_in1k 83.80% 83.80% 83.80% 82.00% OK
EfficientNet-B0 RA4 3600 Epochs ImageNet-1k Classification timm https://huggingface.co/timm/efficientnet_b0.ra4_e3600_r224_in1k 85.20% 85.20% 85.20% 58.80% ACCURACY DROP >5%
EfficientNet-B1 Fine-tuned ImageNet-1k Classification timm https://huggingface.co/timm/efficientnet_b1.ft_in1k 84.20% 84.20% 84.20% 83.80% OK
EfficientNet-B1 Pruned ImageNet-1k Classification timm https://huggingface.co/timm/efficientnet_b1_pruned.in1k 84.80% 84.80% 84.80% 82.80% OK
EfficientNet-B1 RA4 3600 Epochs ImageNet-1k Classification timm https://huggingface.co/timm/efficientnet_b1.ra4_e3600_r240_in1k 87.80% 87.80% 87.80% 75.60% ACCURACY DROP >5%
ViT-Tiny Patch16 224 AugReg ImageNet-21k Classification timm https://huggingface.co/timm/vit_tiny_patch16_224.augreg_in21k 37.60% 37.60% 37.80% 37.60% OK
DINOv2-Small Patch14 518 LVD-142M Classification timm https://huggingface.co/timm/vit_small_patch14_dinov2.lvd142m 0.20% 0.20% 0.20% 0.20% OK
YOLO11n YOLO11 yolo N/A 36.85% N/A 37.01% 36.52% OK
YOLO11s YOLO11 yolo N/A 44.71% N/A 44.88% 43.95% OK
YOLO11m YOLO11 yolo N/A 48.22% N/A 48.65% 47.69% OK
YOLO11l YOLO11 yolo N/A 51.00% N/A 50.11% 49.90% OK
YOLO11x YOLO11 yolo N/A 52.09% N/A 52.42% 51.81% OK

Getitune models, preliminary results before fixes in training_extensions repository

Conversion Summary Report

Model Full Name Model Type Model Library Original URL Original Accuracy FP32 Accuracy FP16 Accuracy INT8 Accuracy Status
DINOv2 Classification Classification getitune N/A N/A 1.00% 1.00% 1.00% OK
EfficientNet-B0 Classification Classification getitune N/A N/A 0.00% 0.00% 0.00% OK
EfficientNet-V2 Classification Classification getitune N/A N/A 0.00% 0.00% 0.00% OK
MobileNet-V3 Large Classification Classification getitune N/A N/A 0.00% 0.00% 0.20% OK
EfficientNet-B3 Classification Classification getitune N/A N/A 0.00% 0.00% 0.00% OK
ViT-Tiny Classification Classification getitune N/A N/A 1.00% 1.00% 1.00% OK
EfficientNet-B0 Multi-Label Classification Classification getitune N/A N/A 0.23% 0.23% 0.17% OK
EfficientNet-V2 Multi-Label Classification Classification getitune N/A N/A N/A N/A N/A OK (no accuracy data)
DINOv2 Multi-Label Classification Classification getitune N/A N/A 0.20% 0.20% 0.27% OK
EfficientNet-B3 Multi-Label Classification Classification getitune N/A N/A 0.21% 0.21% 0.23% OK
MobileNet-V3-Large Multi-Label Classification Classification getitune N/A N/A 0.22% 0.22% 0.22% OK
ViT-Tiny Multi-Label Classification Classification getitune N/A N/A 0.24% 0.24% 0.24% OK
YOLOX-S YOLOX getitune N/A N/A 33.99% 33.99% 32.98% OK
YOLOX-L YOLOX getitune N/A N/A 43.77% 43.77% 42.62% OK
YOLOX-X YOLOX getitune N/A N/A 45.58% 45.58% 44.53% OK
SSD MobileNet-V2 SSD getitune N/A N/A 0.04% 0.04% 0.03% OK
ATSS MobileNet-V2 SSD getitune N/A N/A 0.00% 0.00% 0.00% OK
ATSS ResNet-50 FPN SSD getitune N/A N/A N/A N/A N/A FAILED: conversion
YOLOX-Tiny YOLOX getitune N/A N/A 0.00% 0.00% 0.00% OK
DEIM D-FINE-L SSD getitune N/A N/A 0.00% 0.00% 0.00% OK
DEIM D-FINE-M SSD getitune N/A N/A 0.00% 0.00% 0.00% OK
DEIM D-FINE-X SSD getitune N/A N/A 0.00% 0.00% 0.00% OK
DEIMv2-L SSD getitune N/A N/A 0.00% 0.00% 0.00% OK
DEIMv2-M SSD getitune N/A N/A 0.00% 0.00% 0.00% OK
DEIMv2-S SSD getitune N/A N/A 0.00% 0.00% 0.00% OK
D-FINE-X SSD getitune N/A N/A 0.00% 0.00% 0.00% OK
RF-DETR Large SSD getitune N/A N/A 0.00% 0.00% 0.00% OK
RF-DETR Medium SSD getitune N/A N/A 0.00% 0.00% 0.00% OK
RF-DETR Small SSD getitune N/A N/A 0.00% 0.00% 0.00% OK
RT-DETR ResNet-50 SSD getitune N/A N/A 0.00% 0.00% 0.00% OK

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@mgumowsk
mgumowsk requested a review from a team as a code owner May 27, 2026 12:30
@github-actions github-actions Bot added tests Related to tests python python related changes docs Related to documentation build Related to build scripts samples labels May 27, 2026
Comment thread model_converter/src/model_converter/converters/getitune.py Fixed
Comment thread model_converter/src/model_converter/converters/getitune.py Fixed
Comment thread model_converter/src/model_converter/converters/getitune.py
Comment thread model_converter/src/model_converter/templates/README-getitune-fp16.md Outdated
Comment thread model_converter/src/model_converter/templates/README-getitune-fp16.md Outdated
@mgumowsk mgumowsk changed the title WIP: feat: model_converter support for models from training_extensions feat: model_converter support for models from training_extensions Jun 15, 2026
@piotrgrubicki
piotrgrubicki added this pull request to the merge queue Jun 15, 2026
Merged via the queue into master with commit a703504 Jun 15, 2026
33 checks passed
@piotrgrubicki
piotrgrubicki deleted the mgumowsk/550-training-extensions-integration branch June 15, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Related to build scripts docs Related to documentation python python related changes samples tests Related to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants