Skip to content
BranchesTags

Repository files navigation

Model Gym 🏋

A fitness center for AI models — import, export, optimize, benchmark, and report LLM inference performance across engines and hardware.

Model Gym Cover License: Apache 2.0

A high-performance LLM inference benchmark and optimization framework for deploying LLMs on GPUs using TensorRT, ONNX Runtime, and a custom C++17 inference backend.

Python CUDA TensorFlow C++17 License

GUI User Interface

pages-build-deployment

GUI Interface

🧩 Pipeline Stages

Model Gym is a full pipeline for taking an AI model from raw weights to a production-ready inference report. Here's what each stage does:

Model Registry Inference Benchmarking Quantization ONNX

flowchart LR
    A["📥 Stage 0<br/><br/>Model Importer<br/>HF • PyTorch • GGUF<br/>SafeTensors • NGC"]
    B["📦 Stage 1<br/><br/>Export<br/>PyTorch → ONNX<br/>ONNX → TRT Engine"]
    C["🎛 Stage 2<br/><br/>Optimize<br/>FP16 • INT8 • FP8<br/>W4A16 (GPTQ)"]
    D["🚀️ Stage 3<br/><br/>Inference Engine<br/>CPU • ONNX Runtime<br/>TensorRT"]
    E["📋 Stage 4<br/><br/>Benchmark<br/>Latency • Throughput<br/>Memory • Cost"]
    F["📊 Stage 5<br/><br/>Dashboard<br/>HTML Report<br/>Charts • Tables"]
    A --> B --> C --> D --> E --> F
    style A fill: #0B1220, color: #FFFFFF, stroke: #38BDF8, stroke-width: 3px
    style B fill: #0B1220, color: #FFFFFF, stroke: #34D399, stroke-width: 3px
    style C fill: #1A2332, color: #FFFFFF, stroke: #F59E0B, stroke-width: 3px
    style D fill: #0F2E1F, color: #FFFFFF, stroke: #76B900, stroke-width: 4px
    style E fill: #1E1B4B, color: #FFFFFF, stroke: #8B5CF6, stroke-width: 3px
    style F fill: #3D2C00, color: #FFFFFF, stroke: #FFD700, stroke-width: 3px
Loading

Stage Components

Stage Path GPU Required Details
0 — Import src/importer/ No README — HF, PyTorch, SafeTensors, GGUF, Ollama, NGC
1 — Export src/export/ TRT only README — OnnxExporter (CPU), TensorRTBuilder (GPU)
2 — Optimize src/optimize/ No README — FP16, INT8, FP8 sim, W4A16 GPTQ
3 — Inference inference/ TRT only README — CPU backend runnable; ONNX/TRT need GPU
4 — Benchmark benchmark/ No README — C++ fixtures + Python harness
5 — Dashboard src/dashboard/ No README — HTML report, 4 charts, Markdown table

✨ Quickstart

⚙️ Setup

Clone and create python venv

    git clone https://github.com/HiteshSahu/Model-Gym && cd Model-Gym
    
    # Setup and install
    python3 -m venv .venv && source .venv/bin/activate

Install Dependencies for all steps

pip install -e ".[all]"               # everything

Or Install Dependencies for individual stage

pip install -e ".[importer]"          # Stage 0 — HuggingFace, safetensors, gguf
pip install -e ".[export]"            # Stage 1 — torch, onnx, tensorrt (optional)
pip install -e ".[optimize]"          # Stage 2 — bitsandbytes, auto-gptq
pip install -e ".[benchmark]"         # Stage 4 — onnxruntime, psutil, pynvml
pip install -e ".[dashboard]"         # Stage 5 — matplotlib, rich
pip install -e ".[dev]"               # + pytest, ruff, mypy

▶️ Run Python pipeline (no GPU needed)

Detailed Run Instruction can be found in Build Pipeline README

There are Three Build Paths you can take based on:

  • if you have no GPU
  • have CUDA GPU
  • have Full GPU with TensorRT Engine

Stage 0 — IMPORTER

Load a model from any source:

  • HuggingFace Hub (any task — classification, generation, etc.)
  • Local .pt / .pth PyTorch checkpoints
  • SafeTensors files (single or sharded)
  • GGUF files (llama.cpp / Ollama format — dequantizes on load)
  • Ollama local manifest (no re-download)
  • NVIDIA NGC catalog (model / engine / trtllm assets)

Model Import

    # Stage 0 — import
    pip install -e ".[importer]"
    python -m src.importer.huggingface --model bert-base-uncased --task text-classification

Model will be imported in models directory

Stage 1 — EXPORT

Convert the model to a deployable format:

Export

  • PyTorch → ONNX with dynamic batch + sequence axes, shape inference, and validation
  • ONNX → TensorRT engine in FP32, FP16, or INT8 (GPU required)
    # Stage 1 — export to ONNX
    pip install -e ".[export]"
    python -m src.export.onnx_exporter \
        --model bert-base-uncased --output models/bert.onnx

ONXX Model will be exported in models directory

Stage 2 — OPTIMIZE

Compress the model to run faster and use less memory:

Quantization

INT8

  • INT8 dynamic — ~2.4× smaller, no calibration data needed
  • INT8 static — more accurate, requires a calibration dataset
  • FP8 E4M3/E5M2 — simulates H100 FP8 quantization on any hardware

FP16

  • FP16 — 2× smaller, near-zero accuracy loss
  • W4A16 GPTQ — 4-bit weights with FP16 activations, for memory-bandwidth-bound LLMs (LLaMA, Mistral)

🗜 Quantization Schemes

Optimized model are generated in models directory

Scheme Bits Description
int8_symmetric 8 Per-tensor or per-channel scale, zero-point = 0
int8_asymmetric 8 Per-tensor scale + zero-point (range [0, 255])
fp8_e4m3 8 IEEE-like FP8 for H100 / Ada Lovelace, max = 448
fp8_e5m2 8 Wider dynamic range FP8 variant
fp16 16 Half-precision — .half() conversion, minimal accuracy loss
w4a16 4W/16A GPTQ-style 4-bit weights, FP16 activations 
    # Stage 2 — optimize
    pip install -e ".[optimize]"
    
    # 16 bit
    python -m src.optimize.fp16 \
        --input models/bert.onnx --output models/bert_fp16.onnx
        
    # 8 bit    
    python -m src.optimize.int8 \
        --input models/bert.onnx --output models/bert_int8.onnx --mode dynamic

Optimized Model will be stored in models directory

Stage 3 — INFERENCE

A custom C++17 inference binary with three backends:

CUDA GPU TensorRT Multi Model

  • CPU — runs anywhere, no dependencies, LayerNorm → MatMul → Softmax pipeline
  • ONNX Runtime — uses ORT C++ API with optional CUDA execution provider
  • TensorRT — deserializes .engine file, runs enqueueV3 async on GPU

Full cpp build instructions, CMake options, sanitizers, and tests → inference/README.md

C++ inference engine (no external deps)

# Build and run directly with clang++
clang++ -std=c++17 -O2 -I inference/include \
    inference/src/core/tensor.cpp \
    inference/src/backends/cpu_backend.cpp \
    inference/src/main.cpp -o model_gym

./model_gym --hidden 768 --output 1000 --batch 4 --seq-len 128

Result

Reports mean, p50, p95, p99 latency and throughput QPS.

=== Model Gym — cpu ===
  Samples   : 50
  Mean      : 84.2 ms
  P50       : 83.5 ms
  P95       : 86.1 ms
  P99       : 87.4 ms
  Throughput: 47.9 qps

Stage 4 — BENCHMARK

Two complementary benchmark tiers: Evaluation Benchmarking

  • C++ micro-benchmarks (Google Benchmark) — kernel-level: tensor alloc, copy bandwidth, INT8/FP8 quantization speed
  • Python harness — end-to-end: sweeps backends × batch sizes, measures latency percentiles, throughput, peak GPU/CPU memory, and cost per 1M tokens
    # Stage 4 — benchmark (CPU backend via C++ CLI)
    pip install -e ".[benchmark]"
    python -m src.benchmark.harness \
        --backends cpu --batch-sizes 1 8 32 --out results/bench.json

Benchmark results will be stored in results folder

eg. BERT

Backend   Batch   p50 ms   p99 ms       QPS   GPU MB   CPU MB   $/1M tok  
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
cpu           1     6.27     6.27   20427.6        0       35        N/A  
cpu           8    49.98    49.98   20487.7        0       35        N/A  
cpu          32   202.43   202.43   20233.9        0       35        N/A  
onnx          1    19.27    23.48    6644.0        0      645        N/A  
onnx          8   140.61   205.06    7282.6        0      684        N/A  
onnx         32   717.30   835.62    5710.3        0     1001        N/A

Stage 5 — DASHBOARD

Generates a visual report from benchmark results:

Observability

  • HTML report — standalone file with embedded SVG charts, no server needed
  • Markdown table — for GitHub READMEs or PRs
  • Console output — rich-formatted table in the terminal
  • 4 matplotlib charts — latency vs batch size, throughput comparison, memory footprint, cost heatmap
    # Stage 5 — dashboard
    pip install -e ".[dashboard]"
    python -m src.dashboard.report \
        --results results/bench.json --out dashboard/output

Dashboard HTML Report will be generated in dashboard/output

Dashboard Example BERT

⚡ Tech Stack

Domain What this project demonstrates
Modern C++17 Templates, RAII, smart pointers, move semantics, NVI pattern, constexpr, if constexpr
Inference Frameworks TensorRT engine build + runtime, ONNX Runtime C++ API, CPU reference backend
AI Quantization INT8 symmetric/asymmetric, FP8 E4M3/E5M2, W4A16 GPTQ (4-bit weights, FP16 activations)
Python Tooling HuggingFace → ONNX export, TensorRT Python API, post-training quantization, calibration
Model Formats HuggingFace Hub, SafeTensors shards, GGUF dequant + remap, Ollama manifests, NVIDIA NGC
Build System CMake 3.20+, FetchContent (GoogleTest, Google Benchmark), Find modules for TRT + ORT
Software Quality 21 GoogleTests, Google Benchmark fixtures, clang-tidy, AddressSanitizer, UBSanitizer

📂 Project Layout

Model-Gym/
│
├── src/                              Python pipeline stages
│   ├── importer/                     Stage 0 — load from any source
│   │   ├── huggingface.py            HuggingFace Hub / local HF directory
│   │   ├── local_pytorch.py          .pt / .pth checkpoint files
│   │   ├── safetensors.py            .safetensors (single or sharded)
│   │   ├── gguf.py                   GGUF — dequantize + remap weight names
│   │   ├── ollama.py                 Ollama manifest → GGUF blob
│   │   └── nvidia_ngc.py             NGC catalog (model / engine / trtllm)
│   │
│   ├── export/                       Stage 1 — PyTorch → ONNX / TRT engine
│   │   ├── onnx_exporter.py          torch.onnx.export + shape inference + validation
│   │   └── tensorrt_builder.py       ONNX → .engine (FP32 / FP16 / INT8)
│   │
│   ├── optimize/                     Stage 2 — quantization
│   │   ├── fp16.py                   FP16 conversion
│   │   ├── int8.py                   INT8 symmetric / asymmetric + calibration
│   │   ├── fp8.py                    FP8 E4M3 / E5M2 clamp + scale
│   │   └── w4a16.py                  W4A16 GPTQ block-wise quantization
│   │
│   ├── benchmark/                    Stage 4 — Python harness
│   │   ├── harness.py                BenchmarkHarness — sweeps backends × batch sizes
│   │   └── metrics.py                compute() — latency / throughput / memory / cost
│   │
│   └── dashboard/                    Stage 5 — report generation
│       ├── plot.py                   4 matplotlib charts (SVG + PNG)
│       └── report.py                 HTML report + Markdown table + console output
│
├── inference/                        C++ inference engine (Stage 3)
│   ├── include/
│   │   ├── core/                     tensor.hpp, model_runner.hpp (NVI base)
│   │   ├── backends/                 cpu_backend.hpp, onnx_backend.hpp, tensorrt_backend.hpp
│   │   └── utils/                    timer.hpp, logger.hpp, memory_pool.hpp
│   ├── src/
│   ├── tests/                        21 GoogleTest cases
│   └── CMakeLists.txt
│
├── benchmark/                        Benchmark layer
│   ├── cpp/                          Google Benchmark C++ fixtures (Stage 4)
│   │   ├── bench_inference.cpp       BM_TensorAllocate/Copy/CpuInference/QuantizeINT8/FP8
│   │   └── CMakeLists.txt            FetchContent → google/benchmark v1.8.4
│   └── README.md                     Two-tier benchmark guide
│
├── cmake/                            CMake helpers
│   ├── CompilerWarnings.cmake
│   ├── Sanitizers.cmake
│   ├── FindTensorRT.cmake
│   └── FindONNXRuntime.cmake
│
├── CMakeLists.txt                    Root build — options for TRT/ORT/sanitizers
└── pyproject.toml                    Python package with optional dep groups

© 2026 Hitesh Kumar Sahu · Licensed under Apache 2.0

About

GPU Pipeline for Model Import + Inference + Benchmark + Dashboard + Export

hiteshsahu.github.io/Model-Gym/

Topics

cmakecpp17ggufinference-enginengcollamaonxxquantizationtensorrt-inference

Resources

Readme
License
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages