SnapInfer C++ SDK

SnapInfer is a high-performance, lightweight Machine Learning inference SDK written in C++17. It wraps the ONNX Runtime (C++ API), provides a clean C API shared library (.dll on Windows), and supports multiple execution providers (CPU, CUDA GPU, DirectML), offline benchmarking, and model quantization (INT8).

Designed for edge deployments and seamless embedding into applications, SnapInfer enables developers to run deep learning models with minimum overhead and high flexibility.

---

Key Features

• C++17 Engine Wrapper: High-level, safe wrapper for ONNX Runtime sessions, handling tensor management, memory allocation, and session configurations.
• Stable C API Shared Library: Exposes clean C functions suitable for integration with other languages (Python, C#, Rust, Go, etc.) without exposing C++ runtimes or namespaces.
• Automated Scaffolding & Dependencies: Resolves Microsoft ONNX Runtime binaries dynamically during the CMake configuration phase via NuGet (supporting Windows x86/x64).
• Execution Providers: Easily switch between CPU, NVIDIA CUDA GPU, and Microsoft DirectML execution providers.
• Quantization & Optimization: Includes utilities to export, load, and benchmark quantized INT8 model variants for accelerated inference on resource-constrained hardware.
• Latency Benchmarking: Integrated profiler calculating warm-up times and high-precision latency percentiles ($p_{50}$, $p_{90}$, $p_{99}$).
• Comprehensive Unit Tests: Built with Google Test, covering bounds checking, error handling, shape mismatch, and concurrent multi-session safety.

---

Project Structure

SnapInfer/
├── CMakeLists.txt              # CMake configuration downloading ONNX Runtime & GTest
├── LICENSE                     # MIT License
├── README.md                   # System documentation
├── include/
│   ├── SnapInfer.h            # C++ Inference Engine & Benchmarker header
│   └── snapinfer_c.h          # C API Shared Library header
├── src/
│   ├── SnapInfer.cpp          # C++ Engine & Session implementation
│   ├── Benchmark.cpp          # Latency profiler implementation
│   ├── snapinfer_c.cpp        # C API wrapper implementation
│   └── main.cpp               # CLI demo driver (benchmarks and EP configuration)
├── samples/
│   ├── c_demo.c               # Sample program calling the C API Shared Library (.dll)
│   └── classify_image.cpp     # MobileNetV2 Image Classification Demo
├── tests/
│   ├── CMakeLists.txt         # Test sub-project configuration
│   └── test_snapinfer.cpp     # Unit tests verifying edge-cases & concurrency
└── tools/
    ├── generate_toy_model.py  # Script generating toy model (toy.onnx) for local validation
    └── quantize_model.py      # Python script to convert toy.onnx to toy_int8.onnx

---

Build Requirements

• CMake: version 3.25 or higher
• C++ Compiler: GCC (MinGW-w64) supporting C++17
• Python 3.x: (Optional, for exporting test models)
  • Install dependencies: pip install onnx onnxruntime

---

Building the SDK

To configure and compile the SDK along with all samples and tests, run:

# Create a build directory
mkdir build
cd build

# Configure the project with CMake
cmake -G "MinGW Makefiles" ..

# Build all targets
cmake --build .

After building, you will find the binaries inside build/bin/:

• snapinfer.dll (The shared library wrapper)
• snapinfer_demo.exe (CLI inference & benchmarking tool)
• snapinfer_c_demo.exe (C API DLL sample test client)
• snapinfer_classify.exe (MobileNetV2 classification sample)
• snapinfer_tests.exe (Google Test runner)

---

API Usage Guide

1. C++ Engine API

#include "SnapInfer.h"
#include <iostream>

int main() {
    // Instantiate engine with model path
    SnapInfer::Engine engine(L"toy.onnx");
    
    // Initialize session targeting CPU
    if (!engine.initialize(SnapInfer::DeviceType::CPU)) {
        std::cerr << "Initialization failed" << std::endl;
        return 1;
    }

    // Retrieve input details
    std::cout << "Input Tensor Shape: [" << engine.getInputShape()[0] 
              << ", " << engine.getInputShape()[1] << "]" << std::endl;

    // Prepare inputs & execute inference
    std::vector<float> inputData = {1.0f, 2.0f, 3.0f};
    std::vector<float> outputData = engine.run(inputData);

    std::cout << "Inference completed. Output dimension: " << outputData.size() << std::endl;
    return 0;
}

2. C API wrapper (DLL Integration)

#include "snapinfer_c.h"
#include <stdio.h>

int main() {
    // Create session (UTF-16 string for model path)
    snapinfer_session_t* session = snapinfer_create_session(L"toy.onnx");
    if (!session) return 1;

    size_t input_size = snapinfer_get_input_size(session);
    size_t output_size = snapinfer_get_output_size(session);

    float input_data[] = {1.0f, 2.0f, 3.0f};
    float* output_data = (float*)malloc(output_size * sizeof(float));

    // Execute run
    if (snapinfer_run(session, input_data, input_size, output_data, output_size)) {
        printf("Output[0] = %f\n", output_data[0]);
    }

    // Clean up
    free(output_data);
    snapinfer_destroy_session(session);
    return 0;
}

---

Benchmarks & Quantization

You can evaluate model latencies and compare full-precision versus quantized variants:

Exporting INT8 Models

Run the quantization helper:

python tools/quantize_model.py

This generates a toy_int8.onnx optimized model.

Running Latency Profile Benchmarks

Execute the demo executable with the --benchmark flag:

.\build\bin\snapinfer_demo.exe --model toy.onnx --benchmark
.\build\bin\snapinfer_demo.exe --model toy_int8.onnx --benchmark

Expected Output:

[Demo] Starting SnapInfer Driver...
  Model: toy.onnx
  Device: CPU
...
[Benchmark] Profiling: 100 iterations...

--- Benchmark Latency Results (ms) ---
  Min Latency: 0 ms
  Max Latency: 0.571 ms
  Avg Latency: 0.01621 ms
  p50 Latency: 0 ms
  p90 Latency: 0 ms
  p99 Latency: 0.535 ms
--------------------------------------

---

Unit Testing

To verify structural, thread safety, and edge-case guarantees, run the compiled test executable:

.\build\bin\snapinfer_tests.exe

Unit tests verify:

1. Handling of invalid/non-existent model file paths.
2. Extraction and validation of model input/output shapes and metadata.
3. Logical numerical correctness of forward propagation.
4. Input bounds and shape mismatches error handling.
5. Multithreading safety with concurrent, independent engine sessions executing in parallel.

---

License

This project is licensed under the MIT License.