inference-scheduler

Here are 2 public repositories matching this topic...

JohnScheuer / llm-inference-scheduler

Discrete-event simulator of LLM inference scheduling policies, comparing FCFS, continuous batching, priority, SLO-aware, and chunked prefill under memory pressure and latency SLOs.

simulator cpp latency scheduling performance-analysis kv-cache llm llm-serving continuous-batching inference-scheduler

Updated Jul 9, 2026
C++

gwonxhj / InferEdgeOrchestrator

Star

Lightweight edge inference runtime scheduler for priority/deadline-aware multi-task control, bounded queues, adaptive load shedding, ONNX/TensorRT-backed workers, and Jetson telemetry.

python telemetry jetson tensorrt priority-scheduling load-shedding bounded-queue edge-ai deadline-aware onnxruntime edge-inference tensorrt-inference jetson-orin-nano runtime-scheduler inference-scheduler overload-control multi-task-inference

Updated Jun 23, 2026
Python

Add this topic to your repo

To associate your repository with the inference-scheduler topic, visit your repo's landing page and select "manage topics."

Learn more