eLLM can infer LLM on CPUs faster than on GPUs
-
Updated
Jun 11, 2026 - Rust
#
llm-infernece
Here are 5 public repositories matching this topic...
Efficient LLM inference on Slurm clusters.
inference speech-to-text vlm text-embedding multimodal audio-transcription llm vllm reward-model llm-infernece sglang llm-infrastructure
-
Updated
Jun 11, 2026 - Python
Layered prefill changes the scheduling axis from tokens to layers and removes redundant MoE weight reloads while keeping decode stall free. The result is lower TTFT, lower end-to-end latency, and lower energy per token without hurting TBT stability.
-
Updated
Mar 9, 2026 - Python
🤖🗜️⚡️ Local LLM server for Apple Silicon. 5.4× faster end-to-end on long contexts vs Ollama, 33% less RAM, INT3 support for Qwen3. OpenAI + Ollama drop-in. Built for repeated long-context workloads on memory-constrained Macs.
macos rust metal self-hosted quantization mlx inference-engine kv-cache apple-silicon openai-api int4 llm local-llm local-ai ollama qwen llm-infernece on-device-inference ollama-alternative llama-cpp-alternative
-
Updated
Jun 14, 2026 - Python
Improve this page
Add a description, image, and links to the llm-infernece topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the llm-infernece topic, visit your repo's landing page and select "manage topics."