llmctl

Running Large Language Models (LLMs) locally involves several separate parts: a GPU-accelerated inference server, model files fetched from HuggingFace, and whatever Docker Compose stack sits around them. llmctl is a single command-line tool that manages all three, from one binary.

No cloud dependencies. No telemetry. No API keys.

Requirements

• Linux (NVIDIA, AMD, or Intel GPU; CPU-only is also supported)
• Docker with GPU passthrough configured for your hardware
• curl for HuggingFace API queries and model downloads
• Rust toolchain to build from source

Install

From crates.io:

cargo install local-llmctl

From source:

cargo install --path .

The binary is installed as llmctl.

Quick Start

# 1. Initialize – creates config and a compose/ directory
llmctl init /path/to/your/project

# 2. Download a model
llmctl model download unsloth/Qwen3.6-35B-A3B-GGUF

# 3. Create and start an inference instance
llmctl instance add main
llmctl instance up main

# 4. Start compose services (if any are defined)
llmctl up

The llama.cpp instance is now serving an OpenAI-compatible API at http://localhost:<port>/v1.

Overview

llmctl manages two categories of containers:

1. Instances: standalone llama.cpp Docker containers with automatic GPU passthrough, each serving a single GGUF model on a dedicated port.
2. Compose services: any Docker Compose stacks defined as YAML files in the compose/ directory.

Both categories share a Docker bridge network (ai-network), thus allowing them to communicate by container name.

Project Layout

your-project/
  compose/            # Place .yaml/.yml files here to define services
    webui.yaml
    search.yaml
  ...

llmctl discovers services by scanning compose/ for YAML files at runtime; the filename stem (the filename without its extension) becomes the service name. Thus, adding or removing a service is a matter of adding or removing a file; no source code changes are required.

Commands

Compose Services

llmctl status              # Show running compose services and instances
llmctl up                  # Start all discovered compose services
llmctl up webui search     # Start specific services only
llmctl down                # Stop all compose services
llmctl down search         # Stop a specific service
llmctl logs                # Tail logs for all compose services
llmctl logs webui          # Tail logs for one service
llmctl logs main           # Tail logs for an instance by name

If the compose/ directory is empty, compose commands report that no services are defined.

Instances

Each instance corresponds to a Docker container running llama.cpp in server mode with a chosen GGUF model.

llmctl instance list                         # List with port, status, health, model
llmctl instance list --json                  # JSON output for scripting
llmctl instance add <name>                   # Add interactively
llmctl instance add <name> --model <file> --port 8081  # Add non-interactively
llmctl instance show <name>                  # Show full configuration
llmctl instance show <name> --json           # JSON output
llmctl instance up <name>                    # Start the container
llmctl instance down <name>                  # Stop and remove the container
llmctl instance reload <name>                # Restart with the same model
llmctl instance reload <name> --model <file> # Swap to a different model
llmctl instance set <name> --ctx_size 65536  # Update configuration fields
llmctl instance remove <name>                # Remove from config (stops if running)

Non-interactive creation

All settings can be passed as flags, which is useful for scripting:

llmctl instance add main \
  --model Qwen3-14B-Q8_0.gguf \
  --port 8081 \
  --ctx_size 32768 \
  --n_gpu_layers auto \
  --cache_type_k q8_0 \
  --cache_type_v q8_0 \
  --flash_attn on

llmctl instance add embed \
  --model nomic-embed-text-v1.5.Q8_0.gguf \
  --port 8086 \
  --ctx_size 2048 \
  --embeddings

Default port assignment

When you add an instance (interactively or with --port omitted), llmctl suggests a port. The suggested port comes from the [port_defaults] table in config.toml (created during llmctl init):

[port_defaults]
main = 8081
embed = 8082
coding = 8083
chat = 8084

If the instance name matches a key in the table, that port is suggested. Otherwise, llmctl scans upward from 8082 to find the next free port. The same fallback applies if the suggested port is already taken by another instance.

You can edit this table to suit your environment: add entries for names you use often, remove ones you don't, or change the port numbers.

Models

llmctl model download                       # Interactive: enter repo, pick file
llmctl model download <repo>                # Pick a file from the repo listing
llmctl model download <repo> <file>         # Direct download
llmctl model download <repo> --type embed   # Force type classification
llmctl model search <query>                 # Search HuggingFace for GGUF repos
llmctl model list                           # List with sizes and used/unused markers
llmctl model info <name>                    # Size, quantization type, referencing instances
llmctl model unused                         # Models not referenced by any instance
llmctl model remove <name>                  # Delete (with safety check if in use)
llmctl model remove <name> --force          # Skip safety check
llmctl model clean                          # Remove incomplete .partial downloads
llmctl model flux                           # Download Flux.1-dev image generation models

By default, models are stored at ~/.local/share/llmctl/models/, following the XDG Base Directory Specification. This location can be overridden via models_dir in the config file.

Shell Completions

llmctl generates dynamic completions that include instance names and discovered service names:

# Bash
llmctl completions bash > ~/.local/share/bash-completion/completions/llmctl

# Zsh
llmctl completions zsh > ~/.local/share/zsh/site-functions/_llmctl

# Fish
llmctl completions fish > ~/.config/fish/completions/llmctl.fish

Configuration

The configuration file resides at ~/.config/llmctl/config.toml:

project_dir = "/home/user/my-ai-stack"
# models_dir = "/mnt/fast-ssd/models"  # optional override
# network = "ai-network"               # optional, defaults to "ai-network"

[instances.main]
model = "Qwen3-14B-Q8_0.gguf"
port = 8081
ctx_size = 32768
n_gpu_layers = "auto"
cache_type_k = "q8_0"
cache_type_v = "q8_0"
flash_attn = "on"

[instances.embed]
model = "nomic-embed-text-v1.5.Q8_0.gguf"
port = 8086
ctx_size = 2048
n_gpu_layers = "auto"
embeddings = true
cache_type_k = "q8_0"
cache_type_v = "q8_0"
flash_attn = "on"

Top-level fields

Field	Required	Default	Description
project_dir	yes	n/a	Directory containing compose/
models_dir	no	~/.local/share/llmctl/models/	Override for model storage location
network	no	"ai-network"	Docker network shared by instances and compose services
instances.*	no	n/a	Named instance configurations

Instance fields

Field	Default	Description
model	n/a	GGUF filename (must exist in the models directory)
port	auto	Host port to expose
ctx_size	32768	Context window size in tokens
n_gpu_layers	"auto"	Number of layers offloaded to GPU ("auto" or an integer)
embeddings	false	Enable embeddings mode
cache_type_k	"q8_0"	KV cache quantization for keys
cache_type_v	"q8_0"	KV cache quantization for values
flash_attn	"on"	Flash attention ("on" or "off")
n_parallel	n/a	Number of parallel request slots
threads	n/a	CPU thread count

Compose Services

Defining a service

To add a service, create a YAML file in compose/:

# compose/webui.yaml
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "8080:8080"
    networks:
      - ai-network

networks:
  ai-network:
    external: true

The service is immediately available via llmctl up webui, llmctl down webui, and llmctl logs webui. Removing a service is just as direct: delete or rename the YAML file.

Networking

All compose services and llmctl instances share a Docker bridge network – by default named ai-network, configurable via the network field in config.toml. Thus, compose services can reach instances by container name (for example, http://llmctl-main:8080), and instances can reach compose services by their Docker Compose service name. The network is created automatically when the first instance starts.

Environment variables

To make this networking usable from within a compose file, llmctl passes two environment variables when invoking docker compose:

• LLMCTL_MODELS_DIR: the resolved models directory path
• LLMCTL_NETWORK: the configured network name

Use them in your YAML files:

services:
  my-service:
    volumes:
      - ${LLMCTL_MODELS_DIR}:/models:ro
    networks:
      - ${LLMCTL_NETWORK}

networks:
  ${LLMCTL_NETWORK}:
    external: true

GPU Support

The GPU backend is auto-detected at instance startup:

Backend	Detection	Docker Image
CUDA (NVIDIA)	/dev/nvidia0 exists or nvidia-smi is on PATH	ghcr.io/ggml-org/llama.cpp:server-cuda
Vulkan (AMD/Intel)	/dev/dri/renderD128 exists	ghcr.io/ggml-org/llama.cpp:server-vulkan
CPU	Fallback	ghcr.io/ggml-org/llama.cpp:server

GPU devices are passed through to containers automatically. For Vulkan, the render and card device group IDs are detected and applied as --group-add flags.

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

See LICENSE for the full text.