diff --git a/README.md b/README.md index 11e550e..62e813d 100644 --- a/README.md +++ b/README.md @@ -11,16 +11,17 @@ connect your coding agent to it, talk to it by voice, build a local RAG stack on it, and fine-tune it with LoRA. One command, entirely on your Mac. It runs the community's K-quant and IQ-quant GGUF builds exactly as -published. Size for size, these are the most accurate open quant formats -available ([accuracy per byte](#accuracy-per-byte)). The companion project -[mlx-kquant](https://github.com/asher/mlx-kquant) supplies the Metal kernels -that run these formats natively on Apple's +published. Size for size, these two quant formats are the most accurate +open formats available ([accuracy per byte](#accuracy-per-byte)). The +companion project [mlx-kquant](https://github.com/asher/mlx-kquant) +supplies the Metal kernels that run them natively on Apple's [MLX](https://github.com/ml-explore/mlx) framework. On the same file, gmlx -benchmarks faster than llama.cpp, with the gap widest at the 50-200k-token +benchmarks faster than llama.cpp. The gap is widest at the 50-200k-token contexts where coding agents and long sessions live ([performance](#performance)). A 100B+ model starts generating within -seconds of launch, and a MoE bigger than RAM still runs, streaming its -experts from disk ([bigger than memory](#bigger-than-memory)). +seconds of launch. A mixture-of-experts (MoE) model bigger than RAM still +runs, streaming its experts from disk +([bigger than memory](#bigger-than-memory)). @@ -32,19 +33,25 @@ experts from disk ([bigger than memory](#bigger-than-memory)). DeepSeek-V4-Flash IQ2_XXS: gmlx vs ds4-server prefill and decode vs KV depth -DeepSeek-V4-Flash is measured against antirez's ds4 (dwarfstar) +In both charts, higher is faster, and depth is the number of tokens already +in the context. DeepSeek-V4-Flash is measured against antirez's ds4 +(dwarfstar) ([benchmarks](https://github.com/asher/gmlx/blob/main/docs/benchmarks.md#deepseek-v4-flash-reference-engine-ds4-server)). - +![gmlx chat: a 27B model answering through a running server, with live tokens per second](https://raw.githubusercontent.com/asher/gmlx/main/docs/assets/demo.gif) + +A 27B model, already resident in a local server, answering at 46 tokens per +second. Recorded with `docs/assets/record-demo.sh` at true speed. ## Quickstart -Requires an Apple Silicon Mac. [uv](https://docs.astral.sh/uv/) and pipx -install a suitable Python themselves; a manual install needs Python 3.11+. On -macOS 26 and newer the Metal kernels install as a prebuilt wheel; older macOS -builds them from source (Xcode Command Line Tools, a few minutes). +Requires an Apple Silicon Mac, and macOS 26 or newer is recommended: there +the Metal kernels install as a prebuilt wheel. On older macOS versions the +install builds them from source, which needs full Xcode with its Metal +toolchain and a few minutes; the Command Line Tools alone are not enough +([troubleshooting](https://github.com/asher/gmlx/blob/main/docs/troubleshooting.md#the-install-fails-compiling-the-metal-kernels)). +[uv](https://docs.astral.sh/uv/) and pipx install a suitable Python +themselves; a manual install needs Python 3.11 or newer. ```sh uv tool install "gmlx[all]" # or: pipx install "gmlx[all]" @@ -62,19 +69,50 @@ curl localhost:8080/v1/chat/completions -d \ gmlx stop # the server ran detached ``` -`gmlx[all]` adds every optional feature to the core platform: the upgraded -chat TUI, voice chat, and the MCP assistant. The core alone already carries -serving, vision, embeddings, and the menu bar, so `gmlx[chat]` is a smaller -install that gives up only voice and the assistant. `gmlx init` adds either -one later. `uv tool` and pipx put `gmlx` on your PATH in every terminal. -`pip install "gmlx[all]"` into a venv you manage works too, with the command -available only while that venv is active. +This is the one-file form: any local `.gguf` runs, chats, or serves with no +other setup. The served model id comes from the file name (`qwen3-0.6b` +here). The intended setup for everything else is `gmlx init` +([next section](#set-up-with-gmlx-init)). + +`gmlx[all]` turns on every optional feature: the upgraded chat TUI, voice +chat, and the MCP assistant. The core install already carries serving, +vision, embeddings, and the menu bar, so `gmlx[chat]` is a smaller install +that gives up only voice and the assistant. `gmlx init` offers to add a +missing extra later. `uv tool` and pipx put `gmlx` on your PATH in every +terminal. `pip install "gmlx[all]"` into a venv you manage works too; the +command then exists only while that venv is active. + +A model typically needs roughly its file size in memory, plus the KV cache +(the per-conversation state, which grows with context length). The exception +is MoE models, which can run [bigger than memory](#bigger-than-memory). If +anything misbehaves, `gmlx doctor` checks the runtime, config, model paths, +and services in one pass. + +## Set up with gmlx init + +`gmlx init` is the setup path for the rest of the platform. It finds your +GGUF files, names them, and writes the one config that every other command +reads. Run bare, it opens a guided wizard: + +1. It scans the model folders you name. An existing LM Studio library or + Hugging Face cache works as-is. +2. It shows the models it found. You can rename ids, drop entries, set a + default, and add aliases. +3. It offers the on-disk prompt cache (recommended for coding agents) and + the optional speech, embedding, and reranking services. +4. It previews the config, then writes it to `~/.config/gmlx/gmlx.yaml`. -A model typically needs roughly its file size in memory, plus the KV cache. -The exception is MoE models, which can run -[bigger than memory](#bigger-than-memory). If anything misbehaves, -`gmlx doctor` checks the runtime, config, model paths, and services in one -pass. +```sh +gmlx init # the guided wizard +gmlx serve # start the server: finds the config, detaches, returns +gmlx list # the model ids the config defines +gmlx launch pi # point a coding agent at the server (auto-starts it) +``` + +Every command now takes a model id in place of a path: `gmlx chat +qwen3-0.6b-q4`, or the quickstart curl with the configured id. Every wizard +choice also has a flag, so `gmlx init --models-dir ~/models` scaffolds with +no questions. `gmlx service install` keeps the server running from login. The [getting-started guide](https://github.com/asher/gmlx/blob/main/docs/getting-started.md) is the full walkthrough, from install to a configured server with a @@ -93,7 +131,8 @@ the model (`--report-only`). `chat` is a multi-turn terminal REPL over a persist KV cache: streaming markdown rendering, sessions with `--resume`, `/commands` for live sampling changes, `/!` to stage shell output into a message, and image or audio input with a vision-language model (drag a file in). Sampling defaults come from each -model family's model card, and `@intents` switch the operating point per call: +model family's model card. `@intents` are named presets that switch those +settings per call: ```sh gmlx run model.gguf@creative --prompt "Write a haiku about entropy." @@ -101,8 +140,8 @@ gmlx chat qwen3.6-27b-q6 --profile instruct # --profile NAME = the flag form o ``` A `.gguf` path works with no setup. A bare id like `qwen3.6-27b-q6` names a -model from your server config - `gmlx list` shows yours, and the -[getting-started guide](https://github.com/asher/gmlx/blob/main/docs/getting-started.md) sets one up. +model from your server config: `gmlx list` shows yours, and +[`gmlx init`](#set-up-with-gmlx-init) sets one up. Details: the [CLI reference](https://github.com/asher/gmlx/blob/main/docs/cli.md). @@ -122,21 +161,33 @@ Details: [picking a model for your Mac](https://github.com/asher/gmlx/blob/main/ ### Serve an API -`serve` runs a continuously batched, multi-model server speaking three dialects on -one port: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages, all -streaming. Concurrent requests decode together in one batch, a new prompt's -prefill is paced so in-flight replies keep streaming, and a memory-headroom -gate admits new work only when it fits +`serve` runs a continuously batched, multi-model server. It speaks three +API dialects on one port, all streaming: OpenAI Chat Completions, OpenAI +Responses, and Anthropic Messages. Concurrent requests decode together in +one batch, a new prompt's prefill is paced so in-flight replies keep +streaming, and a memory-headroom gate admits new work only when it fits ([serving concurrent requests](https://github.com/asher/gmlx/blob/main/docs/performance.md#serving-concurrent-requests)). -It handles tool calling, structured output (`response_format: -json_schema`, grammar-constrained), logprobs, and vision messages. A YAML config -gives named models, reusable sampling profiles, aliases, and directory discovery. -Residency is managed (LRU with pinning and idle unload), and repeated prefixes are -served from a cross-request prompt cache with an optional SSD tier. The server -never contacts Hugging Face to satisfy a request. It binds loopback by default, -hardened against browser-borne attacks, with static-key auth for anything wider. -Config-defined assistant ids run the built-in MCP tool loop server-side, so a thin -client gets tools (and optionally memory) with no loop of its own. + +Beyond plain chat completions: + +- Tool calling, structured output (`response_format: json_schema`, + grammar-constrained), logprobs, and vision messages. +- A YAML config, written by [`gmlx init`](#set-up-with-gmlx-init): named + models, reusable sampling profiles, aliases, and directory discovery. +- Managed residency (LRU with pinning and idle unload) and a cross-request + prompt cache for repeated prefixes, with an optional SSD tier. +- Loopback by default and hardened against browser-borne attacks, with + static-key auth for anything wider. The server never contacts Hugging + Face to satisfy a request. +- Config-defined assistant ids run the built-in MCP tool loop server-side, + so a thin client gets tools (and optionally memory) with no loop of its + own. + + Details: the [server config reference](https://github.com/asher/gmlx/blob/main/docs/server-config.md) and the [assistant guide](https://github.com/asher/gmlx/blob/main/docs/assistant.md). @@ -150,6 +201,7 @@ config without touching your dotfiles and auto-starting the server if it is down gmlx launch pi --model qwen3.6-27b-q6@coding ``` +`--model` takes any id from `gmlx list`, with an optional `@intent`. Supported: pi, opencode, omp, claude-code, hermes, goose, the aichat and elia chat clients, and the Open WebUI browser app. A macOS menu-bar app shows what is resident and offers unload, restart, and logs, and `gmlx service install` @@ -279,11 +331,11 @@ recognizes and can synthesize a config for, or you supply `hf_source`. Preflight runs the architecture gate and checks each tensor's codec (its GGUF quantization type, like `Q4_K_M`) before any tensor bytes are read. Codec coverage spans all 19 K-quant, legacy, and IQ codecs plus the -native-fp pair mxfp4/nvfp4 (run directly from the file when the model is -bigger than RAM, repacked for speed when it fits - `GMLX_NATIVE_FP` -overrides the choice); in the rare case a file uses a type with no kernel -(ternary TQ, for example), preflight names it and lists what is supported so -you can pick another variant. +native-fp pair mxfp4/nvfp4 (layout choice covered in the +[streaming guide](https://github.com/asher/gmlx/blob/main/docs/streaming.md)). +In the rare case a file uses a type with no kernel (ternary TQ, for +example), preflight names it and lists what is supported so you can pick +another variant. Vision-language models load as a K-quant LLM GGUF paired with its float `mmproj` GGUF: supported families and caveats in the [VLM guide](https://github.com/asher/gmlx/blob/main/docs/vlm.md). Want a family @@ -323,7 +375,7 @@ flowchart TB serve --> aux["embeddings + rerank
STT + TTS"] ``` -Serving-side mechanics - engine, batching, and the HTTP layers: +Serving-side mechanics (engine, batching, and the HTTP layers): [docs/serving-architecture.md](https://github.com/asher/gmlx/blob/main/docs/serving-architecture.md). ## Python API @@ -372,7 +424,7 @@ Guides: - [docs/troubleshooting.md](https://github.com/asher/gmlx/blob/main/docs/troubleshooting.md): the common failures and their fixes. - [docs/migrating.md](https://github.com/asher/gmlx/blob/main/docs/migrating.md): coming from llama.cpp, Ollama, or - LM Studio - what carries over and what maps to what. + LM Studio: what carries over and what maps to what. Reference: diff --git a/docs/assets/demo.gif b/docs/assets/demo.gif new file mode 100644 index 0000000..91e1981 Binary files /dev/null and b/docs/assets/demo.gif differ diff --git a/docs/assets/demo.tape b/docs/assets/demo.tape deleted file mode 100644 index 4d6d1a9..0000000 --- a/docs/assets/demo.tape +++ /dev/null @@ -1,33 +0,0 @@ -# vhs tape for the README demo GIF. -# -# Record with: -# brew install vhs -# vhs docs/assets/demo.tape # writes docs/assets/demo.gif -# -# Run it from a scratch directory and pre-pull the model once before recording, -# so the pull replay is the instant already-downloaded path and the whole tape -# stays under ~45 seconds: -# gmlx pull hf:unsloth/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q4_K_M.gguf --to . -# -# The Sleep durations below are first guesses; tune them against the actual -# pull-replay and load/generate timings on the recording machine before -# publishing the GIF. - -Output docs/assets/demo.gif - -Set FontSize 16 -Set Width 960 -Set Height 540 -Set TypingSpeed 40ms - -Type "gmlx pull hf:unsloth/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q4_K_M.gguf --to ." -Sleep 500ms -Enter -Sleep 4s - -Type `gmlx run Qwen3-0.6B-Q4_K_M.gguf --prompt "Explain KV caching in one sentence." --max-tokens 60` -Sleep 500ms -Enter -Sleep 25s - -Sleep 3s diff --git a/docs/assets/record-demo.sh b/docs/assets/record-demo.sh new file mode 100755 index 0000000..8333acd --- /dev/null +++ b/docs/assets/record-demo.sh @@ -0,0 +1,130 @@ +#!/bin/bash +# Record the README demo GIF: one chat turn against a running gmlx server. +# +# brew install asciinema agg tmux +# gmlx serve # leave it up; the model stays resident +# docs/assets/record-demo.sh # writes docs/assets/demo.gif +# +# The demo shows the normal steady state: a server is already up, so `gmlx +# chat` connects to it as a plain client and the prompt appears at once. Warm +# the model with one request before recording, otherwise the first turn pays +# the load and the timings misrepresent a working setup. +# +# Timings are not scripted. The driver watches the pane and moves on when the +# prompt appears and when the turn's stats line lands, so the recording runs +# at the speed the machine actually delivers. +set -eu +set -o pipefail + +MODEL=${MODEL:-qwen3.6-27b-q6-k@instruct} +QUESTION=${QUESTION:-"Write a Python LRU cache decorator, then explain it in two bullets."} +OUT=${OUT:-docs/assets/demo.gif} +COLS=${COLS:-110} +ROWS=${ROWS:-28} +HOLD=${HOLD:-2.2} # keep recording this long past the reply +LOOP_PAUSE=${LOOP_PAUSE:-5.0} # still frame at the end, before the loop +FONT_SIZE=${FONT_SIZE:-16} +CAST_OUT=${CAST_OUT:-} # set to keep the cast for re-rendering + +# A thinking model spends the whole recording reasoning before it answers, so +# pick a non-thinking id or intent (@instruct on the Qwen families). + +TM=$(command -v tmux) +SOCK=gmlx-demo +SESSION=d +WORK=$(mktemp -d) +trap 'rm -rf "$WORK"; $TM -L "$SOCK" kill-server 2>/dev/null || true' EXIT + +pane() { $TM -L "$SOCK" capture-pane -p -t "$SESSION" 2>/dev/null; } + +# Type one character at a time so the recording reads as a person typing. +type_str() { + local s=$1 i + for (( i = 0; i < ${#s}; i++ )); do + $TM -L "$SOCK" send-keys -t "$SESSION" -l -- "${s:i:1}" + sleep 0.035 + done +} + +wait_for() { # wait_for REGEX TIMEOUT LABEL + local re=$1 timeout=$2 label=$3 ticks=0 limit + limit=$(( timeout * 2 )) # half-second ticks + while [ "$ticks" -lt "$limit" ]; do + if pane | grep -qE -- "$re"; then + echo "[rec] $label after $(( ticks / 2 ))s" + return 0 + fi + sleep 0.5 + ticks=$(( ticks + 1 )) + done + echo "[rec] gave up waiting for $label after ${timeout}s" >&2 + return 1 +} + +$TM -L "$SOCK" kill-server 2>/dev/null || true # no server yet is the norm +# A throwaway cache keeps the operator's own prompt history out of the frame: +# prompt_toolkit would otherwise autosuggest past prompts as grey ghost text. +$TM -L "$SOCK" new-session -d -s "$SESSION" -x "$COLS" -y "$ROWS" \ + "env PS1='$ ' XDG_CACHE_HOME=$WORK/cache bash --norc --noprofile -i" +$TM -L "$SOCK" set -g status off +sleep 0.5 + +( + # However this driver ends, close the session so the recording stops with + # it instead of running to the asciinema timeout. + trap '$TM -L "$SOCK" kill-session -t "$SESSION" 2>/dev/null || true' EXIT + sleep 1.5 + type_str "gmlx chat $MODEL" + sleep 0.4 + $TM -L "$SOCK" send-keys -t "$SESSION" Enter + wait_for '^>>' 180 "chat ready" + sleep 1.0 + type_str "$QUESTION" + sleep 0.4 + $TM -L "$SOCK" send-keys -t "$SESSION" Enter + wait_for '\[chat\] prompt' 240 "reply complete" + sleep "$HOLD" +) & + +# --window-size pins the geometry: a headless take otherwise falls back to +# 80x24 and rewraps every line. +asciinema rec --overwrite --window-size "${COLS}x${ROWS}" \ + --command "$TM -L $SOCK attach -t $SESSION" "$WORK/demo.cast" +wait + +# End on the stats line. What follows is the session teardown, whose +# "[server exited]" notice reads as a crashed server in a looping GIF. +python3 - "$WORK/demo.cast" "$WORK/trim.cast" <<'PY' +import json +import sys + +src, dst = sys.argv[1], sys.argv[2] +lines = open(src).read().splitlines() +header, events = lines[0], lines[1:] +cut = max((i for i, ln in enumerate(events) + if "[chat] prompt" in json.loads(ln)[2]), default=None) +if cut is None: + sys.exit("no stats line in the cast: did the turn finish?") +kept = [json.loads(ln) for ln in events[:cut + 6]] +with open(dst, "w") as fh: + fh.write(header + "\n") + for ev in kept: + fh.write(json.dumps(ev) + "\n") +PY + +# A black background matching the TUI, and an idle limit high enough that agg +# never silently compresses a real pause into a shorter one. +# --last-frame-duration is what actually sets the rest before the loop +# restarts; agg otherwise caps that final frame at 3 seconds. +agg --font-size "$FONT_SIZE" --idle-time-limit 3600 \ + --last-frame-duration "$LOOP_PAUSE" \ + --theme "000000,e5e5e5,000000,cd3131,0dbc79,e5e510,2472c8,bc3fbc,11a8cd,e5e5e5,666666,f14c4c,23d18b,f5f543,3b8eea,d670d6,29b8db,ffffff" \ + "$WORK/trim.cast" "$OUT" +echo "[rec] wrote $OUT" + +# The cast is the re-renderable source: keeping it means a later change of +# font size, theme, or end pause costs an agg run instead of a new recording. +if [ -n "$CAST_OUT" ]; then + cp "$WORK/trim.cast" "$CAST_OUT" + echo "[rec] wrote $CAST_OUT" +fi diff --git a/docs/getting-started.md b/docs/getting-started.md index bf0fec6..2b75139 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -10,9 +10,12 @@ can use, so stop wherever your needs are met. - An Apple Silicon Mac (any M-series chip). - Python 3.11 or newer. Installing with `uv` or `pipx` fetches one for you. - Disk space for models. -- On macOS versions before 26: the Xcode Command Line Tools - (`xcode-select --install`), because the install compiles the Metal kernels - from source there. +- macOS 26 or newer, recommended: the Metal kernels then install as a + prebuilt wheel. On older macOS versions the install compiles them from + source, which needs full Xcode with its Metal toolchain; the Command Line + Tools alone do not include the `metal` compiler. Recent Xcode versions + fetch that toolchain as a separate download + (`xcodebuild -downloadComponent MetalToolchain`). - For voice and the browser chat app: [Homebrew](https://brew.sh), used below to install `ffmpeg`. @@ -62,7 +65,7 @@ no such step. The `mlx-kquant` dependency (the Metal kernels) arrives as a prebuilt wheel from PyPI on macOS 26 and newer. On older macOS versions -the install builds it from source: the Command Line Tools requirement from +the install builds it from source: the full-Xcode requirement from [What you need](#what-you-need), and a few minutes of compile time. Tab completion is worth the one line: add `eval "$(gmlx completion zsh)"` to diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 45df235..ec6265d 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -12,14 +12,18 @@ the runtime and server. ## The install fails compiling the Metal kernels Symptom: on macOS versions before 26, `pip install` fails partway through -building `mlx-kquant`, usually with a compiler or SDK error. - -On older macOS the kernels build from source, which needs Apple's Xcode -Command Line Tools. If the build cannot find a compiler, install them -(`xcode-select --install`) and re-run the pip install. If you upgraded macOS -recently, the tools may be stale for the new SDK: reinstall them -(`sudo rm -rf /Library/Developer/CommandLineTools && xcode-select --install`) -and try again. On macOS 26 and newer none of this applies - the kernels +building `mlx-kquant`. Typical messages are a compiler or SDK error, or +`cannot execute tool 'metal'`. + +On older macOS the kernels build from source, and the build needs full +Xcode. The C++ parts compile with the Command Line Tools, but the Metal +shaders compile with `xcrun metal`, which the Command Line Tools do not +include. Install Xcode, select it +(`sudo xcode-select -s /Applications/Xcode.app`), and re-run the pip +install. Recent Xcode versions fetch the Metal toolchain as a separate +download: run `xcodebuild -downloadComponent MetalToolchain` once. If the +build still fails after a macOS upgrade, update Xcode so its SDK matches, +and try again. On macOS 26 and newer none of this applies: the kernels arrive as a prebuilt wheel. ## `gmlx: command not found` in a new terminal diff --git a/gmlx/chat.py b/gmlx/chat.py index 9144b7d..4b14b1d 100644 --- a/gmlx/chat.py +++ b/gmlx/chat.py @@ -1599,11 +1599,14 @@ def _toolbar(): parts = [] if state.model_name: parts.append((state.model_name, False)) - parts += [ - (f"temp={s['temp']:g}", True), - (f"top-p={s['top_p']:g}", True), - (f"max-tok={s['max_tokens'] or 'off'}", True), - ] + # In server mode the served profile decides sampling, and this client + # forwards only the knobs it actually set (see _sync_assistant_extra). + # Showing an unforwarded knob would report a value the reply never + # used, so those stay off the toolbar until something moves them. + for key, label in (("temp", "temp"), ("top_p", "top-p")): + if _knob_shown(state, key): + parts.append((f"{label}={s[key]:g}", True)) + parts.append((f"max-tok={s['max_tokens'] or 'off'}", True)) if state.ctx_used and state.ctx_max: parts.append( (f"ctx {_fmt_k(state.ctx_used)}/{_fmt_k(state.ctx_max)}", @@ -2269,17 +2272,33 @@ def cleanup(): return brain, model_request, base_url, extra +def _knob_forwarded(state, key: str) -> bool: + """Whether a sampling knob rides along to the server: the CLI set it, or + a /command moved it off the session baseline. Everything else is left for + the served profile to resolve.""" + baseline = state.assistant_baseline + touched = state.assistant_touched + if baseline is None or touched is None: + return False + return key in touched or state.sampling[key] != baseline[key] + + +def _knob_shown(state, key: str) -> bool: + """Whether the toolbar can name a sampling value. A local session owns + every knob; a server-mode session owns only the forwarded ones.""" + if state.assistant_brain is None: + return True + return _knob_forwarded(state, key) + + def _sync_assistant_extra(state) -> None: """Refresh the forwarded sampling knobs from the live /command values: a knob rides along once the CLI set it or a /command moved it off the session baseline; everything else stays server-side.""" extra = state.assistant_extra - s = state.sampling - baseline = state.assistant_baseline - touched = state.assistant_touched for key, payload in _ASSISTANT_SAMPLING.items(): - if key in touched or s[key] != baseline[key]: - extra[payload] = s[key] + if _knob_forwarded(state, key): + extra[payload] = state.sampling[key] else: extra.pop(payload, None) @@ -2863,8 +2882,9 @@ def cmd_chat(argv: list[str] | None = None, prog: str = "gmlx chat") -> int: if _auto_server(args, parser): args.server = True args.no_start = True # the probe saw it up; never start one - print("[chat] config server is up - chatting through it " - "(--local loads in-process instead)") + # The server-mode banner below names the id and url; it carries the + # --local hint too when the server was picked up automatically. + args.auto_server = True if args.server: args.assistant = True # same server path; extras off in setup if args.assistant: @@ -3235,7 +3255,9 @@ def _finish_load(): print(f"[chat] MTP speculative decoding on ({kind} drafter)") if brain is not None: if args.server: - print(f"[chat] server mode: {model_request} via {base_url}") + hint = (" (--local loads in-process)" + if getattr(args, "auto_server", False) else "") + print(f"[chat] server mode: {model_request} via {base_url}{hint}") else: tools = ", ".join(brain.tools.names()) or "(none)" mem = (f" - memory: {brain.memory.count()} items"