Skip to content

docs(readme): cleanup #57

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
asher merged 1 commit into from
Aug 13, 2026
+21 −15
Merged

docs(readme): cleanup #57

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 21 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ in the context. DeepSeek-V4-Flash is measured against antirez's ds4
A 27B model, already resident in a local server, answering at 46 tokens per
second. Recorded with `docs/assets/record-demo.sh` at true speed.

A [one-minute video](#mtp-and-concurrent-streams) below shows the same server move from a
single chat at full MTP speed to four concurrent streams, and back, with
no break in the live stream.

## Quickstart

Requires an Apple Silicon Mac, and macOS 26 or newer is recommended: there
Expand Down Expand Up @@ -183,17 +187,6 @@ Beyond plain chat completions:
so a thin client gets tools (and optionally memory) with no loop of its
own.

One chat decodes solo at ~150 tok/s with MTP. Three more prompts then
arrive. Speculation pauses. The server paces their prefill, so the live
stream keeps its speed. The four streams then run at 50-57 tok/s each. As
the sessions finish, speculation re-arms mid-stream. The remaining chat
returns to about 150 tok/s, through to the end of a 5200-token response.

The video is real time with no edits. The tok/s counters come from the
server, reflecting the Qwen tokenizer.

https://github.com/user-attachments/assets/de5dab84-3155-4cee-aa57-7d0b9c726ec5

Details: the [server config reference](https://github.com/asher/gmlx/blob/main/docs/server-config.md) and the
[assistant guide](https://github.com/asher/gmlx/blob/main/docs/assistant.md).

Expand Down Expand Up @@ -283,10 +276,23 @@ workloads. KV-cache quantization frees memory at long contexts. And
disk-streamed execution runs MoE models larger than memory
([below](#bigger-than-memory)). The file you pick matters too: a uniform
K-quant decodes meaningfully faster than a heavily mixed one at similar or
better quality. The
[performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md)
covers when each lever pays off, with numbers, and where llama.cpp still
leads.
better quality. Details: the
[performance guide](https://github.com/asher/gmlx/blob/main/docs/performance.md).

### MTP and concurrent streams

The video below shows MTP and concurrent serving together on a live
server. One chat decodes solo at ~150 tok/s with MTP. Three more prompts
then arrive. Speculation pauses. The server paces their prefill, so the
live stream keeps its speed. The four streams then run at 50-57 tok/s
each. As the sessions finish, speculation re-arms mid-stream. The
remaining chat returns to ~150 tok/s, through to the end of a 5200-token
response.

The video is real time with no edits. The tok/s counters come from the
server, reflecting the Qwen tokenizer.

https://github.com/user-attachments/assets/de5dab84-3155-4cee-aa57-7d0b9c726ec5

### Bigger than memory

Expand Down