m5max freezes

[qinyanyu]

Run server on m5max 128g
then do curl talk, whole system freezes

[fry69]

Run server on m5max 128g then do curl talk, whole system freezes

Can you please post a curl request that triggers this behavior?

Otherwise the current main is very stable for me on my M5 Max 128GB, I ran a few stress tests and benchmarks today without problems, see #143 if interested.

Make sure you are running the latest commit:

git pull

And you use a clean build:

make clean && make

The try running the test suite:

make test

Does this work for you and complete without error?

[antirez]

The mount of information provided is strictly non-actionable. That's like going to the doctor and saying a total of "I don't feel very well", more or less :)

[qinyanyu]

liwuliu@Liwus-MacBook-Pro ds4 % make test
cc -O3 -ffast-math -mcpu=native -Wall -Wextra -std=c99 -Wno-unused-function -c -o ds4_test.o tests/ds4_test.c
cc -O3 -ffast-math -mcpu=native -Wall -Wextra -std=c99 -o ds4_test ds4_test.o rax.o ds4.o ds4_metal.o -lm -pthread -framework Foundation -framework Metal
./ds4_test
long-context:
ds4: Metal device Apple M5 Max, 128.00 GiB RAM
ds4: requesting Metal residency (may take tens of seconds)... done
ds4: warming Metal model views... done
ds4: Metal model views created in 1.370 ms, residency requested in 7612.866 ms, warmup 6.451 ms (mapped 82697.67 MiB from offset 5.08 MiB)
ds4: Metal mapped mmaped model as 2 overlapping shared buffers
ds4: metal backend initialized for graph diagnostics
ds4-test: long-context prefill 0/30474
ds4-test: long-context prefill 8192/30474
ds4-test: long-context prefill 16384/30474
ds4-test: long-context prefill 24576/30474
ds4-test: long-context prefill 30474/30474
long-context: OK
tool-call-quality:
ds4-test: tool-call quality fast path
ds4-test: tool-call quality exact path
ds4: Metal device Apple M5 Max, 128.00 GiB RAM
ds4: requesting Metal residency (may take tens of seconds)... done
ds4: warming Metal model views... done
ds4: Metal model views created in 1.491 ms, residency requested in 261.974 ms, warmup 2.287 ms (mapped 82697.67 MiB from offset 5.08 MiB)
ds4: Metal mapped mmaped model as 2 overlapping shared buffers
ds4: metal backend initialized for graph diagnostics
tool-call-quality: OK
logprob-vectors:
ds4: Metal device Apple M5 Max, 128.00 GiB RAM
ds4: requesting Metal residency (may take tens of seconds)... done
ds4: warming Metal model views... done
ds4: Metal model views created in 59.053 ms, residency requested in 257.284 ms, warmup 2.340 ms (mapped 82697.67 MiB from offset 5.08 MiB)
ds4: Metal mapped mmaped model as 2 overlapping shared buffers
ds4: metal backend initialized for graph diagnostics
ds4-test: vector short_italian_fact
ds4-test: vector short_code_completion
ds4-test: vector short_reasoning_plain
ds4-test: vector long_memory_archive
ds4-test: vector long_code_audit
logprob-vectors: OK
metal-kernels:
ds4: requesting Metal residency (may take tens of seconds)... done
ds4: warming Metal model views... done
ds4: Metal model views created in 0.009 ms, residency requested in 528.429 ms, warmup 0.459 ms (mapped 4.00 MiB from offset 0.00 MiB)
ds4: Metal mapped mmaped model as 3 overlapping shared buffers
metal-kernels: OK
server:
0515 15:40:04 ds4-server: kv cache evicted reason=disk-cache-full tokens=512 hits=0 size=0.00 MiB file=/tmp/ds4-kv-evict-test.2taXDb/1111111111111111111111111111111111111111.kv
0515 15:40:04 ds4-server: kv cache evicted reason=disk-cache-full tokens=4096 hits=0 size=0.00 MiB file=/tmp/ds4-kv-current-store-evict-test.WbBvMo/1111111111111111111111111111111111111111.kv
0515 15:40:04 ds4-server: kv cache evicted reason=disk-cache-full tokens=4096 hits=0 size=0.00 MiB file=/tmp/ds4-kv-oversize-store-evict-test.85rDkV/2222222222222222222222222222222222222222.kv
0515 15:40:04 ds4-server: kv cache evicted reason=disk-cache-full tokens=2048 hits=15 size=0.00 MiB file=/tmp/ds4-kv-stale-hit-evict-test.brBiLa/1111111111111111111111111111111111111111.kv
0515 15:40:04 ds4-server: kv cache evicted reason=disk-cache-full tokens=512 hits=0 size=0.00 MiB file=/tmp/ds4-kv-live-prefix-test.6WOQ9S/1111111111111111111111111111111111111111.kv
server: OK
ds4 tests: ok

[qinyanyu]

liwuliu@Liwus-MacBook-Pro ds4 % ./ds4-server --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192
ds4: Metal device Apple M5 Max, 128.00 GiB RAM
ds4: requesting Metal residency (may take tens of seconds)... done
ds4: warming Metal model views... done
ds4: Metal model views created in 1.867 ms, residency requested in 266.976 ms, warmup 3.079 ms (mapped 82697.67 MiB from offset 5.08 MiB)
ds4: Metal mapped mmaped model as 2 overlapping shared buffers
ds4: metal backend initialized for graph diagnostics
0515 15:43:03 ds4-server: context buffers 1896.58 MiB (ctx=100000, backend=metal, prefill_chunk=2048, raw_kv_rows=2304, compressed_kv_rows=25002)
0515 15:43:03 ds4-server: KV disk cache /tmp/ds4-kv (budget=8192 MiB, cross-quant=accept, min=512, cold_max=30000, continued=10000, trim=32, align=2048, hit_half_life=21600s)
0515 15:43:03 ds4-server: listening on http://127.0.0.1:8000

[qinyanyu]

Then I will run the following:
curl http://127.0.0.1:8000/v1/chat/completions
-H 'Content-Type: application/json'
-d '{
"model":"deepseek-v4-flash",
"messages":[{"role":"user","content":"List three Redis design principles."}],
"stream":true
}'

[qinyanyu]

Seems this time it returns as follows without freezing.
I did pull out some new changes from git and made.
liwuliu@Liwus-MacBook-Pro ds4 % curl http://127.0.0.1:8000/v1/chat/completions
-H 'Content-Type: application/json'
-d '{
"model":"deepseek-v4-flash",
"messages":[{"role":"user","content":"List three Redis design principles."}],
"stream":true
}'
data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885098,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885098,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"We "},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885098,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"need "},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885098,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"to lis"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885098,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"t thre"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885098,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"e Redis"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885098,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":" design pri"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"n"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"ciples"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":". R"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"ed"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"is is a "},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"pop"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"ul"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"ar in"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"-memo"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"ry data st"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"ructur"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"e"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":" stor"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"e, "},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"us"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"ed as a d"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"a"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"tabase"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":","},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":" cac"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"he, and "},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"message"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":" "},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":"broker."},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":" Common"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1778885099,"model":"deepseek-v4-flash","choices":[{"index":0,"delta":{"reasoning_content":" design pri"},"finish_reason":null}]}

[qinyanyu]

Now works with Open Web-UI. Thanks.
I think the delta I pulled today could have fixed it.