feat(server): idle model auto-unload + pre-load memory guard (cross-platform CI) - #1
Closed
gqf2008 wants to merge 5 commits into
Closed
feat(server): idle model auto-unload + pre-load memory guard (cross-platform CI)#1gqf2008 wants to merge 5 commits into
gqf2008 wants to merge 5 commits into
Conversation
The 5-minute idle unload used to live in an external Python monitor inside the audio-server wrapper, which polled the server log mtime and called /v1/tasks/unload_all_models. Move it into audiocpp_server itself: - new ServerConfig field idle_unload_ms (default 0 = disabled), parsed from server.json and overridable via --idle-unload-ms - a background thread unloads every resident non-busy model once the server has been idle that long without a model load/run; the next request reloads lazily This drops the log-mtime heuristic (which required --log trace spam) and removes the need for the external Python wrapper.
Before every lazy model load, estimate the model's resident footprint (weights plus runtime overhead) and compare against free host memory and, for GPU backends, the backend device's free memory. Refuse the load with HTTP 503 insufficient_memory when estimate + configured headroom does not fit, instead of exhausting the machine (the previous failure mode was kIOGPUCommandBufferCallbackErrorOutOfMemory after models accumulated on a 16GB Mac). - add ServerConfig.min_free_memory_mb (default 512 MiB headroom), parsed from server.json and overridable via --min-free-memory-mb - add a macOS implementation of available_host_memory_bytes() using Mach VM stats (free + inactive + purgeable pages); Linux/Windows were already covered - add InsufficientMemoryError, mapped to 503 insufficient_memory
Independent review found no criticals; fix the actionable findings: - --min-free-memory-mb help text now matches the actual 512 MiB default - estimate_model_memory_bytes() sums directory-style model trees (with depth/file limits) instead of counting only regular files, so directory models are no longer estimated as 0 - the model load path is serialized even when max_loaded_models is 0, so concurrent lazy loads cannot both pass the memory pre-check - expose engine::core::ensure_backends_loaded() and call it before the GPU memory query so the very first load actually runs the device check - the idle-unload thread now wakes on shutdown in <=250ms slices instead of waiting out a full poll interval - document idle_unload_ms / min_free_memory_mb in app/server/README.md and example.json - add server_config_test coverage for the new fields (defaults, overrides, negative rejection)
Run audiocpp_server build + server_config_test on ubuntu/windows/macos to prove the idle-unload and pre-load memory-check changes compile and behave on all three desktop platforms. GPU backends are off here (the changes query memory through the backend-agnostic ggml_backend_dev_memory; existing workflows already cover CUDA/Vulkan/Metal builds).
The Linux build step used nproc and its condition also matched macOS, so macOS ran both build steps and the nproc one stalled. Scope the nproc step to Linux and let macOS use sysctl -n hw.logicalcpu.
Owner
Author
|
Superseded by upstream PR: 0xShug0#306 (this fork PR only served as CI staging). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
--idle-unload-ms: server-side idle model auto-unload (replaces the external Python monitor that polled the log mtime)--min-free-memory-mb: pre-load memory check against host RAM and GPU VRAM; returns HTTP 503insufficient_memorywhen the estimated footprint plus headroom does not fitavailable_host_memory_bytes()(Mach VM stats: free + inactive + purgeable)Verification
server_config_testpassedNotes
idle_unload_ms=0(disabled),min_free_memory_mb=512(0 disables the extra headroom)max_loaded_models=0so concurrent lazy loads cannot both pass the pre-checkggml_backend_dev_memory(CUDA/HIP/Vulkan/Metal); CPU skips the device check (host check still applies)