server: validate the backend before binding the port - #399
Closed
CryptVenture wants to merge 1 commit into
Closed
Conversation
The server backend defaults to CUDA and nothing verified it at startup, so on a
machine where that backend is not registered the process bound its port and
served the whole UI — health, model list, downloads — with an engine that
could never run anything. /health answered {"status":"ok","backend":"cuda"} on
an Apple-silicon host with no CUDA device, and the operator only learned the
truth when the first model load or generate failed.
main() now initialises the requested backend and device once, before
constructing the server, and frees it again. A missing backend or device
therefore aborts startup with the same message the engine would have raised
later, at the point the operator can still change --backend or --device.
The probe costs one backend init on a path that already loads Metal or CUDA
libraries and does not touch the request path.
Contributor
Author
|
Closing for now to stay inside the 3-concurrent-PR policy (see the discussion on #422). Nothing is wrong with the change and CI is green on it; I will reopen it as review slots free. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #369 as requested: this PR is only the startup backend probe. The
languagecontract gate, the ASR detail fields and the unload locking are separate PRs.The problem
--backenddefaults tocudaand nothing verified it at startup. On a host where that backend is not registered the server still bound its port and served the whole UI — health, model list, downloads — with an engine that could not run anything:/healthreportsokon an Apple-silicon machine with no CUDA device. The operator only finds out at the first model load or generate, by which time the failure looks like a model problem rather than a startup flag.The change
main()initialises the requested backend and device once before constructing the server, then frees it. A missing backend or device now aborts startup with the message the engine would have raised later:The message already names the devices that are available, so the operator can correct
--backend/--deviceimmediately. The probe runs on a path that loads the backend libraries anyway and does not touch the request path.Validation
Backend: Metal, Apple M4 Max, macOS 15.
main--ui --backend cuda/health→{"status":"ok","backend":"cuda"}CUDA backend requested but it is not registered in this build--ui --backend metal/health→{"status":"ok","backend":"metal"}/health→{"status":"ok","backend":"metal"}Build and tests:
cmake --build build/macos-metal-tests -j 12 ctest --test-dir build/macos-metal-tests # 40/40Scope
One file,
app/server/main.cpp, 15 added lines. Startup only; no request, model or response behaviour changes.