Hi,
flexllama webgui has nice features with runtime control to start, stop, restart, and nice visibility for quick and simple metrics.
Possible to include additional details for each model runner to display recent stderr cli log output from llama-server in order to see the actual activity occurring?
This information is beneficial sometimes for surfacing to users and admins in the event llama-server crashes but process does not get killed or restarted, or when llama-server is stuck in a loop with gpu util appearing to be actively working prompt processing but time passes and no stream output arrives; upon review llamacpp stderr the process is stuck in dead inactive loop.
This additional stderr information in webgui would assist in gaining visibility allowing end users to know more confidently when to restart runners services without requiring access to cli and logs etc on the hosts.
For more details, we sometimes have multi turn chat and coding sessions operating on CPU-only machines and when context grows very large prompt response streams, time to first token , after many turns the there is no stream response because it can take many minutes or longer, without any visible streaming tokens out to user or to agent the end user is flying blind just by looking at FlexLLama Status Dashboard.
We have experience running into bugs and issues where llamacpp will continue to loop cpu or gpu utilization pegged near 100pct when in fact the process is doing nothing, never dies, never crashes, and requires intervention to be killed in cli.
If devs are ambitious please can also add feature request for scanning stderr, alerting to stderr stalls and crashes, and to pro-actively catch these llamacpp failing scenarios and kill then restart the runner process automatically. Displaying some interval info from llamacpp GET /health would also be nice.
Thank you devs for great work.
Hi,
flexllama webgui has nice features with runtime control to start, stop, restart, and nice visibility for quick and simple metrics.
Possible to include additional details for each model runner to display recent stderr cli log output from llama-server in order to see the actual activity occurring?
This information is beneficial sometimes for surfacing to users and admins in the event llama-server crashes but process does not get killed or restarted, or when llama-server is stuck in a loop with gpu util appearing to be actively working prompt processing but time passes and no stream output arrives; upon review llamacpp stderr the process is stuck in dead inactive loop.
This additional stderr information in webgui would assist in gaining visibility allowing end users to know more confidently when to restart runners services without requiring access to cli and logs etc on the hosts.
For more details, we sometimes have multi turn chat and coding sessions operating on CPU-only machines and when context grows very large prompt response streams, time to first token , after many turns the there is no stream response because it can take many minutes or longer, without any visible streaming tokens out to user or to agent the end user is flying blind just by looking at FlexLLama Status Dashboard.
We have experience running into bugs and issues where llamacpp will continue to loop cpu or gpu utilization pegged near 100pct when in fact the process is doing nothing, never dies, never crashes, and requires intervention to be killed in cli.
If devs are ambitious please can also add feature request for scanning stderr, alerting to stderr stalls and crashes, and to pro-actively catch these llamacpp failing scenarios and kill then restart the runner process automatically. Displaying some interval info from llamacpp GET /health would also be nice.
Thank you devs for great work.