[pull] master from mudler:master

.agents/api-endpoints-and-auth.md

@@ -236,6 +236,58 @@ Use these HTTP status codes:
 
 If your endpoint should be tracked for usage (token counts, request counts), add the `usageMiddleware` to its middleware chain. See `core/http/middleware/usage.go` and how it's applied in `routes/openai.go`.
 
+## Control-plane database health metrics
+
+In distributed mode the frontend registers three OpenTelemetry gauges over the
+PostgreSQL control-plane database (`core/services/monitoring/control_plane_db.go`,
+wired in `core/application/distributed.go`). They reach `/metrics` through the
+same Prometheus exporter as the rest of the API metrics.
+
+| Metric | Meaning | Page when |
+|--------|---------|-----------|
+| `localai_control_plane_oldest_xmin_age` | Transactions elapsed since the oldest snapshot any backend still holds | above a few million, and rising |
+| `localai_control_plane_longest_transaction_seconds` | Age of the longest open transaction | above 3600 |
+| `localai_control_plane_dead_tuple_ratio` | Dead tuples per live tuple, labelled by `table`, on `backend_nodes`, `node_models` and `gallery_operations` | sustained above ~10 on a small table |
+
+A sustained high `localai_control_plane_oldest_xmin_age` is the one to page on.
+While it grows, autovacuum can reclaim nothing anywhere in the database no
+matter how often it runs, so the dead tuple ratio keeps climbing and a six-row
+registry table can reach hundreds of megabytes. Tuning autovacuum does not help.
+The fix is to find the transaction holding the horizon open and clear it:
+
+```sql
+SELECT pid, state, age(backend_xmin) AS xmin_age, now() - xact_start AS xact_age, query
+FROM pg_stat_activity
+WHERE backend_xmin IS NOT NULL
+ORDER BY age(backend_xmin) DESC;
+```
+
+Then `pg_terminate_backend(pid)` on the offenders, and `VACUUM (VERBOSE)` the
+bloated tables once the horizon has moved.
+
+**A healthy-looking xmin age does not on its own prove the horizon is free.**
+The gauge reads `pg_stat_activity`, which only sees live backends. Two other
+things pin the very same horizon and are invisible there, so either one can hold
+vacuum back while the gauge reads 0:
+
+```sql
+SELECT gid, prepared, database, transaction FROM pg_prepared_xacts;
+SELECT slot_name, active, xmin, catalog_xmin FROM pg_replication_slots;
+```
+
+An orphaned prepared transaction is cleared with `ROLLBACK PREPARED '<gid>'`,
+and a stale slot with `pg_drop_replication_slot('<slot_name>')`. Check both
+before concluding that a bloated table has some other cause.
+
+Sampling is scrape-driven behind a 30 second cache, so scrape frequency does not
+translate into database load. Failed and timed-out samples cost the same interval
+as successful ones, so a database that is already struggling is not retried on
+every scrape. A failed sample reports the last good values rather than failing the
+scrape, because these gauges matter most when the database is struggling. Before
+the first successful sample the gauges are absent rather than zero, since a zero
+xmin age would read as a healthy horizon: alert on `absent()` too if you need to
+distinguish "healthy" from "never sampled".
+
 ## Advertising surfaces — where to register a new capability
 
 Beyond routing and auth, LocalAI publishes its capability surface in **four independent places**. When you add an endpoint — especially one introducing a net-new capability like a new media type or a new auth-gated feature — you must update every relevant surface. These aren't optional: missing them means the endpoint works but is invisible to clients, admins, and the UI.

backend/cpp/audio-cpp/Makefile

@@ -9,7 +9,7 @@
 # recipe is a make target (not a prepare.sh) so 'make purge && make' is a clean
 # rebuild and so the bump bot can see the pin.
 
-AUDIO_CPP_VERSION?=3497b7cc44753e2c141d8fe60ac42cec433e3281
+AUDIO_CPP_VERSION?=f334cff70a68ea3d2e40d6638733e8c1ec434164
 AUDIO_CPP_REPO?=https://github.com/0xShug0/audio.cpp
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

backend/cpp/ds4/generation_limits.h

@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: MIT
+#pragma once
+
+#include <algorithm>
+
+namespace ds4cpp {
+
+inline int EffectiveGenerationLimit(int requested, int context_size,
+                                    int session_position) {
+  const int limit = requested > 0 ? requested : 256;
+  const int room = context_size - session_position;
+  if (room <= 1) return 0;
+  return std::min(limit, room - 1);
+}
+
+inline int RemainingGenerationBudget(int effective_limit, int produced) {
+  if (effective_limit <= produced) return 0;
+  return effective_limit - produced;
+}
+
+inline int SpeculativeAcceptedCapacity(int remaining, int draft_allowance,
+                                       int buffer_capacity) {
+  if (remaining <= 0 || draft_allowance < 0 || buffer_capacity <= 0) return 0;
+  return std::min({remaining, draft_allowance + 1, buffer_capacity});
+}
+
+} // namespace ds4cpp

backend/cpp/ds4/generation_limits_test.cpp

@@ -0,0 +1,92 @@
+// SPDX-License-Identifier: MIT
+
+#include "generation_limits.h"
+
+#include <cstdio>
+
+namespace {
+
+int failures = 0;
+
+void check_equal(int got, int want, const char *name) {
+  if (got == want) return;
+  std::fprintf(stderr, "FAIL %s: got %d, want %d\n", name, got, want);
+  failures++;
+}
+
+// Mutation caught: treating omitted or negative max_tokens as unlimited instead
+// of preserving DS4's legacy 256-token default.
+void test_nonpositive_uses_legacy_default_when_space_permits() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(0, 4096, 100), 256,
+              "zero max_tokens uses legacy default");
+  check_equal(ds4cpp::EffectiveGenerationLimit(-1, 4096, 100), 256,
+              "negative max_tokens uses legacy default");
+}
+
+// Mutation caught: applying the legacy default without clamping it to the
+// post-prefill context room and reserved slot.
+void test_legacy_default_is_clamped_by_context() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(0, 300, 100), 199,
+              "legacy default is context-clamped");
+}
+
+// Mutation caught: allowing an explicitly large request to overrun the
+// post-prefill context boundary.
+void test_large_positive_limit_is_clamped_to_context() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(32768, 32768, 100), 32667,
+              "large positive is context-clamped");
+}
+
+// Mutation caught: replacing every positive request with the legacy default
+// rather than preserving a smaller configured limit.
+void test_smaller_positive_limit_is_preserved() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(64, 4096, 100), 64,
+              "smaller positive is preserved");
+}
+
+// Mutation caught: consuming the final context slot instead of reserving it as
+// required by DS4's generation loop.
+void test_no_usable_room_returns_zero() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(32, 100, 99), 0,
+              "one remaining context slot is not usable");
+}
+
+// Mutation caught: sending the original generation limit to a later
+// speculative cycle instead of subtracting tokens already produced.
+void test_remaining_budget_accounts_for_produced_tokens() {
+  check_equal(ds4cpp::RemainingGenerationBudget(10, 4), 6,
+              "remaining budget subtracts produced tokens");
+  check_equal(ds4cpp::RemainingGenerationBudget(10, 12), 0,
+              "remaining budget never becomes negative");
+}
+
+// Mutation caught: giving speculative evaluation capacity beyond either the
+// output budget, the draft allowance plus its first target token, or the fixed
+// accepted-token buffer.
+void test_speculative_capacity_obeys_all_bounds() {
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(3, 8, 8), 3,
+              "capacity respects remaining output budget");
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 4, 8), 5,
+              "capacity includes one target token beyond draft allowance");
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 8, 6), 6,
+              "capacity respects fixed buffer");
+}
+
+} // namespace
+
+int main() {
+  test_nonpositive_uses_legacy_default_when_space_permits();
+  test_legacy_default_is_clamped_by_context();
+  test_large_positive_limit_is_clamped_to_context();
+  test_smaller_positive_limit_is_preserved();
+  test_no_usable_room_returns_zero();
+  test_remaining_budget_accounts_for_produced_tokens();
+  test_speculative_capacity_obeys_all_bounds();
+
+  if (failures == 0) {
+    std::fprintf(stderr, "all generation limit checks passed\n");
+    return 0;
+  }
+  std::fprintf(stderr, "%d check(s) failed\n", failures);
+  return 1;
+}

backend/cpp/ds4/grpc-server.cpp

@@ -10,6 +10,7 @@
 
 #include "dsml_parser.h"   // populated in Task 12
 #include "dsml_renderer.h" // populated in Task 16
+#include "generation_limits.h"
 #include "kv_cache.h"      // populated in Task 17
 
 extern "C" {
@@ -769,7 +770,6 @@ class DS4Backend final : public backend::Backend::Service {
         }
         ds4_tokens prompt = {};
         build_prompt(g_engine, request, &prompt);
-        int n_predict = request->tokens() > 0 ? request->tokens() : 256;
 
         const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
         const bool starts_in_thinking = think_enabled &&
@@ -792,6 +792,9 @@ class DS4Backend final : public backend::Backend::Service {
         int prompt_len = prompt.len;
         ds4_tokens_free(&prompt);
         if (rc == 0) {
+            const int n_predict = ds4cpp::EffectiveGenerationLimit(
+                request->tokens(), ds4_session_ctx(g_session),
+                ds4_session_pos(g_session));
             const int eos = ds4_token_eos(g_engine);
             const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
             int produced = 0;
@@ -810,9 +813,12 @@ class DS4Backend final : public backend::Backend::Service {
                 if (draft_max > 0 && sp.temperature <= 0.0f) {
                     constexpr int kAcceptedMax = 8;
                     int accepted[kAcceptedMax];
-                    int cap = std::min(kAcceptedMax, draft_max + 1);
+                    const int remaining = ds4cpp::RemainingGenerationBudget(
+                        n_predict, produced);
+                    const int cap = ds4cpp::SpeculativeAcceptedCapacity(
+                        remaining, draft_max, kAcceptedMax);
                     int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, draft_max, eos,
+                        g_session, first, remaining, eos,
                         accepted, cap, err, sizeof(err));
                     if (n < 0) { rc = -1; break; }
                     bool stop = false;
@@ -873,7 +879,6 @@ class DS4Backend final : public backend::Backend::Service {
         }
         ds4_tokens prompt = {};
         build_prompt(g_engine, request, &prompt);
-        int n_predict = request->tokens() > 0 ? request->tokens() : 256;
 
         const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
         const bool starts_in_thinking = think_enabled &&
@@ -891,6 +896,9 @@ class DS4Backend final : public backend::Backend::Service {
         int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
         ds4_tokens_free(&prompt);
         if (rc == 0) {
+            const int n_predict = ds4cpp::EffectiveGenerationLimit(
+                request->tokens(), ds4_session_ctx(g_session),
+                ds4_session_pos(g_session));
             const int eos = ds4_token_eos(g_engine);
             const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
             int produced = 0;
@@ -908,9 +916,12 @@ class DS4Backend final : public backend::Backend::Service {
                 if (draft_max > 0 && sp.temperature <= 0.0f) {
                     constexpr int kAcceptedMax = 8;
                     int accepted[kAcceptedMax];
-                    int cap = std::min(kAcceptedMax, draft_max + 1);
+                    const int remaining = ds4cpp::RemainingGenerationBudget(
+                        n_predict, produced);
+                    const int cap = ds4cpp::SpeculativeAcceptedCapacity(
+                        remaining, draft_max, kAcceptedMax);
                     int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, draft_max, eos,
+                        g_session, first, remaining, eos,
                         accepted, cap, err, sizeof(err));
                     if (n < 0) { rc = -1; break; }
                     bool stop = false;

backend/go/vllm-cpp/Makefile

@@ -11,7 +11,7 @@ JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || e
 
 # vllm.cpp version
 VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
-VLLM_CPP_VERSION?=839ea1ceddb787778b6bd86a38a917a1aab74d8f
+VLLM_CPP_VERSION?=6bf3abb580982f4fd2e4525ef37802ee0ce28981
 
 # MLX GEMM provider (darwin/metal only; see the metal branch below for why).
 # Consumed as the prebuilt pip wheel: building MLX from source needs `xcrun

core/application/distributed.go

@@ -15,6 +15,7 @@ import (
 	"github.com/mudler/LocalAI/core/services/distributed"
 	"github.com/mudler/LocalAI/core/services/jobs"
 	"github.com/mudler/LocalAI/core/services/messaging"
+	"github.com/mudler/LocalAI/core/services/monitoring"
 	"github.com/mudler/LocalAI/core/services/nodes"
 	"github.com/mudler/LocalAI/core/services/nodes/prefixcache"
 	"github.com/mudler/LocalAI/core/services/storage"
@@ -162,6 +163,17 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade
 	}
 	xlog.Info("Node registry initialized")
 
+	// Bound durable heartbeat writes: a beat that only carries a fresher
+	// timestamp is what turned backend_nodes into a 460 MB six-row table.
+	registry.SetHeartbeatCheckpoint(cfg.Distributed.NodeHeartbeatCheckpointOrDefault())
+
+	// Measure the vacuum horizon. The 42 days it stayed open went unnoticed
+	// because no gauge reported it until models started failing to load.
+	if err := monitoring.RegisterControlPlaneDBMetrics(authDB, 30*time.Second); err != nil {
+		// Metrics are diagnostic; a failure here must not stop the frontend.
+		xlog.Warn("Control-plane database metrics unavailable", "error", err)
+	}
+
 	// Let scheduling rules be keyed by a model alias. The registry resolves a
 	// rule's name through the config loader to find the model it governs, so an
 	// operator can pin placement to a stable name like "production" and have it

core/cli/run.go

@@ -182,6 +182,8 @@ type RunCMD struct {
 	BackendUpgradeTimeout        string `env:"LOCALAI_NATS_BACKEND_UPGRADE_TIMEOUT" help:"NATS round-trip timeout for backend.upgrade requests (default 15m)." group:"distributed"`
 	ModelLoadTimeout             string `env:"LOCALAI_NATS_MODEL_LOAD_TIMEOUT" help:"Fixed gRPC deadline for the remote LoadModel call sent to a worker node once its backend is installed and model files are staged. Unset (the default), the deadline is derived from the checkpoint size instead: 5m plus 20s per GiB, capped at 6h, so multi-tens-of-GB diffusion/video checkpoints get the minutes they need without a fixed cliff. Set this only to pin a specific budget; the value is used verbatim, including when it is shorter than the derived one." group:"distributed"`
 	ModelLoadWait                string `env:"LOCALAI_MODEL_LOAD_WAIT" help:"How long an inference request waits for a model that is still cold-loading onto a worker before it is answered with 503, a Retry-After header and live staging progress (default 60s). The request is served the moment the model becomes ready, so a model already most of the way staged needs no client retry. Set to 0 to wait as long as the load takes — only safe when no ingress or load balancer with an idle timeout sits in front." group:"distributed"`
+	StaleNodeThreshold           string `env:"LOCALAI_STALE_NODE_THRESHOLD" help:"How long a worker node may go without a durable heartbeat before the health monitor marks it offline (default 5m). Because a beat that only carries a fresher timestamp is held back by --node-heartbeat-checkpoint, this must stay comfortably wider than that interval; raise both together. Dead-node detection through the per-model gRPC health check and through request-time failure is unaffected by this knob." group:"distributed"`
+	NodeHeartbeatCheckpoint      string `env:"LOCALAI_NODE_HEARTBEAT_CHECKPOINT" help:"Minimum gap between durable heartbeat writes for a worker node (default 60s). A beat that only carries a fresher timestamp is dropped until this interval elapses; every field is compared against the value last written, so a node's first beat, a changed total VRAM/total disk/GPU vendor, and a free VRAM/RAM/disk reading that has moved more than 256 MiB from the written value all still write immediately, and a node that is not active is never suppressed. Set below the worker heartbeat interval to write on every beat." group:"distributed"`
 	NatsAccountSeed              string `env:"LOCALAI_NATS_ACCOUNT_SEED" help:"NATS account signing seed (SU...) used to mint per-node worker JWTs at registration" group:"distributed"`
 	NatsServiceJWT               string `env:"LOCALAI_NATS_SERVICE_JWT" help:"NATS user JWT for the frontend (and agent workers) to publish control-plane messages" group:"distributed"`
 	NatsServiceSeed              string `env:"LOCALAI_NATS_SERVICE_SEED" help:"NATS user signing seed (SU...) paired with LOCALAI_NATS_SERVICE_JWT" group:"distributed"`
@@ -397,6 +399,20 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 		}
 		opts = append(opts, config.WithModelLoadWait(d))
 	}
+	if r.StaleNodeThreshold != "" {
+		d, err := parseDistributedDuration("LOCALAI_STALE_NODE_THRESHOLD", r.StaleNodeThreshold)
+		if err != nil {
+			return err
+		}
+		opts = append(opts, config.WithStaleNodeThreshold(d))
+	}
+	if r.NodeHeartbeatCheckpoint != "" {
+		d, err := parseDistributedDuration("LOCALAI_NODE_HEARTBEAT_CHECKPOINT", r.NodeHeartbeatCheckpoint)
+		if err != nil {
+			return err
+		}
+		opts = append(opts, config.WithNodeHeartbeatCheckpoint(d))
+	}
 	if r.RegistrationToken != "" {
 		opts = append(opts, config.WithRegistrationToken(r.RegistrationToken))
 	}