feat(serve): add GET /v1/load for load-aware routing

[ashalliants]

Adds a cheap, pollable GET /v1/load endpoint to ninfer-serve, so a load balancer or gateway that schedules across several servers can see engine-true load instead of guessing from its own bookkeeping.

What it reports

• capacity: fixed once the Engine is ready.
  • max_concurrency, max_pending_requests, max_admitted_requests, max_context
  • resolved page-aligned Main KV capacity (tokens, pages, page size)
  • Device/Host StateImage slots, Host KV bytes
• requests: admitted (holding ingress capacity), running, prefilling, decode_ready, waiting (in the Engine FIFO, including requests held back for KV entitlement), materializing.
• occupancy: Main KV pages/tokens in use, StateImage slots, Host KV bytes.
• counters: monotonic since startup, so callers derive tok/s by differencing two polls. computed_prefill_tokens, committed_decode_tokens, reused_prompt_tokens, decode_rounds, decode_row_rounds.
• uptime_seconds: counted from when the server became ready.

Auth and readiness work like every other route: the endpoint needs the API key when one is configured, and returns 503 model_loading until warmup completes. The response is sent with Cache-Control: no-store. The full contract is in docs/serving.md under Load.

Design notes

• No Engine execution lock on the poll path. memory_summary() takes the execution lock, so capacity facts are captured once in attach(), before ready_ is published. Each poll only reads the already-published runtime_stats() snapshot and the ingress counter (GenerationService::admitted_requests(), a small mutex on the request-capacity struct). Polling therefore can't stall decode.
• Freshness. Gauges come from the snapshot the Engine publishes at execution boundaries, so they can lag the poll by up to one boundary. This is documented.
• Occupancy caveat. Main KV occupancy includes retained reusable prefixes, which the planner may evict under pressure. A full pool doesn't by itself mean new requests will wait. This is documented too.
• Isolated rendering. The JSON lives in src/serve/load_report.{h,cpp}, separate from the HTTP handler, so it's unit-testable without a server.

Motivation

This is used by a gateway that fronts several engines with per-user parallel limits and a fair queue. It polls /v1/load every 2 s:

• to hold back dispatch to an engine that has a backlog;
• to warn when the gateway's configured slots differ from --max-concurrency;
• to chart engine-true prefill/decode throughput.

The endpoint is purely additive and opt-in by use. Nothing in the generation path changes.

Verification

• tests/test_load_report.cpp (ninfer_load_report_test) covers:
  • capacity derivation from EngineOptions + MemorySummary;
  • JSON shape and field names;
  • counters passed through unchanged;
  • max_admitted_requests = concurrency + pending.
• tests/test_load_report.cpp also covers a zero-capacity report (kv_page_tokens = 0 with no pages).
• Before the rebase onto current master, this built and the serving tests passed in the worktree's build tree. The gateway's parser for this payload is tested separately, but it has not polled a live ninfer-serve yet.
• The rebase onto master (merge: catch up with neroued/master (materialization search rework), adapted for MSVC #94–Port upstream #197 (ignore_eos) and #221 (MTP graph topology class) #101) applied without conflicts. The post-rebase rebuild was still running when this was pushed, so CI or a local ctest -R "load_report|openai|anthropic|serve_options|request_log" should confirm it.
• Not done: a GPU run on the 3090 since the rebase.

🤖 Generated with Claude Code

https://claude.ai/code/session_0152Jc7HkHQ9Ww56WXfM4j8u

[codepulse-review]

✅ No action needed — ship it.

Correctness: No actionable defects found. Readability/Architecture: Rendering is isolated and capacity is published before readiness. Security: Existing authentication and readiness middleware protect the route. Performance: Polling avoids the execution lock. Verification: CI is green; focused tests cover report shape, capacities, counters, and zero-page handling. Project execution was not performed per the static-review constraint.

---

_{Reviewed by CodePulse · AI C++ review with a senior engineer's eye · codepulse.review →}