Problem
During the 2026-06-26/27 production investigation, GET /v1alpha2/mem9s/memories returned many HTTP 503 responses with body runtime usage unavailable.
However, mem9-server does not consistently log the underlying runtime usage error on this path.
Current behavior:
BeforeRecall -> reserve failure returns 503 via handleRuntimeUsageError, but does not log the underlying error.
- Loki mostly only has access logs like
handle request done status=503.
- Detailed runtime usage logs exist only for some post-reservation paths, e.g.
runtime usage release failed.
- This made it hard to distinguish console-server timeout, quota service 5xx, network error, or other runtime usage failures.
Relevant code paths
server/internal/handler/memory.go
BeforeRecall error returns directly through handleRuntimeUsageError.
server/internal/runtimeusage/manager.go
reserve returns non-fail-open errors without logging.
release logs runtime usage release failed.
server/internal/handler/runtime_usage.go
handleRuntimeUsageError maps errors to HTTP responses but does not log context.
Proposed improvement
Add structured logs for runtime usage failures before returning client errors.
Suggested fields:
request_id
operation_id
tenant_id
cluster_id
agent_name
meter
runtime_usage_phase: reserve, commit, release
http_route
err
Do not log raw API keys.
Acceptance criteria
- Reserve failures on recall emit a warning/error log with keyword
runtime usage reserve failed.
- Runtime usage errors returned as HTTP 503 can be correlated by
request_id.
- Logs clearly distinguish:
- reserve failure
- commit/finalize failure
- release failure
- outbox failure
- No sensitive API key material is logged.
- Add focused tests or log-observation coverage where practical.
Problem
During the 2026-06-26/27 production investigation,
GET /v1alpha2/mem9s/memoriesreturned many HTTP 503 responses with bodyruntime usage unavailable.However, mem9-server does not consistently log the underlying runtime usage error on this path.
Current behavior:
BeforeRecall -> reservefailure returns 503 viahandleRuntimeUsageError, but does not log the underlying error.handle request done status=503.runtime usage release failed.Relevant code paths
server/internal/handler/memory.goBeforeRecallerror returns directly throughhandleRuntimeUsageError.server/internal/runtimeusage/manager.goreservereturns non-fail-open errors without logging.releaselogsruntime usage release failed.server/internal/handler/runtime_usage.gohandleRuntimeUsageErrormaps errors to HTTP responses but does not log context.Proposed improvement
Add structured logs for runtime usage failures before returning client errors.
Suggested fields:
request_idoperation_idtenant_idcluster_idagent_namemeterruntime_usage_phase:reserve,commit,releasehttp_routeerrDo not log raw API keys.
Acceptance criteria
runtime usage reserve failed.request_id.