Skip to content

Improve runtime usage logging on memory recall quota path #371

Description

@mashenjun

Problem

During the 2026-06-26/27 production investigation, GET /v1alpha2/mem9s/memories returned many HTTP 503 responses with body runtime usage unavailable.

However, mem9-server does not consistently log the underlying runtime usage error on this path.

Current behavior:

  1. BeforeRecall -> reserve failure returns 503 via handleRuntimeUsageError, but does not log the underlying error.
  2. Loki mostly only has access logs like handle request done status=503.
  3. Detailed runtime usage logs exist only for some post-reservation paths, e.g. runtime usage release failed.
  4. This made it hard to distinguish console-server timeout, quota service 5xx, network error, or other runtime usage failures.

Relevant code paths

  1. server/internal/handler/memory.go
    • BeforeRecall error returns directly through handleRuntimeUsageError.
  2. server/internal/runtimeusage/manager.go
    • reserve returns non-fail-open errors without logging.
    • release logs runtime usage release failed.
  3. server/internal/handler/runtime_usage.go
    • handleRuntimeUsageError maps errors to HTTP responses but does not log context.

Proposed improvement

Add structured logs for runtime usage failures before returning client errors.

Suggested fields:

  1. request_id
  2. operation_id
  3. tenant_id
  4. cluster_id
  5. agent_name
  6. meter
  7. runtime_usage_phase: reserve, commit, release
  8. http_route
  9. err

Do not log raw API keys.

Acceptance criteria

  1. Reserve failures on recall emit a warning/error log with keyword runtime usage reserve failed.
  2. Runtime usage errors returned as HTTP 503 can be correlated by request_id.
  3. Logs clearly distinguish:
    • reserve failure
    • commit/finalize failure
    • release failure
    • outbox failure
  4. No sensitive API key material is logged.
  5. Add focused tests or log-observation coverage where practical.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions