Design — OTel Metrics Export from Valkey Admin Collectors

[ArgusLi]

Problem Statement

Valkey Admin collects per-node metrics but keeps them to itself. Data lands in local NDJSON files inside each collector process and is readable only through that collector's HTTP API, which the Valkey Admin server consumes to draw the dashboard. There is no way to get this data into Prometheus or any other monitoring backend.

Operators already running monitoring backends for their systems want Valkey metrics beside the rest of their infrastructure for long-term retention, alerting, and correlation with application metrics. Today they must run a second agent (valkey-exporter or similar) alongside Valkey Admin, duplicating collection.

Goal

Establish an OTLP export path from each metrics collector, and ship a tested OpenTelemetry Collector configuration that fans it out to Prometheus and any other backend the operator adds.

Requirements

Functional

1. Each metrics collector exports the metrics it gathers over OTLP.
2. Metrics carry per-node identity so a cluster's nodes are distinguishable.
3. Observability backends are reachable by configuration, not by code changes.
4. Export is off by default and enabled by configuration.
5. Works in all deployment modes: Kubernetes, Web, Electron.
6. Exported node coverage follows live cluster topology.

Non-functional

1. Minimal additional load on Valkey and Valkey Admin.
2. No new inbound network surface on the metrics collector.
3. No metric may carry a key name or other unbounded label.
4. Export cadence independent of collection cadence.
5. Operators can modify the Collector config without rebuilding an image.

Out of scope

1. Command logs, MONITOR output, hot keys, and big keys are not exported (rationale below).
2. Traces and logs signals. Metrics only.

Background

Current collector architecture

One collector OS process per Valkey node. Connection ids are <host>-<port>-db<N> but metrics node ids are <host>-<port> because INFO, MEMORY STATS, MONITOR, and COMMANDLOG are server-global, not database-scoped.

In Electron, Web, and Docker modes, apps/server/src/metrics-orchestrator.ts spawns collectors via child_process.spawn(process.execPath, [metricsServerPath], { env }), passing Valkey connection details and a per-node HMAC key as environment variables. In Kubernetes nothing is spawned by Valkey Admin. Sidecars are on the Valkey cluster side. Either way the collector registers at POST /orchestrator/register and pings by default every 10s.

Each collector has two clients: a valkey-glide GlideClient or GlideClusterClient for polling, and a separate iovalkey client created per MONITOR run, because MONITOR requires a dedicated blocking connection.

Inside the collector, two flows are cleanly separated:

write:  fetchers.js  →  collector-rx.js (timer + exhaustMap + bufferTime)  →  ndjson-writer.js

read:   HTTP request  →  ndjson-streamer.js + analyzers/*  →  JSON response

The analyzers are on the read path only. cpuFold and memoryFold are {filterFn, reducer, seed, finalize} bundles handed to streamNdjson at request time.

What is actually polled

Epic	poll_ms	Valkey Command
memory	5000	INFO MEMORY + INFO KEYSPACE
cpu	5000	INFO CPU
commandlog_slow	10000	COMMANDLOG GET 50 slow
commandlog_large_reply	10000	COMMANDLOG GET 50 large-reply
commandlog_large_request	10000	COMMANDLOG GET 50 large-request
monitor	10000	MONITOR for monitoringDuration, capped at maxCommandsPerRun

Everything else runs only on request:

• /info's full INFO
• MEMORY STATS
• /big-keys' SCAN loop
• /hot-keys
• CLUSTER SLOT-STATS

Findings that shaped this design

Why raw CPU counters rather than the derived value.

An OTLP sink tapping the write path sees rows before any analyzer. Exporting it would require either re-reading NDJSON on every export tick or implementing counter-diffing again.

fetchers.js already passes through every numeric field from INFO CPU, so all six counters are present in the rows today. No fetcher change is needed.

Why export cadence must be independent.

startCollector polls at poll_ms (default 5s) then bufferTime(batch_ms) (default 60s) before writing. Feeding that buffered batch to an exporter would push multiple backdated points every tick. OTLP handles this poorly and Prometheus remote-write may reject it as out-of-order.

Why exported primary node coverage is already covered.

Valkey Admin already derives its collector set from the cluster rather than from configuration. discoverCluster populates clusterNodesRegistry with each primary and its replicas, and runReconcileLoop diffs that against metricsServerMap every 10 seconds, spawning and stopping collectors through findDiff. Export inherits this at no cost: a new shard's metrics appear without anyone editing a file, and a departed node's collector is torn down.

This constrains identity. Because collectors are transient, every identity attribute must derive from the Valkey node rather than the collector process, hence, service.instance.id is sanitizeUrl("-") of the node, never a PID or the collector's ephemeral port. Otherwise each reconcile would mint fresh series for nodes that never changed.

Note: In Kubernetes, replicas are covered. In all other deployment modes, only primaries are covered.

Cluster identity requires the one change outside the metrics collector.

clusterNodesRegistry is keyed by clusterId and discoverCluster returns it, but startMetricsServer does not pass clusterId to the child. A collector therefore cannot currently know which cluster it belongs to, and every node of every cluster would export identical service.name with nothing cluster-scoped.

Fixing this requires adding the cluster id to the spawn env in metrics-orchestrator.ts, plus setting it on the sidecar in the Kubernetes manifests where nothing is spawned. Everything else in this design is confined to the metrics collector, so this is the only orchestrator-side change.

What Gets Exposed, and Why

Selection criteria

This design exposes memory and CPU because:

• they are the two saturation signals for a Valkey node
• they are already collected
• they map unambiguously onto OTel instrument types

Memory — eleven gauges

Point-in-time levels, so Gauge in every case.

OTel name	Unit	Source INFO field
valkey.memory.used	By	used_memory
valkey.memory.peak	By	used_memory_peak
valkey.memory.dataset.size	By	used_memory_dataset
valkey.memory.overhead	By	used_memory_overhead
valkey.memory.allocator.active	By	allocator_active
valkey.memory.allocator.resident	By	allocator_resident
valkey.memory.fragmentation.size	By	mem_fragmentation_bytes
valkey.memory.fragmentation.ratio	1	mem_fragmentation_ratio
valkey.memory.allocator.rss.ratio	1	allocator_rss_ratio
valkey.memory.dataset.utilization	1	used_memory_dataset_perc
valkey.keyspace.keys	{key}	keys= per database from INFO KEYSPACE
valkey.memory.limit	By	maxmemory, omitted when 0 (unlimited)

valkey.keyspace.keys is the one metric in this section that carries an attribute:

Attribute	Values
db.namespace	the database index as a string — "0", "1", …

Note: Cardinality is bounded by static config, not by data. Series count per node is the number of non-empty databases, capped by the server's databases setting (while no actual limit, keys written in DB >63 will leak). Per-node totals are 10–11 unattributed memory gauges (valkey.memory.limit is absent on nodes with no maxmemory) plus 1–64 keyspace series plus 6 CPU series, so 17 to 81 series per node.

The flat keys_count row that memory-metrics.js reads stays exactly as it is, still scoped to db0. Changing it to a cross-database sum would alter a number already displayed on the dashboard, which is a separate UI decision and out of scope here.

CPU — one counter with one attribute

OTel name	Instrument	Unit
valkey.cpu.time	Observable Counter	s

Attribute	Values
state	sys, user, sys_children, user_children, sys_main_thread, user_main_thread

Six series, one per INFO CPU field.

Note: The six values overlap. used_cpu_sys_main_thread is a subset of used_cpu_sys, because the main thread is one of the process's threads. *_children counts separate forked processes via a different getrusage target. A single attribute with compound values a state×scope grid would present the two as orthogonal dimensions and invite sum by (state), which silently double-counts the main thread.

This is also why the design does not use system.cpu.time's single-word state. That convention's values are disjoint, and the OTel instrument-naming guidance states that for time instruments "the limit can usually be calculated as the sum of time over all attribute values." Valkey's counters do not satisfy that invariant, so adopting the shape would advertise a guarantee that does not hold.

Resource attributes

Attribute	Value
service.name	valkey
service.namespace	the clusterId
service.instance.id	ownNodeId, i.e. sanitizeUrl("<host>-<port>")
db.system.name	valkey
server.address	VALKEY_HOST
server.port	VALKEY_PORT
k8s.pod.name, k8s.namespace.name	from downward-API env vars when present

What is deliberately excluded

Command logs. commandlog_* rows are {ts, metric, values: [...]} — log records, not metric points. They belong to the OTel logs signal, or to a later phase reducing them to counts and max-duration gauges.

MONITOR and hot keys. Key names as labels is unbounded cardinality and would degrade any TSDB. This is a permanent exclusion, not a phasing decision.

Big keys. An on-demand SCAN with MEMORY USAGE per key. Too expensive to run on an export interval.

The derived CPU percentage. Superseded by the raw counters.

Proposed Design

Data flow

flowchart LR
  subgraph vk["Valkey node (xN)"]
    V[(Valkey)]
  end

  subgraph node["Valkey Admin"]
    subgraph coll["metrics collector (xN)"]
      F[fetchers.js<br/>INFO MEMORY/KEYSPACE/CPU]
      LKV[last-known-value cache]
      ND[ndjson-writer]
      MP[MeterProvider<br/>observable instruments]
      EX[OTLP HTTP exporter]
    end
  end

  OC["OTel Collector<br/>(gateway)"]
  P[(Prometheus)]
  B[other backends]

  V ~~~ F
  F -->|poll 5s| V
  F --> ND
  F --> LKV
  LKV -->|read at export tick| MP
  MP --> EX
  EX -->|OTLP push| OC
  OC -->|:8889/metrics| P
  OC -.->|operator-added exporters| B

Loading

Where the collector actually runs differs by mode, and the diagram shows the majority case. In Web and Electron the server spawns it as a local child process that reaches Valkey over the network, so it belongs to the Valkey Admin side. Kubernetes is the exception where it is a sidecar container inside the Valkey pod, sharing that pod's network namespace.

Metrics Collector design

Within each metrics collector, fetchers poll the metrics collector's respective Valkey node without change, but pass the fetched metrics to an additional metrics exporter branch. There, a Mapping Registry maps selected metrics' Valkey names to {name, instrument, unit, attributes, transform}, dropping unknown names. The rows are then saved in a memory only cache, where they are read at export time. Entries older than a staleness bound are dropped, so a wedged fetcher yields gaps rather than a flat line. This decouples the fetching and exporting cadences and avoids backdated batches. The Meter Provider builds the Resource, registers the observable instruments, and attaches a PeriodicExportingMetricReader with an OTLPMetricExporter. The exporter pushes to the collector shared for the Valkey Admin instance.

The export pathway is gated by an environment variable. If it's disabled, none of the related OTel instrumentation will be utilized.

How the OTel Collector works

We will create and maintain an OTel Collector configuration that works with Valkey Admin. The operator will be responsible for running the OTel collector, processing telemetry through a three-stage pipeline:

• Receivers accept incoming data. Here, the otlp receiver listens on 4317 (gRPC) and 4318 (HTTP) for pushes from every collector process.
• Processors transform data.
• Exporters send data onward.

Why use the OTel Collector?

• OTel Collector allows for Fan-out. A single OTLP exporter in the metrics collector reaches exactly one endpoint. Having the OTel Collector allows for simultaneous delivery to two or more backends.
• The OTel Collector allows for retry and queueing across a backend outage, plus k8sattributes enrichment that would otherwise require giving every sidecar RBAC and Kubernetes API access.

Alternatives considered

Alternative	Why not
Prometheus /metrics on each collector	Ties the export path to a single backend. Every additional destination then becomes another exporter for this project to implement and maintain. OTLP plus a Collector makes a new backend an operator config edit instead. Coverage would also be partial: only the Kubernetes sidecar is scrapable, and in Web, Docker, and Electron the collectors are child processes on ephemeral ports inside the app container, unreachable by any scraper without changing METRICS_BIND_HOST from loopback and exposing every existing endpoint with it.
Push OTLP straight to Prometheus's OTLP receiver	Viable and simpler, but Prometheus-only: no fan-out, no retry, and it requires operators to configure promote_resource_attributes.
Aggregate in the Valkey Admin server	Existing endpoints return folded time series from NDJSON, not current values, so it needs a new endpoint on every collector in addition to scrape-time fan-out with its latency and partial-failure handling.
OTel Collector redis receiver	Configured statically, it requires one block per endpoint. This is as it has no topology awareness, so after a failover or reshard the config is stale until an operator notices and edits it.