Cluster Aggregate View — Technical Design Doc - WIP

[nassery318]

Cluster Aggregate View — Technical Design Doc - WIP

Addressing: #395
Consolidated cluster view merged into the Cluster Topology route

1. Overview

This document proposes a consolidated Cluster view for the Valkey Admin, merged into the existing Cluster Topology route. It adds aggregated, cluster-wide metrics (total memory against capacity, total throughput, cluster-wide hit ratio, and a count of flagged nodes) alongside a per-node breakdown table, so a user can go from "is anything wrong" to "which node, and why" in a single screen.

The reason for node based aggregation is because Valkey does not provide a dedicated command for retrieving cluster-wide metrics such as total memory/capacity, total throughput and so on. Therefore, these metrics must be collected from each node using INFO and aggregated to obtain the overall cluster-level values. The fan-out and the aggregation both happen on the server. GlideClusterClient.info() collects INFO from every node in one call, and the server derives per-node utilization from it before pushing a single payload to the client. Utilization is computed there rather than in the browser because CPU load is a sampled rate — it requires holding the previous reading between polls, which a component cannot do across a route change.

2. Motivation

Currently, clusters can only be assessed by opening individual nodes one at a time in the dashboard (/dashboard) view. There is no way to see cluster-wide totals or compare nodes side by side without manually cross-referencing each node's page. This makes three common diagnostic tasks slow:

• Memory pressure — spotting a node approaching its configured limit while others sit well below it.
• CPU saturation — catching a node whose command loop is running hot before it starts shedding latency.
• Hotspot detection — finding the one primary absorbing disproportionate ops/sec or connections.

A unified view collapses this from N node visits into one screen with searchable, filtered rows.

3. Goal

The goal of the cluster aggregate view is to:

• Show aggregated cluster metrics at the top of the page: total nodes, memory used/capacity, total ops/sec, cluster hit ratio, and count of flagged nodes.
• Show a per-node table below it with role, utilization, memory, cpu, ops/sec, hit ratio and connections — filtered and searchable.
• Visually flag nodes that deviate based on memory and cpu usage (Low / Normal / High) so attention goes to the right rows first.
• Merge this into the Cluster Topology route so topology and utilization are one screen, not two.

4. Current State

The existing Cluster Topology route shows summary counts — Total Nodes, Primary Nodes, Replicas, Connected — and a searchable list of primary/replica pairs by name, host, and port. Each row shows a primary node with its memory usage, cpu usage and connection count, and links to its paired replica.

This view answers “what is the shape of my cluster” but not “which part of my cluster is under pressure.”

There is no aggregate throughput or hit-ratio number, no per-node utilization signal, and no way to compare nodes against each other without opening each one.

5. Proposed Design

The Cluster metrics are added to the top of the Cluster Topology route as a row of stat cards, with the existing node list replaced by an expanded table that carries the new per-node metrics. The primary/replica relationship from Topology is preserved as the row grouping; utilization data is layered on top of it rather than replacing it.

• Stat cards row: Total Nodes, Cluster Memory, Total Ops/sec, Cluster Hit Ratio, Nodes Flagged.
• Search and filter bar: search by name/host/port, plus role filter (All roles) and utilization filter (All utilization). Because only primaries are classified, selecting any utilization level other than "All" also excludes every replica row.
• Per-node table: one row per node (primary and replica both listed, not nested), each primary carrying its own utilization badge and metrics.

Metrics and the utilization badge are rendered on primary rows only; replica rows carry name, host, port and role. The view refreshes on a five-second poll (CLUSTER_DATA_POLL_INTERVAL_MS), started when the route mounts and torn down when it unmounts.

6. INFO Metrics Used

Every number on the aggregate view comes from one INFO call to each node, repeated every five seconds. Nothing else is queried.

INFO returns two kinds of number, and we treat them differently:

• Ready to use — a current value or a rate, like memory in use or commands per second. These are shown as they arrive.
• Running totals — counters that only ever go up since the node started, like CPU time. A running total tells you what a node has done since boot, not what it is doing now. To get a current value we subtract the previous poll's reading and divide by the time between the two polls.

Because that subtraction needs the previous reading, running totals are converted once when the data is collected rather than in the browser, which would lose the previous reading whenever the user leaves the page.

6.1 Metrics used to calculate utilization

Metric	What it is	Why we use it
used_memory	Memory the node is currently using.	This is the number maxmemory is checked against, so the percentage matches what actually causes eviction.
maxmemory	The memory limit set on the node. Used as the denominator when set.	It is the limit the node will really evict at. Reports 0 when no limit is set, which is how we know to fall back to the next row.
total_system_memory	Total RAM on the host. Used as the denominator when maxmemory is 0.	Gives a percentage instead of a blank cell. It is a weaker signal than a real limit, so it is tracked separately and the badge shows it differently.
used_cpu_sys_main_thread, used_cpu_user_main_thread	CPU time used by the thread that runs commands.	Commands run on a single thread, so these cannot exceed one core. That makes the percentage mean the same thing on every machine, without needing to know how many cores it has.
used_cpu_sys, used_cpu_user	CPU time for the whole process. Used only when the two above are missing.	These include background and I/O threads, so the figure can go above 100% and will reach High sooner.

6.2 Metrics shown as reported

Metric	What it is	Why we use it
server_name	The node's name.	Used as the row label and for search. Falls back to host and port when missing.
used_memory_human	Memory in use, already formatted (e.g. 1.05M).	Valkey has done the formatting, so the row does not repeat it.
instantaneous_ops_per_sec	Commands per second.	Valkey already averages this over a short window, so it arrives as a rate and needs no further work — unlike the CPU counters above.
keyspace_hits, keyspace_misses	Cache hits and misses.	The two numbers behind the hit ratio. Both count from boot, so every hit ratio here is a lifetime average: it says whether the cache has been working well, not whether it is working well right now.
connected_clients	Number of connected clients.	A spike here often explains why the same row is high on ops/sec or memory.

Cluster totals are sums, not averages of percentages. Hits and misses from every node are added together before the ratio is worked out, so a busy node counts for more than a quiet one. Cluster Memory adds up used bytes and limits separately.

7. UI Component Breakdown

7.1 Stat cards

Metric	What it shows	Why the user needs it
Total Nodes	Count of all primaries + replicas in the cluster.	Confirms topology size at a glance; baseline for the per-node table below.
Cluster Memory	Sum of used memory across all nodes against the sum of their reported limits (e.g. 4.29 GB / 6.40 GB). Where no node reports a limit the capacity half shows ∞.	Surfaces cluster-wide memory pressure before any single node hits its limit.
Total Ops/sec	Sum of instantaneous_ops_per_sec across all nodes, replicas included.	Gives a single read on cluster load.
Cluster Hit Ratio	Aggregate cache hit ratio across all nodes.	One number for “is the cache doing its job” — drop here means investigate eviction or key sizing.
Nodes Flagged	Count of nodes in the High utilization state.	Directs the user straight to the rows that need attention instead of scanning all nodes.

7.2 Per-node row fields

Metric columns are populated on primary rows only. Replica rows show name, host, port and role, and leave the remaining cells empty.

Field	Example	Interpretation
Role badge	PRIMARY / REPLICA	Carried over from the Topology view; establishes which rows are comparable (primaries to primaries).
utilization badge	Low / Normal / High	Colored as a severity ramp — muted gray (Low), green (Normal), red (High) — so only red rows pull the eye. An empty cell means no measurement yet, not Low.
Memory	1.05M / 128 MB	Used memory against the node's reported limit — its maxmemory where configured, otherwise its host RAM, and ∞ where it reports neither. The used half is the figure Valkey has already formatted, so the two halves do not share a notation.
CPU	62.4%	Share of a single core consumed by the command thread since the previous poll. Empty on the first poll, when there is no prior sample to compare against.
Ops/sec	15.0K/s (primaries only)	Command throughput, used to spot hot shards.
Hit Ratio	97.65%	Per-node view of cache effectiveness.
Conns	46	Connection concentration; a spike here can explain elevated ops or memory on that node.
Actions	-	Connect, open dashboard, open command console for that node.

8. Utilization Classification

Each primary is classified Low, Normal, or High. The badge is the primary scanning aid in the table — it lets a user with a cluster of 50 nodes find the handful that matter without reading every row. Low means the node is lightly loaded, not that it needs attention — in a cache, low utilization is neutral, so it is rendered in muted gray, Normal in green, and only High in red.

The classification takes two inputs, memory utilization and CPU utilization, each judged against its own band: a cache is expected to sit near its memory limit, so memory tolerates higher percentages than CPU, where sustained load near one core means the command loop is saturating. The worse of the two results becomes the node's level, so a node at 30% memory and 90% CPU is High.

State	Working definition	Memory thresholds	CPU thresholds
Low	Node is lightly loaded on both watched metrics.	Below 70%	Below 60%
Normal	Node is working but has headroom.	70% to 90%	60% to 85%
High	Node is near capacity on memory, CPU, or both, and is the first place to look.	90% and above	85% and above

• Memory basis — Memory is measured against the node's maxmemory, or against its total system memory where none is configured — in that case the badge is drawn with a dashed border and its tooltip reads "of host RAM — no maxmemory set." Where a node reports neither, no percentage is produced.
• CPU derivation — INFO only exposes cumulative CPU counters, so the server diffs them against elapsed wall time on each poll and reports the result as a percentage of one core. This gives a live reading rather than a lifetime average, but needs two samples: no figure is produced on a node's first poll, or when the prior sample is stale, the counters have gone backwards, or two samples share a timestamp.
• Which CPU counters — The server prefers used_cpu_sys_main_thread and used_cpu_user_main_thread. Command execution is single threaded, so these are bounded by one core and the percentage reads directly against the 85% CPU band. It falls back to the process-wide used_cpu_sys and used_cpu_user on servers that do not report the main-thread counters; those totals include I/O and background threads, so the reading can exceed 100% and will flag High more readily than the main-thread path.
• Thresholds are absolute — Nodes are judged against fixed percentages, not against each other, so a uniformly loaded cluster shows a uniform badge and a node absorbing three times its peers' traffic goes unflagged until it crosses the band. Peer-relative classification is deferred to Future Work.

9. Edge Cases

• Disconnected node: appears in the table but with stale or missing live metrics — needs a distinct visual state, not just “High.”
• Partial data: if one node's stats fail to load, aggregate cards should not silently drop that node from the average without indicating it.

[bblan0803]

Thanks for sharing! couple of user experience things:

1. Nodes Flagged includes Low, so it's easy to read that number as something being wrong when some of them are just idle. Should Low sit somewhere else?
2. Is the table sorted by badge by default? On 50 nodes I'd want the High ones at the top without having to sort.
3. Does the table still fit at 1024 with the extra columns?
4. When the badge says High, can it say which one tripped it? Otherwise I'm still scanning both columns.
5. Is 85% the right memory band for a cache?
6. Can a High node link into Big Keys or Hot Keys for that node? Otherwise I see which node is bad and then have to go and start a scan myself.

[ravjotbrar]

Let's include how you plan on getting these metrics in the design.

[sarthakaggarwal97]

Thanks for the proposal, I think this is a useful idea. Sharing some comments that may be helpful.

• One thing that I am not sure of: are we collecting metrics from all nodes or only primaries? The doc says all nodes, but if I remember correctly GLIDE’s default INFO call only goes to primaries. Please do check this.

• I’m not sure if we should use total_system_memory as the fallback capacity. Would we count same memory multiple times If multiple nodes are on the same host?.

• How will we show partial results? If one node does not respond, I’d rather see something like “5/6 primaries reporting” than a total that looks complete.

• We should make sure a new INFO request doesn’t start while the previous one is still running. This could get expensive on a large cluster, especially with multiple users viewing the page.

• What happens if a failover occurs while this page is open? Is the page automatically refreshed?.

[nassery318]

Hi @sarthakaggarwal97 ,

Appreciate you taking the time to review the doc and add your feedback. Below are the points raised and how they are currently handled and will be handled as per the recommendations.

• You are right on the GLIDE's info() defaulting to all primaries, not all nodes. That's what we're actually collecting. I'll fix the doc to explicitly mention primaries.
• Good point, two nodes on the same host would count that RAM twice. I will drop total_system_memory from the cluster total: capacity sums only nodes with a real maxmemory, so in the stat card we will have only used memory. The primary Per-node rows will still show the host-RAM percentage, but labelled as such and excluded from the High badge.
• The total nodes in the stat card is combination of both primaries and replicas, reflecting the displayed rows. I agree on showing the partial results or declaring that the results are partial, I believe this one needs a bit of thinking in terms of UI/UX on how to show the partial result stat card and whether or not another stat card specifically for this case - to show if all primaries are connected and their INFO is captured.
• Agreed, the page polls on a fixed 5s timer that fires whether or not the previous response returned, and each viewer drives its own fan-out. So will add a guard for this and it will be one INFO per cluster at a time, its result shared across everyone viewing.
• Today the 5 second poll feeds the stat cards and the node row numbers, but the role labels come from connect time, so a failover leaves them stale - the demoted node keeps its PRIMARY label and the promoted one stays a REPLICA, which also hides its metrics and keeps it out of the "Nodes Flagged" count. Taking the role from the same 5 second INFO keeps it in sync.