Cluster Hot Slots Heatmap — Technical Design Doc - Part 2

[nassery318]

Addressing: #470
Add cluster slot heatmap visualization

Part 1: #498

Overview

This document proposes the second part of the slot-level heatmap in the Hot Keys tab of the Activity view, on clusters running the hot-slots collection mode. Part 1 answers "where did my hot keys land." Part 2 answers "which slots are busy right now, how much of its node's work is one slot doing, and is the hot region a single slot or a run of neighbours" — the last of these matters because the two shapes call for opposite fixes.

Motivation

The original issue asks for three things: a heatmap of slot ranges coloured by CPU time, available only where slot stats are enabled, with drill-down from a hot slot to the keys inside it.

Part 1 delivered the drill-down, meets the availability condition by construction, and built the substrate the first item needs — the slot model and the entire component tree. What it does not do is colour by CPU or draw ranges. Three consequences make that worth extending Part 1:

• The colour repeats the selection criterion. CPU already decided which slots were looked at, so shading the survivors by how many of their keys cleared a frequency cutoff describes the sampling, not the cluster.
• A slot can be hot and still invisible. Where a slot's load is spread over thousands of keys — the case resharding actually fixes — no single key survives the cluster-wide top list, so the slot never gets a tile.
• The grid cannot tell a hot slot from a hot range. Tiles are ordered by key count, so neighbouring slots are scattered across the grid. Yet a single hot slot among cold neighbours means a hash tag has pulled a working set into one place, which resharding will not fix, while a hot run of neighbours is a range imbalance, which it will.

Goal

The goal of Part 2 is to:

• Colour slots by how much CPU they are consuming now, a figure comparable from one refresh to the next, rather than by a count that describes the sampling.
• Report each slot's share of its owning node's load, so "hot" is measured against something.
• Fold neighbouring hot slots into ranges and name the shape, so the view states the diagnosis rather than leaving it to be read off the grid.
• Show slots that are consuming CPU even when they hold none of the top keys, and say what that means.
• Reuse Part 1's components, data path, and visual language as they stand — no new collection machinery, no stored history, no new commands against Valkey.

Current State — What Part 1 Ships

Hot-slots collection asks Valkey which slots are consuming the most CPU (CLUSTER SLOT-STATS ORDERBY CPU-USEC), scans the keys inside each one, and ranks them by access frequency. Part 1 kept the slot attached through that path and built the view on top of it:

• Data model. Each key keeps its slot from the moment it is collected, through the step that adds size and TTL, to the aggregation that stamps on the reporting node. The browser receives every key with its slot, node, size, and TTL alongside it.
• Components. The heatmap opens as a modal with two tabs, Nodes and Slots, the second appearing only when slot-carrying keys came back. The Slots tab groups the key list by slot and renders summary chips, a tile grid, and a drill-down panel. The colour ramp, band snapping, and legend are shared with the node heatmap, the legend doubles as a multi-select filter, and a dropdown scopes the grid to one node.
• What is not carried. CPU time, the number of keys scanned, and summed frequency never leave the collector. Slots exist in the browser only as groupings of the returned key list.

Proposed Design

Stage 1 — Carry CPU time through to the UI

CPU time per slot is already fetched during collection and then dropped. Carrying it forward makes a slot a record in its own right rather than a grouping of keys: the collector returns per-slot summaries — slot, CPU time, keys scanned — as a second channel alongside the ranked key list. Each slot has exactly one owning primary, so per-node summaries concatenate without merging and without any risk of double-counting; aggregation only stamps the owner on.

The two channels differ in size on purpose. The key list is cut to a cluster-wide top N; the slot summaries are not. That difference is surfaced rather than hidden, and it is what lets a slot appear on the grid with no keys under it.

Stage 2 — Turn a counter into a rate

The CPU figure Valkey reports is cumulative: it accumulates for the life of the server and resets only when stats are reset. Colouring by it directly would answer "busy ever", not "busy now" — a confident-looking picture of the wrong number.

Turning it into a rate needs two samples, not a collection pipeline. The metrics process for a node is long-lived, so it holds the previous reply and the time it was taken, and the second refresh yields both figures the view needs:

• Rate — the change in a slot's CPU time divided by the seconds between the two samples. Comparable from one refresh to the next.
• Share — a slot's rate against the summed rate of its node's slots. Comparable across nodes.

The guards are the ones the node CPU view already applies: nothing is reported on a slot's first sample, nothing when no time has passed, and a negative change is discarded as a restart or a stats reset.

Two details make the differencing work. Ask for a wider set of slots than is scanned — requesting slot stats is a single cheap command, while scanning keys and reading their frequencies is the expensive part, so widening the stats request raises the chance a slot appears in both samples at almost no cost. And report the window the rate was measured over, since refreshes are user-driven and a rate averaged across ten minutes must not be read as an instantaneous one.

Stage 3 — Classify the shape

Rate and share say how hot a slot is. Whether its neighbours are hot too says what to do about it. Adjacent hot slots are folded into ranges before rendering, and the range is what the view reports:

Observed shape	Diagnosis	Remediation
One hot slot, neighbours cold	Hash-tag collapse — one tag has pulled a working set into a single slot	Resharding will not help; the key design has to change
Hot run of neighbouring slots	Genuine range imbalance	Reshard — the run's slot numbers are the arguments
Hot node, load even across its slots	Not a slot problem	Add capacity

This is what "slot ranges" in the issue points at, and it is the reading an operator performs by eye today. Stating it is the difference between a picture and a tool.

Stage 4 — Fix the scale

A handful of slots take almost all the traffic, so on a straight linear scale one extreme slot consumes nearly the whole colour range and flattens everything else into the palest band, hiding the second and third hotspots. The ramp becomes logarithmic or rank-based, with the top anchored at a high percentile and the number of slots above that anchor reported, so the colour is spent where the structure actually is.

This also closes a fault Part 1 records as a known deviation. Where every sampled slot holds the same count — including where only one slot came back — every tile currently renders at the deepest band, so a flat distribution reads as a cluster-wide emergency. The scale helper shared with the node heatmap already handles that case correctly; the slot grid's own copy does not. Using the shared one is the fix.

Slot Data Used

Metrics that drive intensity

Metric	What it is	Why we use it
CPU rate	The change in a slot's CPU time between two samples, over the seconds between them.	A real measure of resource use rather than a side effect of the eviction policy, and the metric the issue names.
Share of node	A slot's rate against the summed rate of its node's sampled slots.	Makes "hot" relative to something. A slot at 40% of its node is actionable; a raw CPU figure is not.
Range length	How many neighbouring hot slots the slot belongs to.	The input to the shape classification, and the difference between "reshard" and "redesign your keys".

Fields shown as reported

Field	What it is	Why we use it
Slot number / range	The slot, 0–16383, or the run it belongs to.	The tile label, and what an operator carries into a reshard or migration command.
Owning node	The primary that owns the slot.	Connects the slot back to the node heatmap and to physical remediation.
Hot key count	Keys from the cluster-wide top list falling in this slot.	Demoted from the intensity signal to a reported field. Zero is meaningful — see Edge Cases.
Keys scanned	Keys examined in the slot.	Gives the hot key count a denominator: 40 hot of 50 scanned is a different finding from 40 of 40,000.
Measurement window	The seconds between the two samples the rate came from.	Keeps a long-window average from being read as instantaneous.
Per-key size, TTL	Already surfaced in the hot keys table.	Reused unchanged in the drill-down so the two views agree.

UI Component Breakdown

Summary chips

Part 1's chip row is retained in full — hot slots, hot keys, the node filter dropdown, and the count of nodes that failed to report, which keeps a shrunken total from being presented as complete. One chip is added: the share carried by the top slot, the figure that answers "is anything concentrated" before a single tile is read.

Slot tile and drill-down

Each tile shows its slot number and its share of its node, coloured by rate, with the full record on hover: rate, share, hot keys against keys scanned, owning node, and the measurement window. Ordering changes from hot key count to slot number, so neighbouring slots sit next to each other and a hot run reads as a block — that change is what makes the shape classification legible rather than merely computed, and a detected run is outlined as one region and labelled with its span.

Selecting a tile opens that slot's key list beside the grid exactly as in Part 1, with size and TTL per key, the owning node in the footer, and a click that closes the modal and selects the key in the key browser. Keys within a slot stay ordered by access frequency, a deliberate reversal of Part 1's "frequency is not shown in this view." The counter is too coarse to rank slots against each other, but ordering a handful of keys inside one slot is a much weaker claim, and the column is labelled approximate.

Slot Intensity Classification

Slots are shaded across the five bands of the shared ramp, driven by CPU rate, with the legend doubling as a multi-select filter as in Part 1.

Band	Working definition	Basis
Palest	Active, but a negligible share of its node's load.	Bottom of the scaled range
Low / Mid / High	Progressively larger share; the middle bands carry the shape of the distribution.	Interpolated
Deepest	Carries a disproportionate share of its node's load and is the first place to look.	Top of the scaled range

• Why CPU now, when Part 1 said not CPU. Part 1's objection was that CPU had already selected the slots. That holds for the cumulative figure and dissolves for the rate: selection is by lifetime total, colour is by current rate, and the two routinely disagree. A slot that was hot for a week and is idle now is exactly the case Part 1's colour could not distinguish and this one can.
• The scale is logarithmic or rank-based, not linear — see Stage 4. One sequential ramp, ordered by lightness; magnitude is never encoded as hue, which stays reserved for categorical distinctions such as node identity, drawn as an outline rather than a fill.
• Bands are relative to the current sample, so intensity is not comparable across refreshes. Absolute thresholds remain deferred.

Edge Cases

• First refresh after connecting: no earlier sample exists, so no rate can be produced. The view says it is waiting for a second sample rather than rendering zeros, which would read as an idle cluster.
• Stat reset: a restart or an explicit reset sends the counter backwards. The negative change is discarded and the slot resumes on the next refresh, as the node CPU view already does.
• Slot present in only one sample: possible after a reshard moves ownership, or when the set of slots returned shifts between refreshes. The slot is reported with its cumulative figures and no rate rather than being silently dropped.
• Hot slot with no top keys: the drill-down is empty, and that is a finding rather than a failure — the slot is consuming CPU with no individual key over the frequency threshold, meaning load spread across many keys, which is the shape resharding helps. Part 1 could not produce this case because its tiles were defined by the keys inside them.