Update client-side stats to use light weight Hashtable

[dougqh]

What Does This Do

Replaces the MetricKey based HashMap with a new AggregateTable based on the light Hashtable.

Motivation

By using the light Hashtable, I'm able to avoid the biggest source of allocation in client-side stats: MetricKey

Hashtable provides utilities for searching the entries without constructing a new composite key object.

First, the components are hashed together to find the corresponding bucket
Then the bucket can be traversed to see if the entries match the key components

And the custom Entry can hold multiple fields that comprise the data / metadata needed for metric aggregation and eviction policy

The end result is that both MetricKey and AggregateMetric can be merged into a single class AggregateEntry that is only constructed when there's no existing matching entry.

Additional Notes

Stacked on top of master. #11381 (producer/consumer split + SpanSnapshot inbox) and #11409 (Hashtable utility) have both landed in master, so the diff shown here is only the AggregateTable / AggregateEntry / ClearSignal work plus follow-up cleanups.

Restructures the consumer-side aggregate store. Three logical commits, intended to be reviewed in order:

1. Add AggregateTable + AggregateEntry backed by Hashtable

Introduces a multi-key hash table that lets the consumer thread look up the {labels → counters} entry directly from a SpanSnapshot's raw fields — no MetricKey allocation per snapshot, no per-snapshot UTF8 cache lookups, no CHM operations. Hot-path lookup is keyHash compute → Hashtable.Support.bucket → bucket walk → matches(keyHash, snapshot) → returned entry has the counters to mutate in place.

This commit is standalone — no call sites yet, only the new classes + unit tests for hit/miss/cap-overrun/expunge/clear behavior.

2. Swap Aggregator to use AggregateTable + route disable() clear through a ClearSignal

Replaces LRUCache<MetricKey, AggregateMetric> with AggregateTable in Aggregator. Drops the AggregateExpiry listener — drop reporting (onStatsAggregateDropped) moves to the cap-overrun path inside Drainer.accept.

Threading fix bundled here: ConflatingMetricsAggregator.disable() used to call aggregator.clearAggregates() and inbox.clear() directly from the Sink's IO callback thread, racing with the aggregator thread. That race was tolerable for LinkedHashMap (worst case = corrupted internal state right before everything got cleared anyway); it's not tolerable for Hashtable (chain corruption can NPE or loop). disable() now offers a ClearSignal to the inbox so the aggregator thread itself performs the clear — preserves the single-writer invariant for AggregateTable end-to-end. The offer is best-effort; the system self-heals on a subsequent downgrade cycle if the inbox happens to be full (commented at the call site).

Cap-overrun semantic change: the old LRUCache evicted least-recently-used in O(1). AggregateTable instead scans for a hitCount==0 entry to recycle (O(N) worst-case), and drops the new key if none exists. Practical impact: in steady state, an unrelated burst of new keys gets dropped (and reported via onStatsAggregateDropped) rather than evicting established keys. The cost trade-off is commented at the eviction site — eviction is expected rare because the cap is sized to the working set; cursor-caching is the future option if a workload runs persistently at cap. The existing test that asserted "service0 evicted in favor of service10" is updated to assert the new semantics; the other cap-related test ("evicted entry was already flushed") still passes unchanged.

3. Fold MetricKey + AggregateMetric into AggregateEntry

MetricKey existed for two reasons — being the LRUCache key (replaced by AggregateTable's Hashtable mechanics) and being the labels arg to MetricWriter.add (the only thing left). AggregateMetric was the counter/histogram counterpart. Folds both onto a single AggregateEntry (10 UTF8 label fields + 3 primitives + counters + histograms), changes MetricWriter.add(MetricKey, AggregateMetric) → add(AggregateEntry), and deletes MetricKey.java + MetricKeys.java + AggregateMetric.java.

The 12 UTF8 caches that used to be split between MetricKey (9) and ConflatingMetricsAggregator (3, with overlap) are consolidated on AggregateEntry. One cache per field type now.

Latent bug fix: the prior matches(SpanSnapshot) used Objects.equals on raw fields. If the same logical key was delivered once as String and once as UTF8BytesString (different CharSequence impls of identical content), Objects.equals returned false and the table would split into two entries for the same key. The new matches uses content-equality (UTF8BytesString.toString() returns the underlying String in O(1)), collapsing them correctly.

Test impact: AggregateEntry.of(...) mirrors the prior new MetricKey(...) positional args, so test diffs are mostly mechanical. About 56 test sites migrated across ConflatingMetricAggregatorTest, SerializingMetricWriterTest, and MetricsIntegrationTest.

Review polish

Follow-up commits address review feedback:

• Use Hashtable.Support.create(maxAggregates, Support.MAX_RATIO) + Support.bucket + Support.insertHeadEntry(buckets, keyHash, entry) + Support.mutatingTableIterator to delegate to the helpers added on Add Hashtable and LongHashingUtils utilities #11409 — drops ~50 lines of bespoke bucket-array code.
• Inline the report-time forEach lambda instead of a static BiConsumer constant (the JIT reuses non-capturing lambdas).
• AggregateEntry.matches(long keyHash, SpanSnapshot) overload that pre-checks the hash, so chain walks read as one call.
• @Nullable (javax.annotation) annotations on the four nullable label fields + their getters + of(...) parameters.
• Objects.equals import in AggregateEntry.equals() (no more fully-qualified refs).
• Design-trade-off comments on evictOneStale (O(N) scan rationale) and disable() (best-effort offer rationale).

Additional cleanups

A second round of review surfacing landed these:

• MetricsIntegrationTest .aggregate runtime bug — the legacy entry.aggregate.recordDurations(...) form compiled under Groovy's dynamic dispatch but would have thrown MissingPropertyException at runtime. Fixed to call recordDurations directly on the entry. (Bot-flagged.)
• Spock >> { closure } no-op assertions in ConflatingMetricAggregatorTest — the >> operator stubs a return value, so the closures verifying e.getHitCount() == X && ... were being evaluated and discarded. Wrapped 31 sites in assert so Groovy power-assert surfaces mismatches. All 41 tests still pass, so the previously-unverified assertions happened to hold. (Bot-flagged.)
• Drop dead recordDurations(int, AtomicLongArray) batch API — vestige of master's Batch design. Production now only calls recordOneDuration(long). Migrated the three remaining test callers (AggregateEntryTest, SerializingMetricWriterTest, MetricsIntegrationTest) to loops of recordOneDuration calls, then deleted the batched method and its AtomicLongArray imports.
• AggregateEntry.of() colon-split Javadoc warning — the test factory recovers (name, value) pairs from "name:value" strings by splitting at the first :, which is brittle if a peer-tag value contains a colon (URLs, IPv6, service:env). Added an explicit warning so callers know to keep test data colon-free in values.
• ConflatingMetricsAggregatorDisableTest — new JUnit 5 coverage for the disable() → ClearSignal threading routing. The test fires DOWNGRADED from the test thread, waits for the no-flush window, then publishes a marker span with a distinct resource name and asserts the next flush captures only the marker — proving CLEAR actually wiped the original entry from the table. Catches both the missing-clear regression and the bucket-chain-corruption regression that the original threading race could produce.

Benchmarks

Producer publish() latency (single-threaded, 2 forks × 5 iter × 15s)

	Prior commit (stacked base)	This PR
SimpleSpan bench	3.116 µs/op	3.123 µs/op
DDSpan bench	2.506 µs/op	2.412 µs/op

All within noise on the producer side — this PR is a consumer-side refactor, so producer publish() shouldn't move much. The structural wins (one less class, no per-snapshot MetricKey allocation on the consumer, no double-cache lookups, smaller per-entry footprint) only become visible when the consumer is hammered hard enough that snapshot processing rate matters. That's exactly what the adversarial bench measures.

Adversarial bench suite vs v1.62.0 + master

Re-measured 2026-05-27 with three benches in matrix form: full adversarial (all four label dimensions vary) and two cardinality-isolation companions (only resource varies; only peer.hostname varies). Same machine state, same JMH config (8 producer threads, 2×15s warmup + 5×15s, 1 fork, throughput mode). The HighCardinality* and Adversarial benches were backported onto the v1.62.0 tag using its ConflatingMetricsAggregator constructor and HealthMetrics.NO_OP (v1.62.0 predates the inbox split so per-iteration drop counters are not directly comparable).

	v1.62.0 release	master (post-#11381)	this PR (#11382)	vs v1.62.0	vs master
AdversarialMetricsBenchmark (ops/s)	444,290 ± 1,616,937	14,276,351 ± 1,091,138	32,556,300 ± 4,321,490	73.3×	2.28×
HighCardinalityResource (ops/s)	4,854,335 ± 1,214,233	8,168,005 ± 3,493,716	35,739,452 ± 2,556,684	7.36×	4.38×
HighCardinalityPeer (ops/s)	6,902,209 ± 368,641	10,110,142 ± 3,380,594	37,638,634 ± 6,673,337	5.45×	3.72×
Adversarial onStatsAggregateDropped	n/a (HealthMetrics.NO_OP)	155,251,623	16,568,738	—	9.4× fewer drops

Customer headline: a workload migrating from v1.62.0 sees ~5–7× throughput on single-axis cardinality and ~73× on the adversarial all-axes attack — the all-axes case is the one where v1.62.0's LRU cache thrashes catastrophically (per-iteration progression on v1.62.0 Adversarial: 1.08M warmup → 277K → 199K, classic capacity-degradation curve). The master baseline includes #11381's producer/consumer split, so vs-master numbers represent the consumer-side AggregateTable redesign alone; vs-v1.62.0 is the full CSS rework.

The aggregate-drop counter shape confirms the design intent: master and v1.62.0 saturate tracerMetricsMaxAggregates continuously (LRU thrashing in v1.62.0; eviction-stream in master), shedding work at the expensive end of the pipeline. This PR pushes drops out to the producer-side inbox fast path (onStatsInboxFull) where load shedding is cheap, and the consumer-side onStatsAggregateDropped collapses by ~10×.

Net code delta: +1280 / −903 = +377 lines across 16 files. The growth is dominated by new test coverage (AggregateTableTest, AggregateEntryTest) plus the consolidated UTF8 caches landing on AggregateEntry; the production-code core (less MetricKey + MetricKeys + AggregateMetric minus AggregateEntry's additions) is roughly flat.

Known memory items addressed downstream

Two memory concerns visible at this layer of the stack are addressed in subsequent PRs — flagging here so reviewers don't worry about them in isolation:

• Two Histogram instances per entry are eagerly allocated (worst case ~4 MB heap floor at default maxAggregates=2048 × 2 × ~1 KB DDSketch). Most entries never see an error and so the errorLatencies histogram is unused. Fixed in Memory-efficiency pass on ClientStatsAggregator + adversarial benchmark #11389, which makes errorLatencies lazy: it stays null until the first error is recorded, and getErrorLatencies() returns a shared empty histogram in the no-error case.

• PEER_TAGS_CACHE worst case is 64 outer × 512 inner = 32 K cached UTF8BytesString peer-tag pairs, heap-pinned for the JVM lifetime. Past the inner cap the per-name LRU starts thrashing under unbounded cardinality, and the cache size itself is the only memory backstop. Fixed in Per-component / tag cardinality limits in client-side stats #11387, which adds per-tag TagCardinalityHandler budgets that fold overflow values into a "<tag>:blocked_by_tracer" sentinel; cardinality is bounded per reporting interval rather than per JVM lifetime, and the worst-case cache occupancy collapses to the configured budget.

Test plan

• ./gradlew :dd-trace-core:test --tests 'datadog.trace.common.metrics.*' passes (incl. the new AggregateTableTest and AggregateEntryTest)
• ./gradlew :dd-trace-core:compileJava :dd-trace-core:compileTestGroovy :dd-trace-core:compileJmhJava :dd-trace-core:compileTraceAgentTestGroovy all green
• ./gradlew spotlessCheck clean
• CI muzzle / integration suites
• Validate stats.dropped_aggregates semantics at high cardinality (especially the new "drop new on cap overrun" path vs. the old "evict LRU" path)

🤖 Generated with Claude Code

[dd-octo-sts]

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

• Update the submodule to the latest commit on the master branch using the git submodule update command,
• Check you properly set up your environment for contributing.

[dd-octo-sts]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-05-29 14:37:56 UTC ℹ️ Start processing command /merge

---

2026-05-29 14:38:01 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

---

2026-05-29 15:46:03 UTC ℹ️ MergeQueue: This merge request was merged

[bric3]

Metric aggregation: one entry per key

What this PR does

The Java tracer computes span metrics on a single background aggregator thread.
Before this change, every span it drained built a throwaway MetricKey to look
up a generic, thread-safe LRUCache<MetricKey, AggregateMetric> — two heap
objects per unique key, plus a key that was discarded immediately on the common
case (a cache hit). At a few million spans a second that's a constant stream of
nursery garbage produced for nothing.

This PR collapses the key and the value into a single AggregateEntry and stores
it in a purpose-built AggregateTable — a separately-chained hashtable where each
entry is its own chain node. Because the aggregator thread is now the sole
reader and writer (established by the previous PR), the table doesn't need to be
thread-safe, and lookups can match a raw SpanSnapshot against entries directly.
The upshot: no MetricKey is ever allocated on the lookup path, and a new
AggregateEntry is created only the first time a given key is seen.

Two behaviors worth flagging. Eviction is no longer LRU: at capacity the table
reclaims an idle (hitCount == 0) entry, and if every entry is live it drops the
newcomer rather than evicting an established one. And since the table is
single-writer, the agent-downgrade clear is routed as a CLEAR message through
the same inbox, so the aggregator thread is the one that performs the reset.

How a span gets aggregated

AggregateTable.findOrInsert(snapshot) does only the lookup-or-insert and returns
an AggregateEntry (on a hit or a fresh insert) or null (table full, nothing
stale to evict). The actual recording — and the drop counter — happen one level
up, in the Aggregator drain loop:

flowchart TD
  subgraph FOI["AggregateTable.findOrInsert(snapshot)"]
    direction TB
    H["hashOf → keyHash"] --> W["walk bucket chain<br/>matches(keyHash, s)"] --> Q{match?}
    Q -->|found| RE([return entry])
    Q -->|miss| R{room? / evict stale}
    R -->|yes| NEW["new entry → chain head"] --> RE
    R -->|no| RN([return null])
  end

  subgraph CALL["caller · Aggregator drain loop"]
    direction TB
    REC["recordOneDuration()<br/>dirty = true"]
    DROP["onStatsAggregateDropped()"]
  end

  RE -->|non-null| REC
  RN -->|null| DROP

  classDef neutral fill:#eceae3,stroke:#9a988f,color:#3a3a36;
  classDef ok   fill:#d6f3e8,stroke:#1d9e75,color:#085041;
  classDef warn fill:#fdeccd,stroke:#ba7517,color:#633806;
  classDef bad  fill:#fbdada,stroke:#c0392b,color:#791f1f;
  class H,W,Q,R neutral
  class RE,REC ok
  class NEW warn
  class RN,DROP bad

  style FOI fill:#f5fbf8,stroke:#1d9e75,stroke-width:1px,color:#085041
  style CALL fill:#f5f4f0,stroke:#9a988f,stroke-width:1px,color:#3a3a36

Loading

Inside the chain walk, matches(keyHash, s) compares the cheap 64-bit keyHash
first and only falls through to the field-by-field comparison when the hashes are
equal — which is what makes lookups without a key object fast.

---

Within discussion Claude generated diagram :