Add span-derived primary tags (CSS v1.3.0)#11402
Open
dougqh wants to merge 293 commits into
Open
Conversation
dougqh
added
type: enhancement
dougqh
force-pushed
the
dougqh/metrics-memory-efficiency
branch
from
46c04bd to
823a5d4
Compare
dougqh
force-pushed
the
dougqh/metrics-arbitrary-tags
branch
from
c552e73 to
42947dd
Compare
- Constructor sizing now uses Support.MAX_RATIO_NUMERATOR / _DENOMINATOR instead of an open-coded * 4 / 3. - findOrInsert delegates the chain-head splice to Support.insertHeadEntry. - evictOneStale and expungeStaleAggregates both rewritten in terms of Support.mutatingTableIterator. Drops the bespoke head-vs-mid-chain branching that read as more complicated than the operation actually is. Net -28 lines in AggregateTable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… create() Replace Support.MAX_RATIO_NUMERATOR / _DENOMINATOR with a single float MAX_RATIO constant, and add a Support.create(int, float) overload that takes a scale factor. Callers now write Support.create(n, MAX_RATIO) instead of stitching together the int arithmetic at the call site. The scaled size is truncated (not ceiled) before going through sizeFor. sizeFor already rounds up to the next power of two, so truncation just absorbs float fuzz that would otherwise push a result like 12 * 4/3 = 16.0000005f past 16 and double the bucket array size for no reason. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- AggregateTable: switch to Support.create(maxAggregates, Support.MAX_RATIO) now that the load-factor scaling is a Support concern. - AggregateTable: replace open-coded "keyHash == X && matches(s)" with a new AggregateEntry.matches(long keyHash, SpanSnapshot) overload that bundles the hash gate. - AggregateTable: rename local iterator var "it" -> "iter". - Aggregator: drop WRITE_AND_CLEAR static field, inline as a non-capturing lambda; the JIT reuses non-capturing lambdas, no need for the static until a profile says otherwise. - Aggregator: comment the ClearSignal branch with the thread-safety rationale (single-writer invariant for AggregateTable). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five small cleanups from a design re-review pass: 1. Support javadoc: drop the stale "methods are package-private" sentence; most of them were made public in earlier commits for higher-arity callers. Also drop the "nested BucketIterator" framing (iterators are peers of Support inside Hashtable, not nested inside Support). 2. MAX_RATIO javadoc: drop the Math.ceil recommendation; create(int, float) deliberately truncates and is the canonical pathway. 3. Document the null-hash treatment on D1.Entry.hash and D2.Entry.hash so the behavior difference is explicit: D1 uses Long.MIN_VALUE as a sentinel that's collision-free against any int-valued hashCode(); D2 has no such sentinel and relies on matches() to resolve null/null vs hash-0 collisions. 4. Rename Support.MAX_CAPACITY -> MAX_BUCKETS and sizeFor's parameter to requestedSize. The cap is on the bucket-array length, not entry count; the new name reflects that. Error messages updated to match. 5. Drop the `abstract` modifier on Hashtable in favor of `final` with a private constructor. Nothing actually subclasses Hashtable -- the abstract was a namespace device that read as "intended for extension." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add Support.insertHeadEntry(buckets, long keyHash, entry) overload that derives the bucket index itself. Callers that already have a hash but not the index (the common case) now avoid the redundant bucketIndex(...) hop. - D1.insert, D1.insertOrReplace, D2.insert, D2.insertOrReplace: use the new overload, drop the (thisBuckets local, bucketIndex compute, setNext, store) sequence at each call site. - D2.buckets: drop the `private` modifier to match D1.buckets. Both are package-private so iterator tests in the same package can drive Support.bucketIterator against the table's bucket array. Added a short comment on both fields documenting the rationale. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three follow-ups from the design review: - Make Hashtable.Entry.next private. All same-package readers (BucketIterator) already had a next() accessor; the leftover direct field reads now route through it. Closes the "mixed encapsulation" gap where some readers used the accessor and same-package ones reached for the field. - BucketIterator and MutatingBucketIterator now document that chain-walk work happens in next() (and the constructor for the first match); hasNext() is an O(1) field read. - Add D1.getOrCreate(K, Function) and D2.getOrCreate(K1, K2, BiFunction). Both reuse the lookup hash for the insert on miss, avoiding the double-hash that "get; if null then insert" callers would otherwise pay. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picks up the Support.insertHeadEntry(buckets, long keyHash, entry) overload added on the util-hashtable branch; saves the redundant Support.bucketIndex(buckets, keyHash) hop at the call site. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls in the util-hashtable design pass (MAX_RATIO, insertHeadEntry keyHash overload, MutatingTableIterator, Entry.next privatization, getOrCreate, MAX_BUCKETS rename, doc tightening, etc.) and the AggregateTable simplifications that came with it. Reconciliation notes: - AggregateEntry / AggregateMetric: this branch keeps the split design (AggregateEntry holds labels + an AggregateMetric counter ref). The optimize-metric-key branch had collapsed them into a single AggregateEntry. Resolution: keep the split (HEAD's design) -- it's more recent and supports the cardinality-canonicalization layer. - AggregateEntryTest.java (new in optimize-metric-key, exercises the collapsed design) deleted; AggregateMetricTest.groovy stays as the counter-side coverage for the split design. - AggregateTable: apply the optimize-metric-key cleanups on top of the Canonical-pattern findOrInsert -- Support.create(n, MAX_RATIO), Support.bucket for the chain head, Support.insertHeadEntry(keyHash), Support.mutatingTableIterator for evictOneStale and expungeStaleAggregates, Support.forEach for forEach. Also add the context-passing forEach overload to match the BiConsumer the Aggregator already uses. - Aggregator.report: keep the BiConsumer + context lambda (non-capturing); body adapted to entry.aggregate.clear() for the split design. - Aggregator.Drainer: keep AggregateMetric as the findOrInsert return type (matches the table's actual signature). - SerializingMetricWriter, SerializingMetricWriterTest, ClientStatsAggregatorTest, AggregateTableTest, SpanSnapshot, MetricWriter: restore HEAD's versions where the auto-merge had taken the optimize-metric-key shape (counters via entry.* vs entry.aggregate.*) -- HEAD's versions match this branch's design. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adopts the optimize-metric-key design choice: one entry type that holds both the canonical label fields and the counter / histogram state. The prior split (AggregateMetric for counters, AggregateEntry for labels) required every counter read to hop through entry.aggregate -- ~30 sites across SerializingMetricWriter, the Aggregator, and the test suites. - AggregateEntry now owns ERROR_TAG, TOP_LEVEL_TAG, the okLatencies and errorLatencies histograms, hitCount/errorCount/topLevelCount/duration counters, and the recordOneDuration / recordDurations / clear methods that used to live on AggregateMetric. - AggregateMetric.java and AggregateMetricTest.groovy deleted. - AggregateTable.findOrInsert now returns AggregateEntry (not the inner AggregateMetric); Canonical.toEntry no longer takes an AggregateMetric arg. - Aggregator.Drainer reverts to AggregateEntry; the report lambda calls entry.clear() directly. - SerializingMetricWriter, ClientStatsAggregator imports, and all three test files updated to read counters from entry.* (not entry.aggregate.*). - AggregateEntryTest.java added with the recordOneDuration / recordDurations / clear coverage that AggregateMetricTest.groovy used to provide. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ntry Use javax.annotation.Nullable (the codebase's convention -- see DDSpan, TagInterceptor, ScopeContext, etc.) on the four nullable label fields (serviceSource, httpMethod, httpEndpoint, grpcStatusCode), their getters, and the corresponding parameters of AggregateEntry.of. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Support.MAX_RATIO and the scaled create(int, float) overload already convey the 75% load-factor intent at the call site -- the inline comment was duplicating their self-documentation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Style nit -- the equals() method had eight fully-qualified references to java.util.Objects.equals; add the import and drop the qualifier. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two design-review trade-offs that won't change in this PR but should be explicit at the call sites: - AggregateTable.evictOneStale: O(N) scan per call (vs LRUCache's O(1)), acceptable because the new policy drops the *new* key on cap-overrun rather than evicting an established one -- so eviction is expected to be rare. Cursor-caching is the future optimization if a workload runs persistently at cap. - ConflatingMetricsAggregator.disable: single inbox.offer(CLEAR) is best-effort. If the inbox is full the clear is dropped, but the system self-heals (supportsMetrics() is already false, the next report-sink-rejection retries disable). Worst case is one extra cycle of stale data, not a leak. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
publish() previously did all of the tag extraction (peer-tag pairs, HTTP method/endpoint, span kind, gRPC status) and the SpanSnapshot allocation before calling inbox.offer; on a full inbox the offer failed and everything became garbage. Early-out with an approximate size() vs capacity() check up front. The jctools MPSC queue's size() is best-effort but that's fine: under- estimation falls through to the existing offer-as-source-of-truth path, over-estimation drops a snapshot that would have fit (and onStatsInboxFull was about to fire on the next span anyway). error is computed first so the force-keep return is correct whether or not the snapshot is built. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace `// nullable` comments on AggregateEntry's 4 nullable label fields (entry + Canonical scratch buffer) with `@Nullable` annotations. Also annotate the matching getters and of(...) factory parameters. - Move the cache revision into PeerTagSchema as a final field (peerTagsRevision), built via PeerTagSchema.of(names, revision). One field on the schema carries the cache key, so the hot path is a single volatile read + long compare against schema.peerTagsRevision -- no separate cachedPeerTagsRevision field on ClientStatsAggregator. When peer tags are unconfigured the cache stores an empty schema (size 0) carrying the revision rather than null, so subsequent publishes still short-circuit on the fast path. peerTagSchemaFor treats `schema.size() == 0` as "skip peer-agg processing" for client/producer/consumer kinds. INTERNAL is built with a -1L sentinel revision. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reset was split between two owners: Aggregator.report called AggregateEntry.resetCardinalityHandlers (static handlers + INTERNAL) then ran a separate onResetCardinality callback that ClientStats wired up to reset its cached non-INTERNAL peer-agg schema. Anyone adding a new handler had to know which side to put it on. Make the callback the only entry point. ClientStatsAggregator. resetCardinalityHandlers (renamed from resetCachedPeerAggSchema) now calls AggregateEntry.resetCardinalityHandlers() itself plus the cached peer-agg schema reset. Aggregator.report just runs the callback -- it no longer knows about AggregateEntry's static state. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each handler is now typed to its SpanSnapshot field type, so the HashMap's key class has well-defined equals/hashCode rather than the abstract CharSequence interface. For String-typed fields (service, spanKind, httpMethod, httpEndpoint, grpcStatusCode) the cache hits reliably. For CharSequence-typed fields (resource, operationName, serviceSource, type) consistency still depends on the producer returning a single concrete class per field -- a pre-existing runtime contract -- but the type system now prevents call sites from accidentally passing a different shape. registerOrEmpty is now generic so it threads T through. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously, reset() cleared the only cache, so every reporting cycle re-allocated UTF8BytesString instances for every property value seen again. Sustained allocations on the aggregator thread proportional to the sum of per-field cardinality limits, ~bytes/sec, on every reset. Split the state in two: - seenThisCycle (HashSet<T>): consumed-budget tracking, cleared on reset(). - utf8Cache (LinkedHashMap in access-order, 2x cardinalityLimit): long-lived; survives reset; LRU eviction once full. Workloads with stable value sets pay zero UTF8 allocations after the first cycle. The reused instances also short-circuit downstream equals to identity comparisons. Drops the TODO at the prior allocation site. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 9 property limits and the peer-tag value limit were sprinkled inline. Pull them into a single class with per-field javadoc so the sizing rationale lives in one place. Six values change from the DDCache-inherited defaults based on workload analysis: - RESOURCE 32 -> 128 (highest-cardinality field; tight today) - HTTP_ENDPOINT 32 -> 64 (same shape as RESOURCE for HTTP-heavy) - TYPE 8 -> 16 (DDSpanTypes catalogue is ~30) - HTTP_METHOD 8 -> 16 (WebDAV/custom verbs push past 8) - SPAN_KIND 16 -> 8 (OTel defines exactly 5 standard kinds) - GRPC_STATUS 32 -> 24 (gRPC spec has exactly 17 codes) SERVICE, OPERATION, SERVICE_SOURCE, and PEER_TAG_VALUE keep their current values. Net worst-case memory delta: roughly +90 KB driven by the RESOURCE and HTTP_ENDPOINT bumps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the previous LinkedHashMap-based design for PropertyCardinality Handler (and the HashMap-based TagCardinalityHandler) with parallel Object[] / UTF8BytesString[] arrays and linear-probing open addressing. Two tables per handler, "current cycle" and "prior cycle": - Capacity is the next power of two >= 2 * cardinalityLimit, so the linear-probing load factor stays <= 0.5 even when the budget is full. - Current tracks values that have consumed a slot of the cardinality budget this cycle. - Prior holds the just-completed cycle's entries verbatim. A first-time- this-cycle value that hits in prior reuses its UTF8BytesString instance -- no re-allocation. Implements the cross-reset reuse that the prior commit's LinkedHashMap LRU provided, with less overhead. Reset swaps the table pointers (just-completed cycle -> prior; the 2-cycles-ago tables get nulled out and become the new empty current). One O(capacity) pass, half the work of a copy-then-null. Wins: - No per-entry Node allocations (HashMap / LinkedHashMap) and no access-order linked-list maintenance per get. - Smaller working set: two Object[] + two UTF8BytesString[] per handler vs HashMap + HashSet + LinkedHashMap heap shapes. - Stable workloads pay zero UTF8BytesString allocations after the first cycle and produce identical references across cycles, so downstream equals short-circuits to ==. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The stored UTF8BytesString can serve as the slot's identity on its own: its hashCode() returns the underlying String.hashCode (content-stable with whatever shape the input takes), and equality is checked via stored.toString().contentEquals(value) -- the JDK's content-equality routine that fast-paths to String.equals when the input is a String. Halves the per-handler array footprint: one UTF8BytesString[] per cycle (current + prior) instead of one Object[] + one UTF8BytesString[] per cycle. No behavior change. TagCardinalityHandler keeps the parallel-arrays shape because its stored UTF8 is "tag:value" and cannot be compared directly against the bare incoming value. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ences The limitsEnabled=true/false arms are not a fair "cost of limiting" comparison: with unbounded distinct values, limits=true collapses overflow to one hot sentinel entry (recordOneDuration runs every span) while limits=false saturates the table and drops overflow (never reaches the histogram). GC profiling (3 forks, 2026-06-03) confirmed limits=true allocates MORE per op (888 vs 820 B/op) — the histogram recording, not the sentinel path. Documents this so the arm isn't misread as a regression. Also fixes two stale "cap of 100" references (actual is ADDITIONAL_TAG_VALUE=512). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- AdditionalTagsMetricsBenchmark.tearDown uses System.err (matches the sibling AdversarialMetricsBenchmark); add @SuppressForbidden so forbiddenApisJmh passes - AdditionalTagsSchemaTest: cover the new key validation — empty keys and ':'-containing keys are dropped; all-invalid input returns EMPTY Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
additionalTags was a sparse schema-width UTF8BytesString[] with null slots for absent tags. Switch to a compact array of only the present values (schema order preserved), matching the peerTags representation. Each value carries its "key:" prefix so packing loses no information. Eliminates the null-handling this required: - SerializingMetricWriter: drop countNonNull() pre-scan and the null-skip in the write loop; startArray(additionalTags.length) + tight loop - hashOf: take (array, count) and loop over count, like peer tags - matches: compact additionalTagsEqual() instead of Arrays.equals over the full sparse buffer (which would have compared stale trailing slots) - toEntry: Arrays.copyOf(buffer, size) instead of clone + allNull scan Pure cleanup/consistency; wire format unchanged. No measurable throughput change (these paths are flush-time or feature-gated) but removes a class of null-handling and makes additional tags parallel to peer tags. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… constructors Review feedback from amarziali + dougqh: - ClientStatsAggregator: pass the reset handler (onReportCycle) last to the Aggregator constructor, after additionalTagsSchema (dougqh). - Aggregator: reorder onReportCycle after additionalTagsSchema to match, and drop two dead constructor overloads (no callers). - Remove dead HealthMetrics.onAdditionalTagValueCardinalityBlocked + TracerHealthMetrics override/field/flush (orphaned when the bespoke AdditionalTagsCardinalityLimiter was removed earlier in the PR). - Remove dead AggregateEntry.Canonical.recomputeKeyHash. - AdditionalTagsSchema: move the builder javadoc off the test-convenience from(Set) overload onto the real from(Set, HealthMetrics, boolean). - SerializingMetricWriter / AggregateEntry.getAdditionalTags: fix stale comments/javadoc that still described sparse null-slotted storage; tags are now packed (canonical @return javadoc). - ClientStatsAggregator: drop the two no-additional-tags WellKnownTags constructor overloads; callers pass AdditionalTagsSchema.EMPTY explicitly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ead blockedSentinels - Remove duplicate AggregateEntry.resetCardinalityHandlers() / additionalTagsSchema.resetHandlers() calls from Aggregator.report(); move additionalTagsSchema.resetHandlers() into ClientStatsAggregator.resetCardinalityHandlers() where all resets are consolidated - Restore inbox-full pre-check before tag extraction in ClientStatsAggregator.publish() - Fix has*() identity checks (!=EMPTY) to use length()>0 so empty-string values don't appear as present on the wire - Remove dead blockedSentinels[] from AdditionalTagsSchema; each TagCardinalityHandler already owns its sentinel; add warn-once log when cardinality is exhausted, matching PeerTagSchema behaviour - Add isEmpty() guard for ignoredResources on the publish hot path Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…andlers TagCardinalityHandler now pre-builds its "tag:<name>" String[] at construction. AggregateEntry pre-builds a matching static String[] for each of its nine property fields. HealthMetrics.onTagCardinalityBlocked takes String[] directly so callers pass the pre-built array and TracerHealthMetrics forwards it to StatsD without allocating. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both PropertyCardinalityHandler and TagCardinalityHandler now hold a String name
and lazily build their String[]{"tag:<name>"} on first blocked value. In
production, most handlers never hit their limit so the array is never allocated.
AggregateEntry passes field names at handler construction and drops the 9 static
String[] constants; resetCardinalityHandlers passes the handler itself so the
statsDTag() call only happens when blocked > 0.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…teEntry.LIMITS_ENABLED The two-arg from(configured, healthMetrics) overload was pulling LIMITS_ENABLED from AggregateEntry, creating a schema→entry dependency. Collapsed to the one-arg test-convenience form (defaults to limits-enabled, matching handler test constructors) and updated ClientStatsAggregator to pass AggregateEntry.LIMITS_ENABLED explicitly at the one caller that needs the production flag. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… SerializingMetricWriter ctor - AdditionalTagsSchema.resetHandlers() now calls warnedCardinality.clear() each cycle, matching PeerTagSchema — previously the warn fired only once per JVM lifetime - PropertyCardinalityHandler gains a shouldWarnThisCycle() flag (cleared on reset()); AggregateEntry.reportIfBlocked() logs a warning on the first block per field per cycle, matching the warn-once-per-cycle pattern on peer and additional tags - SerializingMetricWriter 4-arg constructor was ignoring its gitInfoProvider param and constructing new GitInfoProvider() — fixed to assign the injected instance Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nonical PropertyCardinalityHandler.register(): add identity check (existing == value) before contentEquals in both the cur-cycle and prior-cycle probe loops. In steady state the same UTF8BytesString instance recurs every span so the contentEquals call is never reached. Canonical.computeKeyHash(): replace hashOf() with an inline loop so peer-tag and additional-tag entries (variable count) can be hashed without constructing a temporary array or iterating the fields twice. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…hema.INTERNAL reset PropertyCardinalityHandlers for the 9 per-field metrics properties were static fields on AggregateEntry, making them shared global state across all aggregator instances and impossible to reset between tests without explicit teardown. Extract them into a new PropertyHandlers class owned by AggregateTable (via Canonical). AggregateTable now creates its own PropertyHandlers on construction, and ClientStatsAggregator.resetCardinalityHandlers() delegates to Aggregator.resetPropertyHandlers() -> AggregateTable.resetHandlers() instead of holding a redundant outer reference. Also fix a regression: the old AggregateEntry.resetCardinalityHandlers() called PeerTagSchema.INTERNAL.resetCardinalityHandlers(healthMetrics), which was dropped when the static method was deleted. Without this call, INTERNAL's base.service handler never flushes its per-cycle working set, causing premature permanent blocking once the cardinality cap is hit. Re-add the call explicitly. Additional cleanup: - Remove dead Aggregator.additionalTagsSchema field (used only at construction) - Switch PropertyHandlers.reset() from manual enumeration to handlers[] for-loop, matching the pattern in PeerTagSchema and AdditionalTagsSchema - Revert unneeded DDAgentFeaturesDiscovery.discoverIfOutdated() refactor Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three gaps allowed the PeerTagSchema.INTERNAL regression to slip through: 1. No test for PropertyHandlers.reset() with limits enabled. Add PropertyHandlersTest covering: blocked counts reported per-handler, all nine handlers iterated, capacity refreshed after reset, and no HealthMetrics call when nothing was blocked. 2. AggregateTable's 2-arg constructor froze the limitsEnabled flag at class-load time via the LIMITS_ENABLED static final. Change to a live Config.get() read so DDSpecification tests can override the flag via withConfigOverride without forking a JVM. 3. No test for AggregateTable.resetHandlers(). Add resetHandlersClearsBlockedCountsAndRefreshesCapacity in AggregateTableTest, using the 3-arg constructor to inject PropertyHandlers(true) directly -- no Config dependency. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…es helper Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…reset; add Config.toString entry Simplifies AdditionalTagsSchema and PeerTagSchema by removing the warnedCardinality HashSet and isBlockedResult check from register(). The warn log and health metric now fire once per cycle in resetHandlers()/resetCardinalityHandlers() when blocked > 0, which is cleaner and avoids per-call overhead. Removes the now-dead isBlockedResult() method from TagCardinalityHandler. Adds traceStatsCardinalityLimitsEnabled to Config.toString() for diagnostics. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…NO_STATE constant Relocates the frozen cardinality-limits master switch from AggregateEntry.LIMITS_ENABLED to MetricCardinalityLimits.ENABLED, where the per-field limit values already live. The flag is no longer tangled into AggregateEntry (whose per-field handlers were already extracted to PropertyHandlers), and PeerTagSchema/ClientStatsAggregator reference it from its natural home instead of reaching across into AggregateEntry. AggregateTable keeps reading Config fresh per construction so tests can flip the flag via injectSysConfig. Names the bare null state passed to never-reconciled PeerTagSchemas (INTERNAL singleton and test schemas) as the explanatory NO_STATE constant. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tCardinalityHandlers to resetHandlers The per-cycle "if a handler blocked values this cycle, warn + flush count to HealthMetrics" loop was duplicated across PropertyHandlers, PeerTagSchema, and AdditionalTagsSchema, differing only in the warn-message wording. Consolidated into CardinalityBlocks.reportIfBlocked, which takes the already-extracted primitives (PropertyCardinalityHandler and TagCardinalityHandler share no common type) plus the caller's logger so log categories stay per-class. Also renames PeerTagSchema.resetCardinalityHandlers to resetHandlers to match AdditionalTagsSchema.resetHandlers. The aggregator's own no-arg resetCardinalityHandlers() orchestration method keeps its name. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clarifies the intentional fresh-vs-frozen split surfaced in review: Aggregatetable reads Config directly so tests can flip the flag via injectSysConfig, while PeerTagSchema and ClientStatsAggregator use the frozen MetricCardinalityLimits.ENABLED. The two are identical in production. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…torTest Both surfaced after the #11387→#11402 reconcile merge: Config.java's static-import block left TRACE_STATS_ADDITIONAL_TAGS out of alphabetical order (TRACER_* sorts before TRACE_*), and a long SimpleSpan publish line in ClientStatsAggregatorTest needed wrapping. Pure formatting; no behavior change. Restores green spotlessCheck on internal-api and dd-trace-core. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…le-writer metrics code spotbugsMain flagged design-intentional patterns: - AggregateEntry: AT_NONATOMIC_OPERATIONS_ON_SHARED_VARIABLE (recordOneDuration/recordDurations) and AT_STALE_THREAD_WRITE_OF_PRIMITIVE (clearAggregate) on the hit/error/topLevel/duration counters. These are single-writer by design -- mutated only on the aggregator thread (see class javadoc); making them volatile/atomic would cost the hot path for no benefit. - TagCardinalityHandler.register: ES_COMPARING_PARAMETER_STRING_WITH_EQ on the intentional reference-equality fast-path that short-circuits .equals() when key and probe value are the same instance. Suppressed with @SuppressFBWarnings + justifications (repo convention; matches the existing clearAggregate suppression). Trimmed to the exact codes that fire to avoid US_USELESS_SUPPRESSION. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Groovy test was stale on this branch: it built entries via AggregateEntry.hashOf(SpanSnapshot) and a (SpanSnapshot,long) constructor that no longer exist here (hashOf now takes the canonicalized UTF8 field list, and SpanSnapshot gained an additionalTagValues arg). Groovy compiles such calls dynamically, so it only blew up at runtime in the traceAgentTest job (MissingMethodException: hashOf() for (SpanSnapshot)). Rewritten as a self-contained JUnit 5 Java test that builds entries through the production AggregateTable.findOrInsert path (matching AggregateTableTest), with the current 16-arg SpanSnapshot (additionalTagValues=null). Agent wiring is replicated inline (testcontainers locally / CI_AGENT_HOST in CI; host+port via WithConfigExtension); metrics meter initialized in @BeforeAll. One test method, unchanged intent. Also follows the project's no-new-Groovy / migrate-existing-Groovy convention. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… CardinalityBlocks The CardinalityBlocks helper (one 6-arg static taking log/healthMetrics/blocked/name/statsDTag/ message) didn't improve readability over the inline loop -- the call was nearly as long as the body it replaced and forced the logger + message to be passed around. Reverted PropertyHandlers, PeerTagSchema, and AdditionalTagsSchema to the direct "reset -> if blocked > 0: warn + flush" loop and deleted CardinalityBlocks. The resetCardinalityHandlers -> resetHandlers rename is kept. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After rebasing metrics-arbitrary-tags onto control-tag-cardinality + cardinality-limits-config, several issues needed fixing: - Restore MAX_CARDINALITY_LIMIT constant to PropertyCardinalityHandler (lost during conflict resolution) - Fix AggregateTable: Hashtable.Support.create() was missing Hashtable. prefix; canonical.populateFrom() → canonical.populate() (method was renamed) - Re-remove testSchema() from PeerTagSchema and restore package-private constructor (ca5e5ef had done this; the rebase brought testSchema() back) - Remove AggregateEntry.resetCardinalityHandlers() @beforeeach in AggregateEntryTest — method no longer exists after handler refactor - Update AggregateEntryTestUtils.of() for new hashOf()/constructor signatures that include additionalTags + additionalTagCount parameters - Remove LIMITS_ENABLED / isTraceStatsCardinalityLimitsEnabled(): the cardinality-limits-config approach uses per-field config limits with sentinel always on; a global on/off toggle is redundant and was removed from GeneralConfig, Config, MetricCardinalityLimits, PropertyHandlers (which now reads per-field limits from Config.get()), AggregateTable, ClientStatsAggregator, and PeerTagSchema - Fix test sentinel strings: "blocked_by_tracer" → "tracer_blocked_value" Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dougqh
force-pushed
the
dougqh/metrics-arbitrary-tags
branch
from
9aff892 to
ef97e00
Compare
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Jul 1, 2026
Use cached span.kind ordinal in metrics producer; drop tag-map lookup
JFR profiling showed ~21% of producer CPU time spent in tag-map lookups
during ClientStatsAggregator.publish. One of those lookups -- span.kind --
is redundant because DDSpanContext already caches the kind as a byte
ordinal that resolves to a String via a small array.
- Add CoreSpan.getSpanKindString() with a default that falls back to the
tag map for non-DDSpan impls; DDSpan overrides to delegate to the
context's cached resolution.
- Hoist schema.names array out of the capturePeerTagValues loop.
- Avoid an unnecessary toString() in isSynthetic by declaring
SYNTHETICS_ORIGIN as String and using contentEquals.
Benchmark (ClientStatsAggregatorDDSpanBenchmark):
before: 2.410 us/op
after: 1.995 us/op (~17% improvement)
vs. master baseline (6.428 us/op): now ~3.2x faster.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add client metrics pipeline design doc
Captures the producer/consumer split, the canonical-key trick that makes
cardinality-blocking actually save space, the once-per-trace peer-tag
schema sync, the role of each file in datadog.trace.common.metrics, and
the rationale behind the redesign from ConflatingMetricsAggregator.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add unit tests for Hashtable and LongHashingUtils
LongHashingUtilsTest (14 cases):
- hashCodeX null sentinel + non-null pass-through
- all primitive hash() overloads match the boxed Java hashCodes
- hash(Object...) 2/3/4/5-arg overloads match the chained addToHash
formula they are documented to constant-fold to
- addToHash(long, primitive) overloads match the Object-version
- linear-accumulation invariant (31 * h + v) holds across a sequence
- iterable / deprecated int[] / deprecated Object[] variants match
chained addToHash
- intHash treats null as 0 (observable via hash(null, "x"))
HashtableTest (24 cases across 5 nested classes):
- D1: insert/get/remove/insertOrReplace/clear/forEach, in-place value
mutation, null-key handling, hash-collision chaining with disambig-
uating equals, remove-from-collided-chain leaves siblings intact
- D2: pair-key identity, remove(pair), insertOrReplace matches on
both parts, forEach
- Support: capacity rounds up to a power of two, bucketIndex stays
in range across a wide hash sample, clear nulls every slot
- BucketIterator: walks only matching-hash entries in a chain, throws
NoSuchElementException when exhausted
- MutatingBucketIterator: remove from head-of-chain unlinks, replace
swaps the entry while preserving chain, remove() without prior
next() throws IllegalStateException
Tests live in internal-api/src/test/java/datadog/trace/util and use the
already-present JUnit 5 setup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply spotless formatting to Hashtable and LongHashingUtils
Bring the new util/ files in line with google-java-format
(tabs → spaces, line wrapping, javadoc list markup) so
spotlessCheck passes in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add JMH benchmarks for Hashtable.D1 and D2
Compares Hashtable.D1 and Hashtable.D2 against equivalent HashMap
usage for add, update, and iterate operations. Each benchmark thread
owns its own map (Scope.Thread), but @Threads(8) is used so the
allocation/GC pressure that Hashtable is designed to avoid surfaces
in the throughput numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add benchmark results to HashtableBenchmark header
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Guard Support.sizeFor against overflow and use Integer.highestOneBit;
reject capacities above 1 << 30 instead of looping forever.
- Add braces around single-statement while bodies in BucketIterator.
- Split HashtableBenchmark into HashtableD1Benchmark / HashtableD2Benchmark.
- Add regression tests for Support.sizeFor bounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix dropped argument in HashingUtils 5-arg Object hash
The 5-arg Object overload was forwarding only obj0..obj3 to the int
overload, silently dropping obj4. Also align LongHashingUtils.hash 3-arg
signature with its 2/4/5-arg siblings (int parameters) and strengthen
the 5-arg HashingUtilsTest to detect the missing-arg regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Split D1Tests and D2Tests into HashtableD1Test and HashtableD2Test;
extract shared test entry classes into HashtableTestEntries.
- Reduce visibility of LongHashingUtils.hash(int...) chaining overloads
to package-private; they are internal building blocks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop reflection in iterator tests via package-private D1.buckets
The iterator tests need a populated Hashtable.Entry[] to drive
Support.bucketIterator / mutatingBucketIterator. Relaxing D1.buckets
from private to package-private lets the same-package tests read it
directly, removing the reflection helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resize previousCounts for inbox-full health metric
The new reason:inbox_full reportIfChanged call advances countIndex to 51,
but previousCounts was still sized for 51 counters (max index 50), so the
metric never emitted and the resize warning fired every flush. Bump the
array to 52 and add a regression test that exercises the flush path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fold AggregateMetric into AggregateEntry
The label fields and the mutable counters/histograms are 1:1 with each
entry; carrying them on a separate object meant one extra allocation per
unique key plus an indirection on every hot-path update. Merging them
puts the counters directly on AggregateEntry, drops the entry.aggregate
hop, and consolidates ERROR_TAG / TOP_LEVEL_TAG onto the same class the
consumer uses to decode them.
AggregateTable.findOrInsert now returns AggregateEntry. Callers in
Aggregator and SerializingMetricWriter updated. Migrated
AggregateMetricTest.groovy to AggregateEntryTest.java per project policy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Avoid capturing lambda in Aggregator.report
Add a context-passing forEach(T, BiConsumer) overload to AggregateTable,
mirroring TagMap's pattern. Aggregator.report now hands the writer in as
context to a static BiConsumer so no fresh Consumer is allocated each
report cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add context-passing forEach to Hashtable.D1 and D2
Mirrors the TagMap pattern: pairs the existing forEach(Consumer) with a
forEach(T context, BiConsumer<T, TEntry>) overload so callers can hand
side-band state to a non-capturing lambda and avoid the
fresh-Consumer-per-call allocation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move forEach loop body to Support helper
Factors the unchecked (TEntry) cast out of D1.forEach / D2.forEach (and
the BiConsumer variants) into Support.forEach(buckets, ...). The cast
now lives in one place, mirroring how Entry.next() handles it, and the
D1/D2 methods become one-liners. Downstream higher-arity tables built
on Support gain the same helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Delegate AggregateTable.forEach to Support.forEach
Now that Hashtable.Support exposes the parameterized forEach helpers,
AggregateTable's own forEach methods can drop their duplicated loop body
and the (AggregateEntry) cast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move bucket-head cast to Support.bucket helper
Adds Support.bucket(buckets, keyHash) which returns the bucket head
already cast to the caller's concrete entry type. D1.get and D2.get
now drop the raw-Entry intermediate variable and walk the chain via
Entry.next() directly. The unchecked cast lives in one place,
consistent with Entry.next() and Support.forEach.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Use Support.bucket and type chain walks as AggregateEntry
- findOrInsert: walks via Support.bucket(buckets, keyHash) instead of
Hashtable.Entry + intermediate cast; bucketIndex is only computed on
the miss path now.
- evictOneStale / expungeStaleAggregates: chain variables typed as
AggregateEntry from the head down, leveraging Entry.next()'s generic
inference, so the per-iteration getHitCount() checks drop their
(AggregateEntry) cast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop d1_/d2_ prefix from per-table benchmark methods
Holdover from when both lived in a shared HashtableBenchmark; redundant
now that each lives in its own class.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add DDAgentFeaturesDiscovery.peerTagsRevision()
Monotonically increases each time the discovered peerTags Set differs from
the previous one. Lets callers detect peer-tag config changes with a long
compare instead of a Set.equals (or leaning on Set-identity, which was an
implementation accident, not part of the public contract).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move peer-tag schema cache from PeerTagSchema statics to ClientStatsAggregator
PeerTagSchema previously held its current schema + last-synced-Set in static
volatile fields with a synchronized rebuild. The "is it stale?" signal was
an identity check on the Set instance returned by features.peerTags() -- a
correct but indirect reading of a DDAgentFeaturesDiscovery invariant.
Replace that with:
- ClientStatsAggregator keeps its own (volatile PeerTagSchema, volatile long
cachedPeerTagsRevision) cache pair, rebuilt under synchronized when the
revision returned by features.peerTagsRevision() doesn't match.
- PeerTagSchema becomes a pure data holder: static factory PeerTagSchema.of,
the INTERNAL singleton, and an instance resetCardinalityHandlers(). The
static CURRENT, LAST_SYNCED_INPUT, and the synchronized rebuild block are
gone.
- Aggregator gains a Runnable onResetCardinality hook fired right after
AggregateEntry.resetCardinalityHandlers(). ClientStatsAggregator wires it
to reset its cached schema's handlers each report cycle.
- AggregateEntry.resetCardinalityHandlers() resets PeerTagSchema.INTERNAL
directly instead of the removed PeerTagSchema.resetAll().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add Hashtable.Support helpers: MAX_RATIO, insertHeadEntry, MutatingTableIterator
Three consumer-facing helpers that callers building higher-arity tables on
top of Hashtable.Support kept open-coding:
- MAX_RATIO_NUMERATOR / _DENOMINATOR: the 4/3 multiplier for sizing a
bucket array from a target working-set under a 75% load factor.
- insertHeadEntry(buckets, bucketIndex, entry): the (setNext + array-store)
pair for splicing a new entry at the head of a bucket chain.
- MutatingTableIterator + Support.mutatingTableIterator(buckets): walks
every entry in the table (not filtered by hash) with remove() support,
for sweeps like eviction and expunge that aren't keyed to a specific
hash. Sibling of MutatingBucketIterator.
Tests cover the table-wide iterator at head-of-bucket and mid-chain
removal, empty buckets between live entries, exhaustion, and
remove-without-next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Simplify AggregateTable via new Hashtable.Support helpers
- Constructor sizing now uses Support.MAX_RATIO_NUMERATOR / _DENOMINATOR
instead of an open-coded * 4 / 3.
- findOrInsert delegates the chain-head splice to Support.insertHeadEntry.
- evictOneStale and expungeStaleAggregates both rewritten in terms of
Support.mutatingTableIterator. Drops the bespoke head-vs-mid-chain
branching that read as more complicated than the operation actually is.
Net -28 lines in AggregateTable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Swap MAX_RATIO numerator/denominator pair for a single float + scaled create()
Replace Support.MAX_RATIO_NUMERATOR / _DENOMINATOR with a single float
MAX_RATIO constant, and add a Support.create(int, float) overload that
takes a scale factor. Callers now write Support.create(n, MAX_RATIO)
instead of stitching together the int arithmetic at the call site.
The scaled size is truncated (not ceiled) before going through sizeFor.
sizeFor already rounds up to the next power of two, so truncation just
absorbs float fuzz that would otherwise push a result like 12 * 4/3 =
16.0000005f past 16 and double the bucket array size for no reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Address second-round review on AggregateTable / Aggregator
- AggregateTable: switch to Support.create(maxAggregates, Support.MAX_RATIO)
now that the load-factor scaling is a Support concern.
- AggregateTable: replace open-coded "keyHash == X && matches(s)" with a
new AggregateEntry.matches(long keyHash, SpanSnapshot) overload that
bundles the hash gate.
- AggregateTable: rename local iterator var "it" -> "iter".
- Aggregator: drop WRITE_AND_CLEAR static field, inline as a non-capturing
lambda; the JIT reuses non-capturing lambdas, no need for the static
until a profile says otherwise.
- Aggregator: comment the ClearSignal branch with the thread-safety
rationale (single-writer invariant for AggregateTable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Hashtable docs + rename MAX_CAPACITY to MAX_BUCKETS
Five small cleanups from a design re-review pass:
1. Support javadoc: drop the stale "methods are package-private" sentence;
most of them were made public in earlier commits for higher-arity
callers. Also drop the "nested BucketIterator" framing (iterators are
peers of Support inside Hashtable, not nested inside Support).
2. MAX_RATIO javadoc: drop the Math.ceil recommendation; create(int, float)
deliberately truncates and is the canonical pathway.
3. Document the null-hash treatment on D1.Entry.hash and D2.Entry.hash so
the behavior difference is explicit: D1 uses Long.MIN_VALUE as a
sentinel that's collision-free against any int-valued hashCode(); D2
has no such sentinel and relies on matches() to resolve null/null vs
hash-0 collisions.
4. Rename Support.MAX_CAPACITY -> MAX_BUCKETS and sizeFor's parameter to
requestedSize. The cap is on the bucket-array length, not entry count;
the new name reflects that. Error messages updated to match.
5. Drop the `abstract` modifier on Hashtable in favor of `final` with a
private constructor. Nothing actually subclasses Hashtable -- the
abstract was a namespace device that read as "intended for extension."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dedupe chain-head splice in D1/D2 via keyHash insertHeadEntry overload
- Add Support.insertHeadEntry(buckets, long keyHash, entry) overload that
derives the bucket index itself. Callers that already have a hash but
not the index (the common case) now avoid the redundant bucketIndex(...)
hop.
- D1.insert, D1.insertOrReplace, D2.insert, D2.insertOrReplace: use the
new overload, drop the (thisBuckets local, bucketIndex compute,
setNext, store) sequence at each call site.
- D2.buckets: drop the `private` modifier to match D1.buckets. Both are
package-private so iterator tests in the same package can drive
Support.bucketIterator against the table's bucket array. Added a short
comment on both fields documenting the rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Entry.next encapsulation; doc hasNext; add D1/D2 getOrCreate
Three follow-ups from the design review:
- Make Hashtable.Entry.next private. All same-package readers
(BucketIterator) already had a next() accessor; the leftover direct
field reads now route through it. Closes the "mixed encapsulation"
gap where some readers used the accessor and same-package ones
reached for the field.
- BucketIterator and MutatingBucketIterator now document that chain-walk
work happens in next() (and the constructor for the first match);
hasNext() is an O(1) field read.
- Add D1.getOrCreate(K, Function) and D2.getOrCreate(K1, K2, BiFunction).
Both reuse the lookup hash for the insert on miss, avoiding the
double-hash that "get; if null then insert" callers would otherwise
pay.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Use keyHash insertHeadEntry overload in AggregateTable.findOrInsert
Picks up the Support.insertHeadEntry(buckets, long keyHash, entry)
overload added on the util-hashtable branch; saves the redundant
Support.bucketIndex(buckets, keyHash) hop at the call site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge dougqh/optimize-metric-key into dougqh/control-tag-cardinality
Pulls in the util-hashtable design pass (MAX_RATIO, insertHeadEntry
keyHash overload, MutatingTableIterator, Entry.next privatization,
getOrCreate, MAX_BUCKETS rename, doc tightening, etc.) and the
AggregateTable simplifications that came with it.
Reconciliation notes:
- AggregateEntry / AggregateMetric: this branch keeps the split design
(AggregateEntry holds labels + an AggregateMetric counter ref). The
optimize-metric-key branch had collapsed them into a single
AggregateEntry. Resolution: keep the split (HEAD's design) -- it's
more recent and supports the cardinality-canonicalization layer.
- AggregateEntryTest.java (new in optimize-metric-key, exercises the
collapsed design) deleted; AggregateMetricTest.groovy stays as the
counter-side coverage for the split design.
- AggregateTable: apply the optimize-metric-key cleanups on top of the
Canonical-pattern findOrInsert -- Support.create(n, MAX_RATIO),
Support.bucket for the chain head, Support.insertHeadEntry(keyHash),
Support.mutatingTableIterator for evictOneStale and
expungeStaleAggregates, Support.forEach for forEach. Also add the
context-passing forEach overload to match the BiConsumer the
Aggregator already uses.
- Aggregator.report: keep the BiConsumer + context lambda
(non-capturing); body adapted to entry.aggregate.clear() for the
split design.
- Aggregator.Drainer: keep AggregateMetric as the findOrInsert return
type (matches the table's actual signature).
- SerializingMetricWriter, SerializingMetricWriterTest,
ClientStatsAggregatorTest, AggregateTableTest, SpanSnapshot,
MetricWriter: restore HEAD's versions where the auto-merge had taken
the optimize-metric-key shape (counters via entry.* vs
entry.aggregate.*) -- HEAD's versions match this branch's design.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fold AggregateMetric into AggregateEntry
Adopts the optimize-metric-key design choice: one entry type that holds
both the canonical label fields and the counter / histogram state. The
prior split (AggregateMetric for counters, AggregateEntry for labels)
required every counter read to hop through entry.aggregate -- ~30 sites
across SerializingMetricWriter, the Aggregator, and the test suites.
- AggregateEntry now owns ERROR_TAG, TOP_LEVEL_TAG, the okLatencies and
errorLatencies histograms, hitCount/errorCount/topLevelCount/duration
counters, and the recordOneDuration / recordDurations / clear methods
that used to live on AggregateMetric.
- AggregateMetric.java and AggregateMetricTest.groovy deleted.
- AggregateTable.findOrInsert now returns AggregateEntry (not the inner
AggregateMetric); Canonical.toEntry no longer takes an AggregateMetric
arg.
- Aggregator.Drainer reverts to AggregateEntry; the report lambda calls
entry.clear() directly.
- SerializingMetricWriter, ClientStatsAggregator imports, and all three
test files updated to read counters from entry.* (not entry.aggregate.*).
- AggregateEntryTest.java added with the recordOneDuration /
recordDurations / clear coverage that AggregateMetricTest.groovy used
to provide.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove accidentally-staged .claude/worktrees entries
Replace // nullable comments with @Nullable annotations on AggregateEntry
Use javax.annotation.Nullable (the codebase's convention -- see DDSpan,
TagInterceptor, ScopeContext, etc.) on the four nullable label fields
(serviceSource, httpMethod, httpEndpoint, grpcStatusCode), their
getters, and the corresponding parameters of AggregateEntry.of.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop redundant load-factor comment from AggregateTable ctor
Support.MAX_RATIO and the scaled create(int, float) overload already
convey the 75% load-factor intent at the call site -- the inline
comment was duplicating their self-documentation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Import java.util.Objects in AggregateEntry instead of fully qualifying
Style nit -- the equals() method had eight fully-qualified references
to java.util.Objects.equals; add the import and drop the qualifier.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Document evictOneStale cost and disable() best-effort offer
Two design-review trade-offs that won't change in this PR but should be
explicit at the call sites:
- AggregateTable.evictOneStale: O(N) scan per call (vs LRUCache's O(1)),
acceptable because the new policy drops the *new* key on cap-overrun
rather than evicting an established one -- so eviction is expected to
be rare. Cursor-caching is the future optimization if a workload runs
persistently at cap.
- ConflatingMetricsAggregator.disable: single inbox.offer(CLEAR) is
best-effort. If the inbox is full the clear is dropped, but the
system self-heals (supportsMetrics() is already false, the next
report-sink-rejection retries disable). Worst case is one extra cycle
of stale data, not a leak.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Skip SpanSnapshot allocation when the inbox is already at capacity
publish() previously did all of the tag extraction (peer-tag pairs,
HTTP method/endpoint, span kind, gRPC status) and the SpanSnapshot
allocation before calling inbox.offer; on a full inbox the offer
failed and everything became garbage.
Early-out with an approximate size() vs capacity() check up front. The
jctools MPSC queue's size() is best-effort but that's fine: under-
estimation falls through to the existing offer-as-source-of-truth
path, over-estimation drops a snapshot that would have fit (and
onStatsInboxFull was about to fire on the next span anyway).
error is computed first so the force-keep return is correct whether
or not the snapshot is built.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review on AggregateEntry nullables + PeerTagSchema revision
- Replace `// nullable` comments on AggregateEntry's 4 nullable label
fields (entry + Canonical scratch buffer) with `@Nullable`
annotations. Also annotate the matching getters and of(...) factory
parameters.
- Move the cache revision into PeerTagSchema as a final field
(peerTagsRevision), built via PeerTagSchema.of(names, revision). One
field on the schema carries the cache key, so the hot path is a
single volatile read + long compare against schema.peerTagsRevision
-- no separate cachedPeerTagsRevision field on ClientStatsAggregator.
When peer tags are unconfigured the cache stores an empty schema
(size 0) carrying the revision rather than null, so subsequent
publishes still short-circuit on the fast path. peerTagSchemaFor
treats `schema.size() == 0` as "skip peer-agg processing" for
client/producer/consumer kinds.
INTERNAL is built with a -1L sentinel revision.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidate cardinality-handler reset behind one entry point
Reset was split between two owners: Aggregator.report called
AggregateEntry.resetCardinalityHandlers (static handlers + INTERNAL)
then ran a separate onResetCardinality callback that ClientStats
wired up to reset its cached non-INTERNAL peer-agg schema. Anyone
adding a new handler had to know which side to put it on.
Make the callback the only entry point. ClientStatsAggregator.
resetCardinalityHandlers (renamed from resetCachedPeerAggSchema) now
calls AggregateEntry.resetCardinalityHandlers() itself plus the
cached peer-agg schema reset. Aggregator.report just runs the
callback -- it no longer knows about AggregateEntry's static state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Parameterize PropertyCardinalityHandler on T extends CharSequence
Each handler is now typed to its SpanSnapshot field type, so the
HashMap's key class has well-defined equals/hashCode rather than the
abstract CharSequence interface. For String-typed fields (service,
spanKind, httpMethod, httpEndpoint, grpcStatusCode) the cache hits
reliably. For CharSequence-typed fields (resource, operationName,
serviceSource, type) consistency still depends on the producer
returning a single concrete class per field -- a pre-existing runtime
contract -- but the type system now prevents call sites from
accidentally passing a different shape.
registerOrEmpty is now generic so it threads T through.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add long-lived LRU cache to PropertyCardinalityHandler
Previously, reset() cleared the only cache, so every reporting cycle
re-allocated UTF8BytesString instances for every property value seen
again. Sustained allocations on the aggregator thread proportional to
the sum of per-field cardinality limits, ~bytes/sec, on every reset.
Split the state in two:
- seenThisCycle (HashSet<T>): consumed-budget tracking, cleared on
reset().
- utf8Cache (LinkedHashMap in access-order, 2x cardinalityLimit):
long-lived; survives reset; LRU eviction once full.
Workloads with stable value sets pay zero UTF8 allocations after the
first cycle. The reused instances also short-circuit downstream equals
to identity comparisons.
Drops the TODO at the prior allocation site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Centralize per-field cardinality limits in MetricCardinalityLimits
The 9 property limits and the peer-tag value limit were sprinkled
inline. Pull them into a single class with per-field javadoc so the
sizing rationale lives in one place. Six values change from the
DDCache-inherited defaults based on workload analysis:
- RESOURCE 32 -> 128 (highest-cardinality field; tight today)
- HTTP_ENDPOINT 32 -> 64 (same shape as RESOURCE for HTTP-heavy)
- TYPE 8 -> 16 (DDSpanTypes catalogue is ~30)
- HTTP_METHOD 8 -> 16 (WebDAV/custom verbs push past 8)
- SPAN_KIND 16 -> 8 (OTel defines exactly 5 standard kinds)
- GRPC_STATUS 32 -> 24 (gRPC spec has exactly 17 codes)
SERVICE, OPERATION, SERVICE_SOURCE, and PEER_TAG_VALUE keep their
current values. Net worst-case memory delta: roughly +90 KB driven by
the RESOURCE and HTTP_ENDPOINT bumps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reimplement cardinality handlers as open-addressed flat arrays
Replaces the previous LinkedHashMap-based design for PropertyCardinality
Handler (and the HashMap-based TagCardinalityHandler) with parallel
Object[] / UTF8BytesString[] arrays and linear-probing open addressing.
Two tables per handler, "current cycle" and "prior cycle":
- Capacity is the next power of two >= 2 * cardinalityLimit, so the
linear-probing load factor stays <= 0.5 even when the budget is full.
- Current tracks values that have consumed a slot of the cardinality
budget this cycle.
- Prior holds the just-completed cycle's entries verbatim. A first-time-
this-cycle value that hits in prior reuses its UTF8BytesString
instance -- no re-allocation. Implements the cross-reset reuse that
the prior commit's LinkedHashMap LRU provided, with less overhead.
Reset swaps the table pointers (just-completed cycle -> prior; the
2-cycles-ago tables get nulled out and become the new empty current).
One O(capacity) pass, half the work of a copy-then-null.
Wins:
- No per-entry Node allocations (HashMap / LinkedHashMap) and no
access-order linked-list maintenance per get.
- Smaller working set: two Object[] + two UTF8BytesString[] per handler
vs HashMap + HashSet + LinkedHashMap heap shapes.
- Stable workloads pay zero UTF8BytesString allocations after the first
cycle and produce identical references across cycles, so downstream
equals short-circuits to ==.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop parallel keys array in PropertyCardinalityHandler
The stored UTF8BytesString can serve as the slot's identity on its own:
its hashCode() returns the underlying String.hashCode (content-stable
with whatever shape the input takes), and equality is checked via
stored.toString().contentEquals(value) -- the JDK's content-equality
routine that fast-paths to String.equals when the input is a String.
Halves the per-handler array footprint: one UTF8BytesString[] per
cycle (current + prior) instead of one Object[] + one
UTF8BytesString[] per cycle. No behavior change.
TagCardinalityHandler keeps the parallel-arrays shape because its
stored UTF8 is "tag:value" and cannot be compared directly against the
bare incoming value.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop type parameter from PropertyCardinalityHandler
The type parameter was load-bearing when slot identity went through a
parallel Object[] keys array (where T determined the runtime class
whose equals/hashCode the HashMap used). The single-array shape probes
via UTF8BytesString.hashCode() (content-stable with the underlying
String) and stored.toString().contentEquals(value), so any CharSequence
input -- String, UTF8BytesString, anything else with a content-stable
hash -- collapses to the right slot.
register(CharSequence value) is enough. AggregateEntry's 9 static
handler declarations and the registerOrEmpty helper lose their type
parameters too.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Guard cardinality-handler ctor against pathological inputs
- Both handlers now reject cardinalityLimit > 2^29 to prevent overflow
in the (cardinalityLimit * 2 - 1) capacity calc. Practical limits are
8..512 so this is well beyond any realistic configuration.
- TagCardinalityHandler's keys array is now String[] (was Object[]) to
match the actual contract -- minor clarity win.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make EMPTY the universal absent sentinel for AggregateEntry UTF8 fields
PropertyCardinalityHandler.register(null) now returns UTF8BytesString
.EMPTY. All AggregateEntry UTF8 fields are non-null. Callers stop
checking for null at every site.
- AggregateEntry: drop @Nullable on serviceSource/httpMethod/
httpEndpoint/grpcStatusCode (both the entry fields and the
Canonical scratch buffer). Drop @Nullable on getters and on the of
factory parameters. Drop the unused registerOrEmpty helper.
- Canonical.populate: each field is now this.field = HANDLER.register
(s.field) -- no inline conditionals.
- of() factory: drop the value == null ? null : createUtf8(value)
pattern; createUtf8 already returns EMPTY on null.
- SerializingMetricWriter: switch the four presence checks from !=
null to != EMPTY (identity comparison on the singleton).
Net win: nine identically-shaped call sites in Canonical.populate
and a smaller null surface across the package.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Use EMPTY consistently for absent values in peer-tag canonicalization
- TagCardinalityHandler.register now mirrors PropertyCardinalityHandler:
null input returns UTF8BytesString.EMPTY.
- Canonical.populatePeerTags now calls register for every schema slot
and tests the result against EMPTY rather than the input against null.
The wire-format buffer still holds only present peer tags (EMPTY is
elided), but the check is now consistent with how AggregateEntry's
scalar UTF8 fields handle absence.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten handler visibility + add tests for EMPTY-on-null contract
#4: PropertyCardinalityHandler and TagCardinalityHandler are only
consumed within this package; drop `public` from the class declarations,
constructors, and methods. They're package-private now.
#6: Add tests that lock down the EMPTY-on-null contract that the rest
of the package depends on:
- CardinalityHandlerTest covers both handlers: register(null) -> EMPTY,
and registering null repeatedly doesn't consume the cardinality
budget.
- AggregateEntryTest covers the entry shape: optional fields built
from a snapshot with null inputs resolve to EMPTY; populated
optional fields carry their value.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Notify on peer-tag cardinality blocks
Adds a per-cycle one-shot warn log + HealthMetrics counter
(`stats.tag_cardinality_blocked` with `tag:<name>`) when a peer-tag
value gets collapsed to the `blocked_by_tracer` sentinel because its
cardinality budget is exhausted. Implemented as a `register(int i,
String value)` method on `PeerTagSchema` that does the post-block
notification work; `TagCardinalityHandler` exposes `blockedSentinel()`
so the schema can identity-compare and stays free of logger / health
metric coupling. Warn-once gating uses a `Set<String>` of names seen
this cycle, cleared by `resetCardinalityHandlers()`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address PR #11387 review: dual-role docs, rename, @Nullable, consumer-side reconcile
- PropertyCardinalityHandler / TagCardinalityHandler: header comment
explaining the limiter-and-cache dual role and the prior-cycle reuse
trick that preserves UTF8 caching across resets.
- ClientStatsAggregator: rename peerAggSchema -> peerTagSchema across
field, method, and parameter; disambiguate the inner per-span local as
spanPeerTagSchema (return of peerTagSchemaFor).
- SpanSnapshot: replace prose "or null" docstrings with
javax.annotation.@Nullable on peerTagSchema/peerTagValues fields and
their constructor params.
- Consumer-side peer-tag reconciliation:
* DDAgentFeaturesDiscovery: drop State.peerTagsRevision + bump logic +
peerTagsRevision() accessor. Expose getLastTimeDiscovered().
* PeerTagSchema: rename peerTagsRevision -> lastTimeDiscovered, drop
final (consumer-thread-only mutation), add hasSameTagsAs(Set).
* ClientStatsAggregator: producer hot path is now a single volatile
read with a one-time synchronized bootstrap; resetCardinalityHandlers
runs reconcilePeerTagSchema first, which fast-paths on timestamp
equality and either bumps in place (preserving warm handlers when
the tag set is unchanged) or swaps in a fresh schema. The schema's
timestamp field no longer needs to be volatile because mutation is
confined to the aggregator thread.
Note: the @Nullable annotations on AggregateEntry's errorLatencies and
related fields only apply after the downstream lazy-init / Canonical
buffer work; those land in a separate commit on the downstream branches.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lock in cardinality-handler prior-cycle UTF8 reuse with explicit tests
Addresses PR #11387 review (test coverage gap):
- Fix misleading comment in propertyResetRefreshesBudget ("the previous
instances aren't reused") -- they ARE reused; the test only passed
because it asserted on .toString() content rather than identity.
- Add propertyPriorCycleInstancesAreReusedAcrossReset: explicit
assertSame check that registering the same value after a reset returns
the SAME UTF8BytesString instance from the prior cycle. This is the
"dual role as cache" property the canonical-key lookup depends on.
- Add propertyPriorCycleReuseSurvivesOneResetButNotTwo: nails down the
reuse window depth (one cycle, not two).
- Add tagPriorCycleInstancesAreReusedAcrossReset mirroring the property
handler test for the tag handler (cached "tag:value" UTF8BytesString).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hashtable: add missing braces and detach removed/replaced entries
Addresses PR #11409 review comments:
- #3267164119 / #3267165525: wrap every single-line if/break body in
braces (7 sites across BucketIterator, MutatingBucketIterator, and the
full-table Iterator).
- #3275947761 / #3275948108 (sarahchen6): null out the removed/replaced
entry's next pointer after splicing it out of the chain in
MutatingBucketIterator.remove / .replace. Applied the same fix to the
full-table Iterator.remove for consistency.
Rationale: detaching prevents accidental traversal through a removed
entry via a stale reference and lets the GC reclaim a chain tail that
the removed entry was the last referrer to.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Add Hashtable and LongHashingUtils to datadog.trace.util
Two general-purpose utilities used by the client-side stats aggregator
work (PR #11382 and follow-ups), extracted into their own change so the
metrics-specific PRs can build on a smaller, reviewable foundation.
- Hashtable: a generic open-addressed-ish bucket table abstraction
keyed by a 64-bit hash, with a public abstract Entry type so client
code can subclass it for higher-arity keys. The metrics aggregator
uses it to back its AggregateTable.
- LongHashingUtils: chained 64-bit hash combiners with primitive
overloads (boolean, short, int, long, Object). Used in place of
varargs combiners to avoid Object[] allocation and boxing on the
hot path.
No callers within internal-api itself yet -- the metrics aggregator PR
will introduce the first usages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add unit tests for Hashtable and LongHashingUtils
LongHashingUtilsTest (14 cases):
- hashCodeX null sentinel + non-null pass-through
- all primitive hash() overloads match the boxed Java hashCodes
- hash(Object...) 2/3/4/5-arg overloads match the chained addToHash
formula they are documented to constant-fold to
- addToHash(long, primitive) overloads match the Object-version
- linear-accumulation invariant (31 * h + v) holds across a sequence
- iterable / deprecated int[] / deprecated Object[] variants match
chained addToHash
- intHash treats null as 0 (observable via hash(null, "x"))
HashtableTest (24 cases across 5 nested classes):
- D1: insert/get/remove/insertOrReplace/clear/forEach, in-place value
mutation, null-key handling, hash-collision chaining with disambig-
uating equals, remove-from-collided-chain leaves siblings intact
- D2: pair-key identity, remove(pair), insertOrReplace matches on
both parts, forEach
- Support: capacity rounds up to a power of two, bucketIndex stays
in range across a wide hash sample, clear nulls every slot
- BucketIterator: walks only matching-hash entries in a chain, throws
NoSuchElementException when exhausted
- MutatingBucketIterator: remove from head-of-chain unlinks, replace
swaps the entry while preserving chain, remove() without prior
next() throws IllegalStateException
Tests live in internal-api/src/test/java/datadog/trace/util and use the
already-present JUnit 5 setup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply spotless formatting to Hashtable and LongHashingUtils
Bring the new util/ files in line with google-java-format
(tabs → spaces, line wrapping, javadoc list markup) so
spotlessCheck passes in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add JMH benchmarks for Hashtable.D1 and D2
Compares Hashtable.D1 and Hashtable.D2 against equivalent HashMap
usage for add, update, and iterate operations. Each benchmark thread
owns its own map (Scope.Thread), but @Threads(8) is used so the
allocation/GC pressure that Hashtable is designed to avoid surfaces
in the throughput numbers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add benchmark results to HashtableBenchmark header
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Guard Support.sizeFor against overflow and use Integer.highestOneBit;
reject capacities above 1 << 30 instead of looping forever.
- Add braces around single-statement while bodies in BucketIterator.
- Split HashtableBenchmark into HashtableD1Benchmark / HashtableD2Benchmark.
- Add regression tests for Support.sizeFor bounds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix dropped argument in HashingUtils 5-arg Object hash
The 5-arg Object overload was forwarding only obj0..obj3 to the int
overload, silently dropping obj4. Also align LongHashingUtils.hash 3-arg
signature with its 2/4/5-arg siblings (int parameters) and strengthen
the 5-arg HashingUtilsTest to detect the missing-arg regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address review feedback on Hashtable
- Split D1Tests and D2Tests into HashtableD1Test and HashtableD2Test;
extract shared test entry classes into HashtableTestEntries.
- Reduce visibility of LongHashingUtils.hash(int...) chaining overloads
to package-private; they are internal building blocks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop reflection in iterator tests via package-private D1.buckets
The iterator tests need a populated Hashtable.Entry[] to drive
Support.bucketIterator / mutatingBucketIterator. Relaxing D1.buckets
from private to package-private lets the same-package tests read it
directly, removing the reflection helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add context-passing forEach to Hashtable.D1 and D2
Mirrors the TagMap pattern: pairs the existing forEach(Consumer) with a
forEach(T context, BiConsumer<T, TEntry>) overload so callers can hand
side-band state to a non-capturing lambda and avoid the
fresh-Consumer-per-call allocation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move forEach loop body to Support helper
Factors the unchecked (TEntry) cast out of D1.forEach / D2.forEach (and
the BiConsumer variants) into Support.forEach(buckets, ...). The cast
now lives in one place, mirroring how Entry.next() handles it, and the
D1/D2 methods become one-liners. Downstream higher-arity tables built
on Support gain the same helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move bucket-head cast to Support.bucket helper
Adds Support.bucket(buckets, keyHash) which returns the bucket head
already cast to the caller's concrete entry type. D1.get and D2.get
now drop the raw-Entry intermediate variable and walk the chain via
Entry.next() directly. The unchecked cast lives in one place,
consistent with Entry.next() and Support.forEach.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop d1_/d2_ prefix from per-table benchmark methods
Holdover from when both lived in a shared HashtableBenchmark; redundant
now that each lives in its own class.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add Hashtable.Support helpers: MAX_RATIO, insertHeadEntry, MutatingTableIterator
Three consumer-facing helpers that callers building higher-arity tables on
top of Hashtable.Support kept open-coding:
- MAX_RATIO_NUMERATOR / _DENOMINATOR: the 4/3 multiplier for sizing a
bucket array from a target working-set under a 75% load factor.
- insertHeadEntry(buckets, bucketIndex, entry): the (setNext + array-store)
pair for splicing a new entry at the head of a bucket chain.
- MutatingTableIterator + Support.mutatingTableIterator(buckets): walks
every entry in the table (not filtered by hash) with remove() support,
for sweeps like eviction and expunge that aren't keyed to a specific
hash. Sibling of MutatingBucketIterator.
Tests cover the table-wide iterator at head-of-bucket and mid-chain
removal, empty buckets between live entries, exhaustion, and
remove-without-next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Swap MAX_RATIO numerator/denominator pair for a single float + scaled create()
Replace Support.MAX_RATIO_NUMERATOR / _DENOMINATOR with a single float
MAX_RATIO constant, and add a Support.create(int, float) overload that
takes a scale factor. Callers now write Support.create(n, MAX_RATIO)
instead of stitching together the int arithmetic at the call site.
The scaled size is truncated (not ceiled) before going through sizeFor.
sizeFor already rounds up to the next power of two, so truncation just
absorbs float fuzz that would otherwise push a result like 12 * 4/3 =
16.0000005f past 16 and double the bucket array size for no reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Hashtable docs + rename MAX_CAPACITY to MAX_BUCKETS
Five small cleanups from a design re-review pass:
1. Support javadoc: drop the stale "methods are package-private" sentence;
most of them were made public in earlier commits for higher-arity
callers. Also drop the "nested BucketIterator" framing (iterators are
peers of Support inside Hashtable, not nested inside Support).
2. MAX_RATIO javadoc: drop the Math.ceil recommendation; create(int, float)
deliberately truncates and is the canonical pathway.
3. Document the null-hash treatment on D1.Entry.hash and D2.Entry.hash so
the behavior difference is explicit: D1 uses Long.MIN_VALUE as a
sentinel that's collision-free against any int-valued hashCode(); D2
has no such sentinel and relies on matches() to resolve null/null vs
hash-0 collisions.
4. Rename Support.MAX_CAPACITY -> MAX_BUCKETS and sizeFor's parameter to
requestedSize. The cap is on the bucket-array length, not entry count;
the new name reflects that. Error messages updated to match.
5. Drop the `abstract` modifier on Hashtable in favor of `final` with a
private constructor. Nothing actually subclasses Hashtable -- the
abstract was a namespace device that read as "intended for extension."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dedupe chain-head splice in D1/D2 via keyHash insertHeadEntry overload
- Add Support.insertHeadEntry(buckets, long keyHash, entry) overload that
derives the bucket index itself. Callers that already have a hash but
not the index (the common case) now avoid the redundant bucketIndex(...)
hop.
- D1.insert, D1.insertOrReplace, D2.insert, D2.insertOrReplace: use the
new overload, drop the (thisBuckets local, bucketIndex compute,
setNext, store) sequence at each call site.
- D2.buckets: drop the `private` modifier to match D1.buckets. Both are
package-private so iterator tests in the same package can drive
Support.bucketIterator against the table's bucket array. Added a short
comment on both fields documenting the rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten Entry.next encapsulation; doc hasNext; add D1/D2 getOrCreate
Three follow-ups from the design review:
- Make Hashtable.Entry.next private. All same-package readers
(BucketIterator) already had a next() accessor; the leftover direct
field reads now route through it. Closes the "mixed encapsulation"
gap where some readers used the accessor and same-package ones
reached for the field.
- BucketIterator and MutatingBucketIterator now document that chain-walk
work happens in next() (and the constructor for the first match);
hasNext() is an O(1) field read.
- Add D1.getOrCreate(K, Function) and D2.getOrCreate(K1, K2, BiFunction).
Both reuse the lookup hash for the insert on miss, avoiding the
double-hash that "get; if null then insert" callers would otherwise
pay.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hashtable: add missing braces and detach removed/replaced entries
Addresses PR #11409 review comments:
- #3267164119 / #3267165525: wrap every single-line if/break body in
braces (7 sites across BucketIterator, MutatingBucketIterator, and the
full-table Iterator).
- #3275947761 / #3275948108 (sarahchen6): null out the removed/replaced
entry's next pointer after splicing it out of the chain in
MutatingBucketIterator.remove / .replace. Applied the same fix to the
full-table Iterator.remove for consistency.
Rationale: detaching prevents accidental traversal through a removed
entry via a stale reference and lets the GC reclaim a chain tail that
the removed entry was the last referrer to.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename LongHashingUtils.hashCodeX(Object) to hash(Object) for API consistency
Addresses PR #11409 review comment #3276167001. The method parallels the
primitive hash(boolean) / hash(int) / hash(long) / ... family, so naming
it hash(Object) -- with null collapsing to Long.MIN_VALUE as a sentinel
distinct from any real hashCode -- matches the rest of the public surface.
Test call sites that pass a literal null now disambiguate against
hash(int[]) / hash(Object[]) / hash(Iterable) via an (Object) cast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge remote-tracking branch 'origin/master' into dougqh/optimize-metric-key
Merge branch 'dougqh/util-hashtable' into dougqh/optimize-metric-key
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-cardinality
Merge remote-tracking branch 'origin/master' into dougqh/conflating-metrics-background-work
Introduce slim PeerTagSchema; capture peer-tag values not pairs
Addresses sarahchen6's review comment on ConflatingMetricsAggregator
extractPeerTagPairs: replaces the worst-case-allocation + trim-and-copy
flat-pairs layout with a parallel-array carrier.
- New PeerTagSchema: minimal carrier of String[] names. Two flavors -- a
static INTERNAL singleton (one entry: base.service) for internal-kind
spans, and per-discovery built schemas for client/producer/consumer
spans. Deliberately no cardinality limiters or per-cycle state; that
layers on top in a later PR.
- ConflatingMetricsAggregator: caches the peer-aggregation schema keyed
on reference equality of features.peerTags() -- a single volatile read
+ a long compare on the steady-state producer hot path, no allocation.
The producer now captures only a String[] of values parallel to the
schema's names; the schema reference is carried on SpanSnapshot. The
prior "build worst-case pairs then trim" code is gone.
- SpanSnapshot: replaces String[] peerTagPairs with PeerTagSchema +
String[] peerTagValues. Producer drops the schema reference if no
values fired so the consumer short-circuits on null.
- Aggregator.materializePeerTags: now reads name/value pairs at the same
index from (schema.names, snapshot.peerTagValues). Counts hits once
for exact-size allocation; preserves the singletonList fast path for
the common one-entry case (e.g. internal-kind base.service).
Producer-side cost goes from "allocate String[2n] + walk + maybe trim"
to "single volatile read + walk + lazy String[n] only on first hit".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-cardinality
Address PR #11381 review (round 2)
- Aggregator.materializePeerTags: fold the firstHit-discovery nested if
into a single guarded post-increment (amarziali, #3279243138). One
body line: `if (values[i] != null && hitCount++ == 0) firstHit = i;`.
- Drop redundant isKind(SpanKindFilter) overrides in both
TraceGenerator.groovy files (amarziali, #3279264553 / #3279382648).
CoreSpan.java:84 already supplies a default implementation that reads
the same span.kind tag.
- Bump TRACER_METRICS_MAX_PENDING default from 2048 -> 131072 to address
the capacity regression amarziali flagged (#3279378375). Without
producer-side conflation, the inbox now holds 1 SpanSnapshot per
metrics-eligible span instead of 1 conflated Batch per ~64 spans;
restoring effective capacity parity (~2048 * ~64 = 131072) prevents a
~64x rise in inbox-full drops at the same span rate. ~100 B per
SpanSnapshot puts the worst-case heap floor at ~13 MB -- bounded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cover inbox-full fast-path in ConflatingMetricsAggregator.publish
Addresses PR #11381 review (amarziali, #3279325340 -- "Are the existing
tests covering this case?").
New ConflatingMetricsAggregatorInboxFullTest constructs the aggregator
with a small inbox (queueSize=8), deliberately does NOT call start() so
the consumer thread never drains, then publishes enough spans to
overflow the inbox. Verifies that healthMetrics.onStatsInboxFull() is
called at least once -- the fast-path's `inbox.size() >= inbox.capacity()`
short-circuit triggers when the producer-side queue is at capacity.
Test is Java + JUnit 5 + Mockito per the project convention for new
tests; uses a CoreSpan Mockito mock rather than the SimpleSpan Groovy
fixture so we don't depend on Groovy-then-Java compile order from the
test source set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reconcile PeerTagSchema once per reporting cycle on the aggregator thread
Addresses amarziali's review comment #3279340181 ("It would be more
efficient to trigger from the other side"). The producer-side reference
compare on every publish goes away; the aggregator thread reconciles
the cached schema against feature discovery once per reporting cycle.
- DDAgentFeaturesDiscovery: expose getLastTimeDiscovered() so callers
can detect a discovery refresh without copying the peerTags Set.
- PeerTagSchema: add `long lastTimeDiscovered` (plain, aggregator-only)
and `hasSameTagsAs(Set)`. of(Set, long) takes the timestamp; INTERNAL
uses a -1L sentinel since it's never reconciled.
- ConflatingMetricsAggregator:
* Drop the cachedPeerTagsSource volatile and the per-publish reference
compare.
* Producer fast path is now `cachedPeerTagSchema` volatile read +
null-check; first publish takes the one-time synchronized bootstrap.
* Add reconcilePeerTagSchema() that runs once per cycle on the
aggregator thread: fast-path timestamp compare, slow-path set
compare, bump-in-place when the set is unchanged.
- Aggregator: new `Runnable onReportCycle` constructor parameter, run at
the start of report() (before the flush, so any test awaiting
writer.finishBucket() observes the schema in its post-reconcile state
and so the next publish sees the new schema without a handoff).
- Update "should create bucket for each set of peer tags" to drive two
reporting cycles separated by a report() that triggers reconcile. The
old test relied on per-publish reference detection, which the new
design intentionally doesn't preserve -- the schema is now stable
within a cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-cardinality
Add bootstrap + reconcile coverage for PeerTagSchema
Addresses round-3 review nice-to-haves on PR #11381.
- PeerTagSchemaTest: unit coverage for hasSameTagsAs() (the predicate
that drives the reconcile fast/slow path split), the of(Set, long)
factory, and the INTERNAL singleton. The hasSameTagsAs cases include
same-content-different-Set-reference (the case the reconcile fast path
relies on after a discovery refresh) and content-mismatch in either
direction.
- ConflatingMetricsAggregatorBootstrapTest: integration coverage for
the producer-side bootstrap + aggregator-thread reconcile flow.
* bootstrapHappensOnceOnFirstPublish -- three publishes against an
un-started aggregator (no consumer thread, no reconciles); verifies
features.peerTags() and features.getLastTimeDiscovered() are each
called exactly once.
* reconcileSkipsDeepCompareWhenTimestampMatches -- two cycles with
constant features.getLastTimeDiscovered(); each post-report
reconcile short-circuits on the timestamp fast path, so peerTags()
is called only by bootstrap (1 total).
* reconcileSurvivesTimestampBumpWhenTagsUnchanged -- timestamps bump
every reconcile, forcing the slow set-compare path; the tag set
stays identical, so the schema is preserved and continues to flush
buckets correctly across cycles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Use writer.finishBucket() count in bootstrap test for cascade compatibility
The verify(writer).add(MetricKey, AggregateMetric) signature is unique
to #11381; downstream branches use AggregateEntry. Switching to
verify(writer, times(2)).finishBucket() keeps the same behavioral
guarantee (both cycles flushed) across the stack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Use writer.finishBucket() count in bootstrap test for cascade compatibility
The verify(writer).add(MetricKey, AggregateMetric) signature is unique
to #11381; downstream branches use AggregateEntry. Switching to
verify(writer, times(2)).finishBucket() keeps the same behavioral
guarantee (both cycles flushed) across the stack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/optimize-metric-key' into dougqh/control-tag-cardinality
Rename bootstrap test to ClientStatsAggregator + adapt PeerTagSchemaTest
#11387's ClientStatsAggregator renames ConflatingMetricsAggregator; the
test file's name and class refs need to match. PeerTagSchemaTest's
PeerTagSchema.of() calls need the (Set, long, HealthMetrics) signature
this branch introduced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'master' into dougqh/conflating-metrics-background-work
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Preserve TRACER_METRICS_MAX_PENDING semantic + drop stale imports
TRACER_METRICS_MAX_PENDING previously counted conflating Batch slots
(~64 spans each). The inbox now holds 1 SpanSnapshot per slot, so
multiply the configured value by LEGACY_BATCH_SIZE (64) to keep
pre-existing customer overrides delivering the same effective
span-throughput capacity. Default stays at 2048 logical -> 131072
snapshot slots, identical to the prior 2048 batches * 64 spans.
Also drops two unused datadog.trace.core.SpanKindFilter imports left
behind in TraceGenerator.groovy after the isKind() override was removed
in favor of the CoreSpan default implementation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Add AdversarialMetricsBenchmark for capacity-bound stress testing
Ports the adversarial JMH benchmark from #11402 down to this branch so
we can compare #11381 vs master on a high-cardinality, high-throughput
workload. Adapted to use ConflatingMetricsAggregator (pre-rename) and
the FixedAgentFeaturesDiscovery / NullSink helpers already in
ConflatingMetricsAggregatorBenchmark.
8 producer threads hammer publish() with unique (service, operation,
resource, peer.hostname) per op so the aggregate cache fills+evicts
continuously and the inbox saturates. tearDown prints the drop
counters (inboxFull vs aggregateDropped) so the test verifies the
subsystem stayed bounded under attack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Trim AdversarialMetricsBenchmark counters and clarify printout
Drop traceComputedCalls / totalSpansCounted: under 8-way contention
the volatile-long ++/+= pattern was losing ~20% of updates (296M
counted vs 245M reported), and the numbers duplicate signal JMH's
ops/s already provides.
Switch inboxFull / aggregateDropped to LongAdder so the printed drop
shape (the order-of-magnitude story the bench is built to tell) is
accurate under contention.
Replace the stale "both forks combined for this run" string with text
that matches the actual @Fork(value=1) config and notes that counters
accumulate across warmup + measurement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Close PeerTagSchema reconcile race + cover the swap branch
buildPeerTagSchema previously read features.peerTags() before
features.getLastTimeDiscovered(). DDAgentFeaturesDiscovery exposes
those as two separate accessors against its volatile State -- a
state-swap interleaving could leave the cached schema tagged with a
NEWER timestamp than its names, after which the next reconcile
short-circuits on the timestamp compare and misses the tag-set update
until the next discovery refresh (~minute later).
Swap the read order so timestamp is captured first. With this
ordering, an interleaving leaves the schema OLDER than its names
instead -- the next reconcile sees a timestamp mismatch, runs the
deep compare, and self-heals on the very next cycle.
Also adds reconcileSwapsSchemaWhenTagSetChanges, which closes the
test gap on the slow-path swap branch
(cachedPeerTagSchema = PeerTagSchema.of(...)). End-to-end check via
the writer's captured MetricKeys: pre-swap snapshot carries only
peer.hostname, post-swap snapshot carries both peer.hostname and
peer.service.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
Adapt reconcileSwapsSchemaWhenTagSetChanges to AggregateEntry shape
#11382 collapses MetricWriter.add(MetricKey, AggregateMetric) into
add(AggregateEntry). Re-target the captor and accessors on this branch
so the test compiles and the same end-to-end peer-tag verification
holds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Clarify materializePeerTags hit-counting loop
Splits the `if (values[i] != null && hitCount++ == 0)` conjunction
into nested ifs. Same semantics, no codegen impact after JIT --
just visibly says what the loop is doing rather than relying on
post-increment-inside-conjunction. Closes amarziali's review thread
on this block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merge branch 'dougqh/conflating-metrics-background-work' into dougqh/optimize-metric-key
# Conflicts:
# dd-trace-core/src/main/java/datadog/trace/common/metrics/Aggregator.java
Fix MetricsIntegrationTest entry recording call site
AggregateEntry consolidated MetricKey + AggregateMetric so recordDurations
lives directly on AggregateEntry now. The previous entry1.aggregate.
recordDurations(...) form compiles under Groovy's dynamic dispatch but
would throw MissingPropertyException at runtime since there is no
`aggregate` property. Resolves chatgpt-codex-connector's review comment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make ConflatingMetricAggregatorTest counter checks actually verify
The `1 * writer.add(value) >> { closure }` pattern treats the closure
as a stubbed return value -- Spock evaluates it but discards the
result, so `e.getHitCount() == X && ...` was a silent no-op across
31 occurrences. Wrapping the expression in `assert` makes Groovy's
power-assert throw on mismatch, which Spock surfaces as a real
failure. Resolves chatgpt-codex-connector's review comment.
All 41 tests still pass, so the previously-unverified assertions
happened to hold.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop dead recordDurations(int, AtomicLongArray) batch API
This method was a vestige of master's Batch design where multiple
producer threads wrote into an AtomicLongArray slot concurrently and
the aggregator drained ~64 durations per Batch in one call. The new
producer/consumer split publishes one SpanSnapshot per span, so
production only ever calls recordOneDuration(long).
Migrate the three remaining callers (AggregateEntryTest,
SerializingMetricWriterTest, MetricsIntegrationTest) to a loop of
recordOneDuration(long) calls, then delete the batched method and its
AtomicLongArray imp…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What This Does
Adds support for span derived primary tags - via
DD_TRACE_ADDITIONAL_TAGSMotivation
From RFC...
Additional Notes
Stack: master → #11382 → #11478 → #11387 → this PR. (#11389 was closed — its memory-efficiency work was folded into #11478's lazy-histogram +
namesHashcaching.) Implements Client-Side Stats v1.3.0 span-derived primary tags on the new producer/consumerClientStatsAggregatorarchitecture. Users configureDD_TRACE_STATS_ADDITIONAL_TAGS(comma-separated tag keys); the tracer extracts the matching span tag values and includes them as additional aggregation dimensions onClientGroupedStats.AdditionalMetricTags.All canonicalization (
UTF8BytesStringinterning, per-tag cardinality cap) runs on the aggregator thread; the producer just captures raw values into aString[]parallel to the schema.Design
Wire format: new
AdditionalMetricTagsfield onClientGroupedStats, emitted asrepeated stringof"<key>:<value>"entries (mirrorsPeerTags). Schema-ordered (alphabetical by key); null slots skipped; field omitted when no slots are populated so customers who don't configure additional tags pay zero payload overhead.Configuration:
DD_TRACE_STATS_ADDITIONAL_TAGS/dd.trace.stats.additional.tags— comma-separated tag keys.Cardinality protection —
AdditionalTagsSchemaholds oneTagCardinalityHandlerper configured key, exactly mirroringPeerTagSchema:MAX_ADDITIONAL_TAG_KEYS = 10— configured-key count cap. Excess keys dropped at startup with a warn log.MetricCardinalityLimits.ADDITIONAL_TAG_VALUE = 512— per-key distinct-value cap per reporting cycle. Once a key's budget is exhausted, further new values get the per-key"<key>:blocked_by_tracer"sentinel (when limits are enabled) so they collapse into one bucket instead of fragmenting the table.:are rejected with a warn log; the key set is sorted + deduped."key:value"string is re-allocated.Threading: the per-key
TagCardinalityHandlers and their block counters are aggregator-thread-only.AdditionalTagsSchema.resetHandlers()runs once per reporting cycle alongside the property/peer handler resets, flushing each key's block count toHealthMetrics.onTagCardinalityBlocked.What's new vs. PoC (#11358)
unsafeGetTag(name)per configured key →String[]parallel to the schema.TagCardinalityHandler(same component as peer tags) rather than a bespoke limiter — gets UTF8 caching for free and removes the separate global stat-entry counter the PoC used.UTF8BytesString[]on the entry, writing each non-null slot directly — no per-write byte composition.New files
dd-trace-core/src/main/java/datadog/trace/common/metrics/AdditionalTagsSchema.javadd-trace-core/src/test/java/datadog/trace/common/metrics/AdditionalTagsSchemaTest.javadd-trace-core/src/test/java/datadog/trace/common/metrics/AggregateTableAdditionalTagsTest.javadd-trace-core/src/test/java/datadog/trace/common/metrics/SerializingMetricWriterAdditionalTagsTest.javadd-trace-core/src/jmh/java/datadog/trace/common/metrics/AdditionalTagsMetricsBenchmark.javaHealth metric
HealthMetrics.onTagCardinalityBlocked(String tag, long count)— fires per key per cycle for any key whose per-cycle value budget was exceeded (shared with peer tags and property fields).TracerHealthMetricssurfaces this asstats.cardinality_blocked(tagged by field).Benchmarks
No-additional-tags baseline (8 producer threads, 2×15s warmup + 5×15s)
HighCardinalityResourceMetricsBenchmark/HighCardinalityPeerMetricsBenchmarkset no additional tags, so they measure the cost of the plumbing being threaded through but not populated. On a clean machine (2026-06-03) this PR was at parity with #11387 and master (38–43M ops/s band), confirming the per-snapshot path is a single null check + early return inCanonical.populateAdditionalTagswhen no schema is configured.Additional-tags path (
AdditionalTagsMetricsBenchmark, new)Configures two keys and generates unbounded distinct values so the per-key budget saturates immediately. Runs both
limitsEnabledarms via@Param. 3-fork run with-prof gc(2026-06-03):limitsEnabledThe two arms are not a fair "cost of limiting" comparison and should not be read as a regression. With unbounded distinct values they diverge into different downstream behavior, not just a different branch in
register:limitsEnabled=true: every over-budget span collapses to the single"<key>:blocked_by_tracer"sentinel entry →findOrInsertalways hits a live entry →recordOneDuration(a DDSketch histogram insert) runs for every drained span.limitsEnabled=false: every over-budget value canonicalizes to a distinct entry → the table saturates atmaxAggregates→ most subsequent spans are dropped atfindOrInsert, never reaching the histogram.So
limitsEnabled=truemeasures lower throughput here precisely because it does more useful work per span (keeps the masked data and records it) where the disabled arm drops the overflow. The GC profile confirms this: limits=true allocates more per op (888 vs 820 B/op) — that delta is the histogram recording, not the sentinel path (which allocates nothing). My initial hypothesis (that the disabled arm would allocate more from freshUTF8BytesStringcreation) was wrong — the drop path short-circuits before those allocations matter.A production workload with a bounded value set never saturates the budget and sees neither arm's overflow behavior, which is why
HighCardinalityResourceis at parity with limits on/off. Throughput CIs here are wide (>20%, single-machine thermal drift across the sequential@Paramarms); the per-op allocation figures are the reliable signal.Test plan
:dd-trace-core:test --tests "datadog.trace.common.metrics.*"— all pass (existing + three new test files for the feature).:-containing keys.ADDITIONAL_TAG_VALUEcollapse to the per-key sentinel; existing entries still hit normally.AdditionalMetricTagsfield present with schema-ordered"key:value"entries; omitted when nothing matches; null slots skipped.Notes for reviewers
masterandConflatingMetricsAggregator. This branch sits on Per-component / tag cardinality limits in client-side stats #11387 and ships against the producer/consumer-splitClientStatsAggregator.setMeasuredTag(key, value)programmatic API,DD_TAGShandling, per-value length cap (removed from this revision — revisit if production shows oversized values).🤖 Generated with Claude Code