Add RxJava 3 instrumentation by ValentinZakharov · Pull Request #11849 · DataDog/dd-trace-java

ValentinZakharov · 2026-07-02T22:27:51Z

What Does This Do

We already trace RxJava 1 and 2 but not 3. RxJava 3 moved its types to the io.reactivex.rxjava3.core.* namespace, so the existing rxjava-2.0 instrumentation never matches it. This adds a new rxjava-3.0 module that brings RxJava 3 to parity.

What's in it:

New dd-java-agent/instrumentation/rxjava/rxjava-3.0 module. It ports the rxjava-2.0 logic to the RxJava 3 namespace: context capture on the five reactive types (Flowable, Observable, Single, Maybe, Completable) and re-attachment around subscriber callbacks, plus the async result extension that finishes @WithSpan / @Trace spans when the stream completes, errors, or is cancelled.
Muzzle has a pass for rxjava3 [3.0.0,) and a fail block that asserts the advice can never match the rxjava2 artifact, so the two versions stay isolated. The module also pulls in rxjava-2.0 as a test runtime dependency to confirm both instrumenters coexist.
Registered in the GraalVM native image build time list next to the rxjava2 entry.
Tests are JUnit 5 (44 cases): subscription propagation, the core propagation suite, the @WithSpan result extension across all five types, and a Java 8 interop check (fromCompletionStage, fromStream, fromOptional). latestDepTest runs the same suite against the newest published rxjava3.

One thing reviewers should look at closely: this also touches the shared java-concurrent module. While testing delayed chains I found that RxJava 3 added io.reactivex.rxjava3.internal.schedulers.AbstractDirectTask, whose static initializer builds two sentinel FutureTask instances. When that initializer first runs under an active trace (the first delay or timeout hop inside a span), the executor instrumentation captures a continuation on those static singletons that never gets cancelled, so the trace stays pending and is never reported. ~~RxJava 2 has no equivalent class, which is why the byte for byte equivalent rxjava-2.0 code never hit this~~(correction: 2.0.9+ seem to have it). The fix disables async propagation while that type initializer runs, following the same pattern already used for Reactor's SchedulerTask and WorkerTask. The matcher is an exact class name, so it stays inert unless RxJava 3 is on the classpath.

The Java 8 interop check found no propagation gaps once that fix was in place.

Motivation

Additional Notes

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Add your completed PR to the merge queue by commenting /merge. You can also:
- Customize the commit message associated with the merge with /merge --commit-message "..."
- Remove your PR from the merge queue with /merge -c
- Skip all merge queue checks with /merge -f --reason "reason"; please use this judiciously, as some checks do not run at the PR-level (note: the PR still needs to be mergeable, this will only skip the pre-merge build)
- Get more information in this doc

Jira ticket: [PROJ-IDENT]

dd-octo-sts · 2026-07-02T22:43:58Z

🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)

Suite	Status
Startup	🟡 warning

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.95 s	13.95 s	[-0.9%; +0.9%] (no difference)
startup:insecure-bank:tracing:Agent	12.99 s	13.00 s	[-1.1%; +0.8%] (no difference)
startup:petclinic:appsec:Agent	17.01 s	16.77 s	[+0.2%; +2.7%] (maybe worse)
startup:petclinic:iast:Agent	16.95 s	16.90 s	[-0.6%; +1.1%] (no difference)
startup:petclinic:profiling:Agent	16.71 s	16.86 s	[-1.8%; +0.1%] (no difference)
startup:petclinic:sca:Agent	16.96 s	16.70 s	[+0.8%; +2.3%] (maybe worse)
startup:petclinic:tracing:Agent	16.03 s	15.63 s	[-1.8%; +6.9%] (no difference)

Commit: a6cb915e · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

PerfectSlayer

Reviewed structure and tests. Left a bunch of comments.

PerfectSlayer · 2026-07-03T06:18:45Z

+// NOTE: This test lives in the `testdog` package (not `datadog`) on purpose: the agent ignores
+// `datadog.*` classes for instrumentation, so `@Trace`-annotated methods declared under `datadog.*`
+// would never be instrumented. See RxJava3Test for the same convention.


🧹 chore: Please clean up internal comments and LLM reasoning‏

Cleared, thanks

PerfectSlayer · 2026-07-03T06:22:13Z

+            SORT_BY_START_TIME,
+            span().root().operationName("interop-parent").resourceName("interop-parent"),
+            span()
+                .childOf(Worker.parentId)


🎯 suggestion: ‏There is a childOfIndex() that would simplify the test case, cleaning up parentId, activeSpan().getSpanId(), etc... This is applicable to multiple parts in the test files.

Oh, I didn't know that - looks like childOfIndex() method was introduced quite recently
Thanks

PerfectSlayer · 2026-07-03T06:24:09Z

+import org.junit.jupiter.params.provider.EnumSource;
+
+@WithConfig(key = "trace.otel.enabled", value = "true")
+@WithConfig(key = "integration.opentelemetry-annotations-1.20.enabled", value = "true")


🧹 chore: ‏‏This is a duplicate of trace.otel.enabled

Duplication removed

PerfectSlayer · 2026-07-03T06:26:34Z

+    String method = "traceAsync" + type.type;
+    assertTraces(
+        trace(
+            otelSpan("RxJava3TracedMethods." + method)


👏 praise: ‏Interesting use of factory method

PerfectSlayer · 2026-07-03T06:27:09Z

+// NOTE: This test lives in the `testdog` package (not `datadog`) on purpose: the agent ignores
+// `datadog.*` classes for instrumentation, so `@Trace`-annotated methods declared under `datadog.*`
+// would never be instrumented. See the java-lang-21 tests for the same convention.


PerfectSlayer · 2026-07-03T06:50:30Z

+  /**
+   * RxJava 3's AbstractDirectTask creates FINISHED/DISPOSED sentinel FutureTask instances in its
+   * static initializer. If that initializer runs while a trace is active (e.g. the first scheduled
+   * delay/timeout under a span), the executor instrumentation captures a continuation on those
+   * static singletons that is never cancelled, leaking the pending trace. Disable async propagation
+   * while the type initializer runs.
+   */


🧹 chore: We don't want LLM reasoning here, here is a trimmed version to keep relevant info only:‏

Suggested change

/**

* RxJava 3's AbstractDirectTask creates FINISHED/DISPOSED sentinel FutureTask instances in its

* static initializer. If that initializer runs while a trace is active (e.g. the first scheduled

* delay/timeout under a span), the executor instrumentation captures a continuation on those

* static singletons that is never cancelled, leaking the pending trace. Disable async propagation

* while the type initializer runs.

*/

/**

* RxJava 3's AbstractDirectTask creates FINISHED/DISPOSED sentinel FutureTask instances in its

* static initializer.

*/

Java docs was cleared and simplified

PerfectSlayer · 2026-07-03T06:54:13Z

+    @Advice.OnMethodExit(suppress = Throwable.class)
+    public static void onConstruct(@Advice.This final Maybe<?> maybe) {
+      Context parentContext = Java8BytecodeBridge.getCurrentContext();
+      if (parentContext != null && parentContext != Java8BytecodeBridge.getRootContext()) {


🧹 chore: ‏parentContext will never be null.

Good catch, fixed

PerfectSlayer · 2026-07-03T06:55:23Z

+    @Advice.OnMethodExit(suppress = Throwable.class)
+    public static void onConstruct(@Advice.This final Observable<?> observable) {
+      Context parentContext = Java8BytecodeBridge.getCurrentContext();
+      if (parentContext != null && parentContext != Java8BytecodeBridge.getRootContext()) {


🧹 chore: ‏parentContext will never be null.

Good catch, fixed

PerfectSlayer · 2026-07-03T06:56:42Z

+  @Override
+  public Map<String, String> contextStore() {
+    final Map<String, String> store = new HashMap<>();
+    store.put("io.reactivex.rxjava3.core.Flowable", Context.class.getName());


🎯 suggestion: ‏Use temporary constant for context class name?

Agree, this way much better

PerfectSlayer · 2026-07-03T06:57:12Z

+    @Advice.OnMethodExit(suppress = Throwable.class)
+    public static void onConstruct(@Advice.This final Single<?> single) {
+      Context parentContext = Java8BytecodeBridge.getCurrentContext();
+      if (parentContext != null && parentContext != Java8BytecodeBridge.getRootContext()) {


🧹 chore: ‏parentContext will never be null.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…izer RxJava 3 introduced io.reactivex.rxjava3.internal.schedulers.AbstractDirectTask, whose static initializer constructs FINISHED/DISPOSED sentinel FutureTask instances. When that initializer first runs under an active trace (e.g. the first delay/timeout scheduler hop inside a span), the executor instrumentation captures a ScopeContinuation on those static singletons that is never cancelled, leaking the pending trace so it is never reported. Disable async propagation while AbstractDirectTask's type initializer runs, mirroring the existing reactor.core.scheduler.SchedulerTask/WorkerTask handling. The matcher is inert unless RxJava 3 is on the classpath. RxJava 2 has no equivalent class. Restores the delayed-Maybe coverage in RxJava3Test, which fails without this fix and passes with it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ava3 Two tests were under datadog.trace.instrumentation.rxjava3 while the @Trace-using tests must live under testdog.* (the agent ignores datadog.* for instrumentation). Move all four tests to testdog.* for consistency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…dvice

…servable and Single

…Observer

…a3ResultExtensionTest

…ct mode

…oss the codebase

…n why

…ration

PerfectSlayer

Thanks for the follow up changes!

dd-octo-sts · 2026-07-03T10:08:23Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-03T10:08:26Z

View all feedbacks in Devflow UI.

2026-07-03 10:08:26 UTC ℹ️ Start processing command /merge

2026-07-03 10:08:31 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-07-03 12:09:05 UTC ❌ MergeQueue: The build pipeline has timeout

The merge request has been interrupted because the build 64026140340904028 took longer than expected. The current limit for the base branch 'master' is 120 minutes.

Possible reasons:

some mandatory checkruns are failing:
- Check pull request labels

ValentinZakharov · 2026-07-03T14:14:37Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-03T14:14:40Z

View all feedbacks in Devflow UI.

2026-07-03 14:14:40 UTC ℹ️ Start processing command /merge

2026-07-03 14:14:45 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-07-03 15:46:42 UTC ℹ️ MergeQueue: This merge request was merged

ValentinZakharov self-assigned this Jul 2, 2026

ValentinZakharov requested review from a team as code owners July 2, 2026 22:27

ValentinZakharov requested review from PerfectSlayer and removed request for a team July 2, 2026 22:27

ValentinZakharov added the type: enhancement Enhancements and improvements label Jul 2, 2026

ValentinZakharov requested review from ygree and removed request for a team July 2, 2026 22:27

ValentinZakharov added inst: others All other instrumentations tag: ai generated Largely based on code generated by an AI or LLM labels Jul 2, 2026

ygree approved these changes Jul 2, 2026

View reviewed changes

PerfectSlayer reviewed Jul 3, 2026

View reviewed changes

amarziali and others added 16 commits July 3, 2026 10:37

Scaffold rxjava-3.0 instrumentation module

dcdd946

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add rxjava-3.0 Tracing observer/subscriber wrappers

49cf92d

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add rxjava-3.0 type instrumentations, module and async result extension

55ad198

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Register rxjava3 async result extension for Graal native-image

1f65614

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add rxjava-3.0 SubscriptionTest

49c7174

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add rxjava-3.0 core context propagation test

c21e594

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add rxjava-3.0 @WithSpan async result extension test

52d4976

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add rxjava-3.0 Java 8 interop context-propagation investigation test

3724055

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

review

93a4fb2

Add config json entry

e290336

Skip root context capture in Observable and Single CaptureParentSpanA…

b4cf310

…dvice

Add scheduler context propagation and no-spurious-traces tests for Ob…

23e7443

…servable and Single

Fix misleading comment in correctParentsFromSubscriptionTime test

fcd0e5b

Extend cancel test to cover all five reactive types

9734820

ValentinZakharov added 13 commits July 3, 2026 10:37

Ensure publisher-parent span is always finished in cancelUnderTrace

af25c55

Normalize PropagateParentSpanAdvice guard and add @nonnull to Tracing…

e54fba3

…Observer

spotless

647452a

spotless

1978e70

Extend SubscriptionTest to cover all five RxJava 3 reactive types

5ad8d82

Add cancelledNever test to cover dispose-path span lifecycle in RxJav…

7677d50

…a3ResultExtensionTest

Remove strictTraceWrites(false) overrides — all tests pass under stri…

1ddb984

…ct mode

Remove redundant package comment — testdog convention is implicit acr…

0edde12

…oss the codebase

Remove redundant comments that describe what the code does rather tha…

d7f2a8c

…n why

Use childOfIndex(0) instead of childOf(parentId) in RxJava3InteropTest

14f51b0

Remove redundant @WithConfig for opentelemetry-annotations-1.20 integ…

3ea2d88

…ration

Remove redundant null check on parentContext in CaptureParentSpanAdvice

90abb3f

Extract local variable for repeated Context class name in contextStore()

a6cb915

ValentinZakharov force-pushed the vzakharov/rxjava3 branch from 25a0260 to a6cb915 Compare July 3, 2026 08:58

PerfectSlayer added the tag: concurrency Virtual Threads, Coroutines, Async, RX, Executors label Jul 3, 2026

PerfectSlayer approved these changes Jul 3, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

ValentinZakharov added this pull request to the merge queue Jul 3, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 3, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit bb6b14a into master Jul 3, 2026
590 of 591 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the vzakharov/rxjava3 branch July 3, 2026 15:46

github-actions Bot added this to the 1.64.0 milestone Jul 3, 2026

Uh oh!

Conversation

ValentinZakharov commented Jul 2, 2026

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

dd-octo-sts Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

dd-octo-sts Bot commented Jul 3, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ValentinZakharov commented Jul 3, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dd-octo-sts Bot commented Jul 2, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 3, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 3, 2026 •

edited

Loading