Skip to content

Prevent duplicate class inclusion in runtime artifacts #11853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gh-worker-dd-mergequeue-cf854d merged 5 commits into from
Jul 3, 2026
+78 −32
Merged

Prevent duplicate class inclusion in runtime artifacts #11853

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ef6bff8
fix(build): Stop publishing extra source-set output twice
bric3 Jul 3, 2026
42247c9
fix(build): Stop publishing agent-installer source-set output twice
bric3 Jul 3, 2026
9133955
fix(build): Keep servlet5 relocated jar off `runtimeElements`
bric3 Jul 3, 2026
adcf141
chore(build): Use lazy API on jakarta-servlet-5.0 shadow jar task
bric3 Jul 3, 2026
f873f22
fix(build): Muzzle supports projects having jars outside runtimeClass…
bric3 Jul 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion buildSrc/src/main/kotlin/datadog/gradle/plugin/muzzle/tasks/MuzzleTask.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import datadog.gradle.plugin.muzzle.MuzzleExtension
import datadog.gradle.plugin.muzzle.allMainSourceSet
import org.gradle.api.Project
import org.gradle.api.artifacts.Configuration
import org.gradle.api.file.ConfigurableFileCollection
import org.gradle.api.file.FileCollection
import org.gradle.api.file.RegularFileProperty
import org.gradle.api.invocation.BuildInvocationDetails
Expand Down Expand Up @@ -59,6 +60,10 @@ abstract class MuzzleTask @Inject constructor(
@get:Classpath
protected val agentClassPath = providers.provider { createAgentClassPath(project) }

@get:InputFiles
@get:Classpath
abstract val extraAgentClasspath: ConfigurableFileCollection

@get:InputFiles
@get:Classpath
protected val muzzleClassPath = providers.provider { createMuzzleClassPath(project, name) }
Expand Down Expand Up @@ -119,7 +124,7 @@ abstract class MuzzleTask @Inject constructor(
buildStartedTime.set(invocationDetails.buildStartedTime)
bootstrapClassPath.setFrom(muzzleBootstrap)
toolingClassPath.setFrom(muzzleTooling)
instrumentationClassPath.setFrom(agentClassPath.get())
instrumentationClassPath.setFrom(agentClassPath.get(), extraAgentClasspath)
testApplicationClassPath.setFrom(muzzleClassPath.get())
if (muzzleDirective != null) {
assertPass.set(muzzleDirective.assertPass)
Expand Down
18 changes: 16 additions & 2 deletions dd-java-agent/agent-installer/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,26 @@ tasks.named("compileMain_java25Java", JavaCompile) {
}

dependencies {
// Wire the Java-version-specific source sets into this project without publishing
// their raw output:
//
// - each additional source set compiles against the main output and only the
// dependencies it needs
// - their compiled outputs stay visible to tests through `testImplementation`
// - their compiled outputs are folded into this project's jar by the jar task below
//
// Main code loads these classes reflectively at agent runtime, so downstream
// consumers should get them from the agent-installer jar. Do not add these
// outputs to `runtimeOnly`: `runtimeElements` publishes both the jar artifact
// and runtime dependencies. Publishing the raw outputs there would expose the
// same classes twice, once from the jar and once from the class directories.
main_java11CompileOnly project(':dd-java-agent:agent-tooling')
main_java11CompileOnly sourceSets.main.output
testImplementation sourceSets.main_java11.output

main_java25CompileOnly project(':dd-trace-core')
main_java25CompileOnly sourceSets.main.output
runtimeOnly sourceSets.main_java11.output
runtimeOnly sourceSets.main_java25.output
testImplementation sourceSets.main_java25.output
}

tasks.named("jar", Jar) {
Expand Down
30 changes: 7 additions & 23 deletions dd-java-agent/instrumentation/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,31 +119,15 @@ if (project.gradle.startParameter.taskNames.any { it.endsWith("generateMuzzleRep
}

tasks.named('shadowJar', ShadowJar) {
duplicatesStrategy = DuplicatesStrategy.FAIL
// Shadow 8.x silently wrote duplicate class (130+) entries into the instrumentation jar.
// Those duplicate entries did not survive as duplicates in the final agent jar,
// because `:dd-java-agent:expandAgentShadowJarInst` expands the instrumentation jar
// with a `Sync` task, and the specific `inst` expansion keeps the first duplicate path.
// (explicit `duplicatesStrategy = DuplicatesStrategy.EXCLUDE` on the `inst` prefix).
// Keep duplicate detection enabled on the aggregate instrumentation jar.
//
// Shadow 9 refactored deeply it's "copy" pipeline via https://github.com/GradleUp/shadow/pull/1233
// by `Project.zipTree` and can now honor `DuplicatesStrategy`, see in particular
// * https://github.com/GradleUp/shadow/issues/1223 - Duplicates from jars are not handled per duplicatesStrategy
// * https://github.com/GradleUp/shadow/issues/488 - DuplicatesStrategy.FAIL doesn't work with transform
// The previous source-set fixes and the servlet5 relocated-jar fix keep extra classes in
// producer jars without also publishing their raw output through `runtimeElements`. The old
// per-path `EXCLUDE` workaround is no longer needed.
//
// Keeping `duplicatesStrategy = FAIL` only, Shadow 9 now fails while creating the instrumentation jar.
filesMatching([
// agent-installer also publishes this Java 11 source-set output at runtime.
'datadog/trace/agent/tooling/bytebuddy/DDJava9ClassFileTransformer*.class',
// agent-installer also publishes this Java 25 source-set output at runtime.
'datadog/trace/agent/tooling/servicediscovery/MemFDUnixWriterFFM*.class',
// aggregate instrumentation includes some classes directly and via jars.
'datadog/trace/instrumentation/**/*.class',
// same aggregate duplication, for Java 11 exception profiling advice.
'datadog/exceptions/instrumentation/**/*.class',
]) {
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
}
// Shadow 9 honors `DuplicatesStrategy.FAIL` for these inputs. If a producer publishes the same
// classes from both its jar and a runtime file or directory, this task should fail again.
duplicatesStrategy = DuplicatesStrategy.FAIL
dependencies {
// the tracer is now in a separate shadow jar
exclude(project(":dd-trace-core"))
Expand Down
29 changes: 26 additions & 3 deletions dd-java-agent/instrumentation/servlet/jakarta-servlet-5.0/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar
import datadog.gradle.plugin.muzzle.tasks.MuzzleTask

plugins {
id 'java-test-fixtures'
Expand Down Expand Up @@ -30,7 +31,7 @@ configurations {
javaxClassesToRelocate
}

tasks.register('relocatedJavaxJar', ShadowJar) {
def relocatedJavaxJar = tasks.register('relocatedJavaxJar', ShadowJar) {
relocate 'javax.servlet', 'jakarta.servlet'
relocate 'datadog.trace.instrumentation.servlet3', 'datadog.trace.instrumentation.servlet5'
relocate 'datadog.trace.instrumentation.servlet', 'datadog.trace.instrumentation.servlet5'
Expand All @@ -49,9 +50,24 @@ tasks.register('relocatedJavaxJar', ShadowJar) {

includeEmptyDirs = false
}
def relocatedJavaxJarFile = relocatedJavaxJar.flatMap { it.archiveFile }

dependencies {
implementation files(relocatedJavaxJar.outputs.files)
// Wire the relocated javax->jakarta advice into this project without publishing
// the relocated jar as runtime output:
//
// - keep the relocated jar visible to main compilation through `compileOnly`
// - keep the relocated jar visible to tests through `testImplementation`
// - include the relocated classes into this project's jar (jar task config)
//
// NOTE:
// Do not add the relocated jar to `implementation`: `runtimeElements` publishes
// both the jar artifact and runtime dependencies derived from `implementation`.
// Publishing the relocated jar there would expose the same servlet5 advice classes
// twice to downstream projects, once from the module jar and once from the
// relocated jar.
compileOnly files(relocatedJavaxJarFile)

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep relocated servlet helpers on the muzzle classpath

In the jakarta-servlet-5.0 project, putting relocatedJavaxJarFile only on compileOnly/testImplementation drops it from sourceSets.main.runtimeClasspath. The muzzle task builds its instrumentation loader from allMainSourceSet.runtimeClasspath in MuzzleTask.createAgentClassPath, and MuzzleVersionScanPlugin then locates every helperClassNames() entry for helper injection. Servlet5RequestBodyInstrumentation lists BufferedReaderWrapper, AbstractServletInputStreamWrapper, and Servlet31InputStreamWrapper, which are produced only by relocatedJavaxJar; when running :dd-java-agent:instrumentation:servlet:jakarta-servlet-5.0:muzzle, those helpers are no longer visible even though they are later unpacked into the jar.

Useful? React with 👍 / 👎.

@mcculls mcculls Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this doesn't affect the content of the final build, but does mean that

./gradlew :dd-java-agent:instrumentation:servlet:jakarta-servlet-5.0:muzzle

currently fails - this just needs fixing before merge

@bric3 bric3 Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I didn't account muzzle for jakarta-servlet, since it was working for the other instrumentations. I'll look at how to fix that.

@bric3 bric3 Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f873f22

I made the muzzle task able to be provided an extra classpath. I evaluated another approach by feeding the relocated jar into the runtimeClasspath, but I think this wrong and could be confusing.

Re-ran jardiff on the jar produced by this commit

$ jardiff -c classdata --include '**/*.class,**/*.classdata' \
  ../master-273405/dd-java-agent/build/libs/dd-java-agent-1.64.0-SNAPSHOT.jar \
  dd-java-agent/dd-java-agent/build/libs/dd-java-agent-1.64.0-SNAPSHOT.jar

testImplementation files(relocatedJavaxJarFile)
compileOnly group: 'jakarta.servlet', name: 'jakarta.servlet-api', version: '5.0.0'
testImplementation group: 'jakarta.servlet', name: 'jakarta.servlet-api', version: '5.0.0'
testImplementation group: 'jakarta.servlet.jsp', name: 'jakarta.servlet.jsp-api', version: '3.0.0'
Expand All @@ -76,5 +92,12 @@ dependencies {
}

tasks.named("jar", Jar) {
from zipTree(relocatedJavaxJar.outputs.files.asPath)
from zipTree(relocatedJavaxJarFile)
}

// Make the relocated jar visible to muzzle through `extraAgentClasspath`.
// Muzzle uses the runtime classpath of a project, and this jar is produced
// outside of it.
tasks.withType(MuzzleTask).configureEach {
extraAgentClasspath.from(relocatedJavaxJarFile)
}
26 changes: 23 additions & 3 deletions gradle/java_no_deps.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,13 +103,33 @@ class TracerJavaExtension {
}
}

// "socket-utils" is only set to compileOnly because the implementation dependency incorrectly adds Java17 classes to all jar prefixes.
// This causes the AgentJarIndex to search for other non-Java17 classes in the wrong prefix location and fail to resolve class names.
if (sourceSetConfig.compileOnly.orElse(false).get()) {
// The ":utils:socket-utils" is only set to `compileOnly` because the `implementation` dependency
// adds Java17 classes to all jar prefixes, which is incorrect for this project.
// This causes the AgentJarIndex to search for other non-Java17 classes in the wrong prefix
// location and to fail to resolve class names.
project.dependencies.add("compileOnly", mainForJavaVersionSourceSet.output)
} else {
// Wire this additional source-set into this project without publishing its raw output
// (handled by the jar configuration). The following
//
// * make this additional source-set's _compile classpath_ visible to **main compilation** through `compileOnly`
// * make this additional source-set's _compiled output_ visible to **main compilation** through `compileOnly`
// * make this additional source-set's _compiled output_ visible to tests through `testImplementation`
//
// The jar task produces the artifact that consumer wants to see, and it is configured
// below to add classes of this additional source-set.
//
// NOTE:
// We don't want to add this source-set's output to the `implementation` configuration!
// Otherwise this additional source-set output directory (where classes are) would end up
// being published as a runtime dependency through `runtimeElements`.
// The `runtimeElements` configuration publishes the jar artifact by default ; thus
// `runtimeElements` would have both the jar, and the added output folder.
// In short this makes classes of this extra source-set in both the jar and the directory.
project.dependencies.add("compileOnly", project.files(mainForJavaVersionSourceSet.compileClasspath))
project.dependencies.add("implementation", mainForJavaVersionSourceSet.output)
project.dependencies.add("compileOnly", mainForJavaVersionSourceSet.output)
project.dependencies.add("testImplementation", mainForJavaVersionSourceSet.output)
}

project.tasks.named("jar", Jar) {
Expand Down