Fix resolution of generated and provider-forwarded code in the IDE by FrankPortman · Pull Request #401 · JetBrains/hirschgarten

Frank Portman (FrankPortman) · 2026-07-13T23:47:36Z

Companion to JetBrains/intellij-aspect#143. Fixes resolution of generated and provider-forwarded code, diagnosed and verified against a production monorepo (generated Scala/Python, scrooge thrift, provider-forwarding custom rules). Runnable repro of all problem classes: pluginTests/testData/testProjects/generated_code_resolution.

Commit	Bug	Fix
`5e5a6a3`	`.srcjar` is not an archive file type, so `jar://...srcjar!/` sources roots are dead: decompiled view, manual attach fails too	Register `.srcjar` as ARCHIVE (legacy: `BlazeFileTypeFactory`)
`47c8c01`	`toJarUrlString()` writes `file://` URLs for `.srcjar`, unusable as sources roots	jar protocol for `jar`, `srcjar`, `zip` (legacy: `LibraryModifier`)
`a6a2a07`	Jars already provided by a real library get a second sourceless jdeps library that wins resolution	Skip jars covered by known libraries
`fd81ce7`	Python resolve index maps generated sources to hard-link copies outside all content roots; navigation opens orphan files where nothing resolves	Prefer the real output path, which the module content roots cover
`6b9e045`	Python and JVM importers derive the same module name for dual-language targets; the second entity write is rejected (`MutableEntityStorageImpl: symbolicId already exist`), so whichever importer runs second loses its module, nondeterministically per target and sync	Skip such targets in the Python importer, interim until BAZEL-2214. Removing the check was tested on the repro monorepo and breaks resolution randomly
`54f37d3`		Demo project

Two fixes still needed from intellij-aspect#143: python_target_info.generated_sources (feeds the resolve index, CustomRuleTest there), and dependency edge serialization for custom attributes, which BAZEL-3364's test classification relies on for provider-forwarding rules.

Evgenii Pasynkov (noxvost) · 2026-07-15T08:20:10Z

We need a demo project which highlights the problems being fixed. Though the changes are looking reasonable (with some flaws), it is impossible for me to evaluate the PR without seeing the root cause

Frank Portman (FrankPortman) · 2026-07-15T15:27:20Z

Evgenii Pasynkov (@noxvost) please see the demo project linked in this PR.

I tried to make it as little code as possible, while showcasing the exact setup we have in our monorepo. I am not sure if some of what happened in my monorepo is a "perfect storm" scenario, but you can see the README (LLM generated, but I verified) in the demo project which shows the 4 things that were broken in our monorepo. Really it's 1 thing (Scala/Python codegen did not resolve) but I discovered stuff along the way to trying to fix that at varying levels.

To restate them plainly:

Scala/Python targets didn't resolve at all
Naive/simple fixes to the above would not let us GoToRef inside a code generated source to a different code generated source. This needed extra handling.
thrift_library + scrooge_scala_library from open source rules_scala did not GoToRef initially, and naive/simple fixes related to the above points led to them "resolving" but not actually displaying source code via GoToRef. This needed extra handling.
(This one I understand the least) Some of our generated Scala was only resolving from test targets. This one took the most amount of testing and iteration to fix but it seems fine with the current state of this PR + Detect Scala/Python targets beyond rule kinds, report generated Python, serialize custom dependency edges intellij-aspect#143

NothingButYou (Krystian20857)

Thanks for PR,
However I'll take a look at provided reproduction cases myself and check what's exactly is going on inside.

NothingButYou (Krystian20857) · 2026-07-15T16:13:55Z

+        // binaryOutputs and not outputJars: the latter is filtered to files that exist
+        // on disk when the hard links are created, so jars that were never built (e.g.
+        // in a configuration produced by a transition) would silently drop out of the
+        // index and the corresponding libraries would never be redirected to their
+        // source module. Identity comes from the aspect info and does not require the
+        // files to exist.
+        (target.findBuildData<JvmBuildTarget>()?.binaryOutputs ?: SourceFileCollection.EMPTY).getFiles()
+          .map { it.normalizeOutputPath() to target.targetKey }


This one was suppose to substitute compiled JARs (mainly hjars) which are pulled by manually constructed JavaInfo provider by code generator.

from what I understand, the problem with outputJars for this index is the existence filter, i.e. in a transitioned configuration the jars are never built locally, so exactly those entries drop out and the libraries never redirect to their source modules.

binaryOutputs carries the same identities without needing files on disk. if outputJars has to stay as the hjar-substituted view for the codegen case, happy to point the index at a dedicated field instead.

Fair point, I'll take closer look at this particular case

NothingButYou (Krystian20857) · 2026-07-15T16:14:37Z

  @VisibleForTesting
  fun calculateTargetsToMarkAsTest(
    targets: Set<RawBuildTarget>,
    labelToTargetInfo: Map<Label, RawBuildTarget>,
-    executableTargets: Map<ResolvedLabel, List<Label>>,
  ): Set<Label> {
    val (testTargets, nonTestTargets) =
      targets.partition { it.isTestTarget() }
-    val libraryTargets = nonTestTargets.filter { it.kind.ruleType == RuleType.LIBRARY }.map { it.id }
-    val directTestDependencies = libraryTargets.filter { library ->
-      val executables = executableTargets[library]?.mapNotNull { labelToTargetInfo[it] }.orEmpty()
-      val executablesWithoutSelfReference = executables.filterNot { it.id == library }
-      val directExecutables = executablesWithoutSelfReference.filter { it.dependsOn(library) }
-      return@filter directExecutables.isNotEmpty()
-                    && executablesWithoutSelfReference.all { it.isTestTarget() }
-                    && directExecutables.any { it.sources.isEmpty() && it.id.packagePath == library.packagePath }
-    }.toSet()
+
+    // Everything any non-test executable depends on, transitively, is production code and
+    // must never be marked as test sources. ExecutableTargetsComputer cannot be used to
+    // answer this: its same-package attribution shortcut and its result cap are run
+    // configuration heuristics, and both can hide production dependents of a library that
+    // also has a test right next to it (e.g. a code generation macro emitting a library
+    // together with a same-package sourceless test).
+    val dependenciesOf = targets.associate { it.id to it.dependencies.map { dep -> dep.targetKey.label } }
+    fun reachableFrom(roots: Collection<Label>): Set<Label> {
+      val visited = HashSet<Label>(roots)


Marcin Kocot (@mrkocot) Can you take a look at this?

i think this ended up being the fix for what #401 (comment) was also related to. I am hoping the configuration stuff was not actually needed. will check

I've tested it with both normal and tricky cases I remember used to be problematic, works for them well

can i resolve this comment?

NothingButYou (Krystian20857) · 2026-07-15T16:21:16Z

+      // Dual-language targets (e.g. custom rules forwarding both PyInfo and JavaInfo) are
+      // owned by the JVM importer: both importers derive the same module name, so creating
+      // a Python module here would clobber the JVM module and break JVM consumers.
+      if (target.rawBuildTarget.kind.isJvmTarget()) continue


This is a known issue, however we should aim for more generic solution
https://youtrack.jetbrains.com/issues/BAZEL?preview=BAZEL-2214

that link doesn't seem to take me to any specific issue

ah, issue is private, but related to java + python module names over lapping, we're working on this one

Frank Portman (@FrankPortman) could you please check whenever removing this check impact code resolution inside your repository?

tested it: unfortunately this breaks resolution. claude claims it breaks non-deterministically based on a race condition (the workspace model rejects the second write for dual-language targets), but I did not try a bunch of cold runs because of how long it takes to sync my full monorepo.

Verbatim from the log:

MutableEntityStorageImpl - Entity with symbolicId: ModuleId(name=<redacted>-514860d) already exist

In this case many objects were fine but I happened to see a few breaking in different test files that were fine before. Confirmed it's normal again after adding this commit back.

One extra data point, that Claude seems to think is relevant: even with this change I still see LibraryId collisions between the *_python entities of configuration duplicates, so the overlap isn't just java vs python.

Verbatim from Claude:

Yes, exactly. On the current review-tip build (skip in place), this session's log shows: - ModuleId collisions: 0 — the java-vs-python module clobber is fully gone - LibraryId collisions: 126 — all shaped like <pkg>.<target>_python-514860d already exist, i.e. the Python importer's own source-dependency libraries colliding with themselves: two instances of the same Python target (config/aspect duplicates) both writing a library under the same name It's benign in practice — the colliding libraries are content-identical duplicates, so whichever write wins carries the same roots, which is why everything stays green. But it demonstrates the naming overlap is python-vs-python too, not just java-vs-python — useful scope information for their generic BAZEL-2214 fix, which is why it earns the one-sentence mention in the comment.

Source jars produced by Bazel rules (e.g. scrooge_scala_library) use the .srcjar extension. Without the archive mapping, library source roots pointing at them cannot be opened by the VFS: the IDE decompiles class jars even though the source jar is attached, and manually attaching sources fails too. The legacy plugin registers the extension the same way, see BlazeFileTypeFactory in bazelbuild/intellij.

Library roots for source jars with the .srcjar extension were written as file:// URLs, which cannot serve as sources roots: the IDE cannot look inside the archive. Matches the extension handling of the legacy plugin's LibraryModifier (jar, srcjar, zip).

…braries A jar already provided by a target's library got a second, synthetic jdeps library without source jars. The duplicate competes with the proper library during resolution and the IDE ends up decompiling class files even though sources are attached to the real library.

The resolve index mapped generated sources to their hard link copy, which is not covered by any module content root: navigation opened an orphan file in which no reference resolves further. The module's content roots are anchored at bazel-bin, so prefer the real output path and keep the hard link only as a fallback for files that are not on disk.

Both the Python and the JVM importer derive the same module name for a target that carries both PythonBuildTarget and JVM data (e.g. a custom rule forwarding PyInfo and JavaInfo). The Python module has no JVM dependencies, so when it wins, every JVM consumer of the target stops resolving. Skipping such targets in the Python importer is an interim fix; the proper modelling is a Python facet on the JVM module, as done by the legacy plugin.

One minimal Bazel project with four packages, one per problem class fixed in this PR and in the companion aspect PR: generated Scala and Python that do not resolve after a plain sync, multi-hop navigation between generated libraries, scrooge thrift sources resolving to decompiled class files, and a library classified as test sources because its codegen macro emits a same-package sourceless test next to it while production code reaches it only through a provider-forwarding rule. The README maps each package to the fixing commits. Builds standalone with Bazel 9.1.1.

Frank Portman (FrankPortman) mentioned this pull request Jul 13, 2026

Detect Scala/Python targets beyond rule kinds, report generated Python, serialize custom dependency edges JetBrains/intellij-aspect#143

Open

Frank Portman (FrankPortman) changed the title ~~Fix resolution of generated code: srcjar handling, library dedup, Python navigation, dual-language modules~~ Fix resolution of generated and provider-forwarded code in the IDE Jul 14, 2026

Frank Portman (FrankPortman) mentioned this pull request Jul 14, 2026

Fix "Go to Definition" for exported java_library targets #399

Open

Evgenii Pasynkov (noxvost) requested a review from NothingButYou (Krystian20857) July 15, 2026 08:14

NothingButYou (Krystian20857) reviewed Jul 15, 2026

View reviewed changes

Frank Portman (FrankPortman) added 6 commits July 20, 2026 11:36

Frank Portman (FrankPortman) force-pushed the fix/generated-code-resolution branch from 61017e3 to 54f37d3 Compare July 20, 2026 15:48

Uh oh!

Conversation

Frank Portman (FrankPortman) commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Evgenii Pasynkov (noxvost) commented Jul 15, 2026

Uh oh!

Frank Portman (FrankPortman) commented Jul 15, 2026

Uh oh!

NothingButYou (Krystian20857) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Frank Portman (FrankPortman) commented Jul 13, 2026 •

edited

Loading