perf(py): index Python imports in runtime venvs

[zbarsky-openai]

Changes are visible to end-users: yes

Adds opt-in indexed imports for private runtime virtual environments:

--@aspect_rules_py//py:experimental_indexed_imports=true

The existing physical layout remains the default. Public virtual environments
stay physical, and individual binaries or tests can opt out with
indexed_imports = False.

How Python finds imports

For import foo, Python's default finder searches sys.path until it finds
the directory that owns foo. A normal virtual environment collects installed
packages under one site-packages directory, letting Python use the filesystem
to find each package directly.

rules_py recreates that directory with package symlinks for every binary and
test. If many targets share the same wheels, the number of projection actions
grows with the number of targets multiplied by the number of packages.
First-party import roots can also make sys.path unnecessarily large.

What changes

Each indexed binary or test adds one build action that records which wheel or
first-party directory owns each import. A .pth file registers a finder on
sys.meta_path; that finder looks up the owner and delegates loading to
Python's existing import machinery. The action consumes declared artifact
paths without reading source contents.

Wheel metadata and RECORD parsing identify safe package projections while
preserving collision policy, package metadata, namespace packages,
pkgutil.extend_path, executable .pth files, native extensions, console
scripts, and PEP 427 data-file handling. Packages that require the existing
physical behavior retain their projections or whole-wheel fallbacks.

Source-wheel build tools keep physical environments. Filesystem-based
typecheckers, IDE integrations, and programs that copy sys.path into a new
interpreter can use public virtual environments or indexed_imports = False.

In a representative large repository, the compatibility-preserving
configuration registered 56% fewer actions, reduced analysis-only build
wall time by 7%, and reduced peak process memory by 6%. A fully
indexed configuration reached 74% fewer actions when filesystem-only
consumers were also virtualized. Building an index from roughly 37,000 input
records took approximately 127 ms including Python interpreter startup.

Test plan

54 focused Bazel tests cover import-index generation, wheel collision
resolution, wheel metadata and exclusions, PEP 427 data files, legacy namespace
packages, indexed private virtual environments, unchanged public environments,
and per-target opt-outs.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 18b68859-1acb-4854-b120-5dfaf8e03369

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}