Gathering transitive prebuilt_python_library dependencies

[Ralith]

I want to write a rule to package a Blender extension. Such packages are a zip archive of:

• First-party Python code
• Third-party dependencies in wheel form, exactly as downloaded from pypi
• A manifest file enumerating the wheels

I see PythonLibraryManifestsInterface defines many fields, which I guess are functions that return transitive sets of information about a library and its transitive dependencies? This looks like it could be used to gather all the first-party source code, though I haven't found a way to actually convert any of them to a list of paths I could pass to AnalysisActions.copied_dir. I think this is possible but I don't understand how to use transitive sets; guidance/examples would be very welcome.

More fundamentally, I don't see any fields that look like they describe all the prebuilt_python_library sources. Does such a thing exist? If not, could it be added? I believe this is useful for Python packaging in general: see applications discussed at astral-sh/uv#1681.

[thebenmerlin]

Short version: use .traverse(), not PythonLibraryManifestsInterface — and you don't need prebuilt sources at all, because the .whl is already the target's default output.

1. First-party sources

PythonLibraryManifestsInterface is projection-only (python.bzl:133-143) — every field returns cmd_args, and projection output is opaque to Starlark by design. It exists so make_py_package can dump the whole closure onto a command line. That's why you can't get paths out of it.

.traverse() yields real PythonLibraryManifests records, and ManifestInfo.artifacts is already list[(Artifact, dest_path)] (manifest.bzl:22-27) — exactly what copied_dir wants. (.traverse() = whole graph, deduped; .value = this node only.)

2. Prebuilt sources

A field describing them can't exist. prebuilt_python_library does feed the same tset, but its manifest is a single entry — the whole extracted directory (manifest.bzl:75-90):

    # TODO: enumerate directory?
    return ManifestInfo(manifest = ..., artifacts = [(extracted, param)])

The real file list comes from create_manifest_for_source_dir.py doing an os.walk when the action runs. Analysis doesn't read file contents, so buck2 genuinely doesn't know what's inside a .whl yet. Anything consuming wheel contents has to be an action reading the manifest JSON at build time — create_link_tree.py is the canonical example.

But you want wheels verbatim, not their contents, and that's already exposed — prebuilt_python_library sets its default output to the unmodified input (prebuilt_python_library.bzl:155):

source = binary_src          # the .whl, untouched
providers.append(DefaultInfo(default_output = source, sub_targets = sub_targets))

So dep[DefaultInfo].default_outputs[0] is the wheel byte-for-byte as pypi served it, and .basename is the filename for the manifest. For transitive collection buck2 has no aspects, but attrs.query() resolves to a list[Dependency] over the matched subgraph — python_wheel uses exactly this (python_wheel.bzl:566).

3. The rule

# tools/blender_extension.bzl
load("@prelude//:paths.bzl", "paths")
load("@prelude//python:python.bzl", "PythonLibraryInfo", "PythonLibraryManifestsTSet")

def _blender_extension_impl(ctx: AnalysisContext) -> list[Provider]:
    tree = {}  # dest path in the zip -> Artifact

    # First-party Python, from the transitive manifest tset.
    manifests = ctx.actions.tset(
        PythonLibraryManifestsTSet,
        children = [d[PythonLibraryInfo].manifests for d in ctx.attrs.deps],
    )
    for node in manifests.traverse():
        # Skip prebuilt libs: their "srcs" is one opaque extracted directory.
        if str(node.label.raw_target()) in ctx.attrs.wheel_labels:
            continue
        for manifest in [node.srcs, node.default_resources[0] if node.default_resources else None]:
            if manifest == None:
                continue
            for artifact, dest in manifest.artifacts:
                existing = tree.get(dest)
                if existing != None and existing != artifact:
                    fail("conflicting entries for {}: {} and {}".format(dest, existing, artifact))
                tree[dest] = artifact

    # Third-party wheels, exactly as downloaded.
    wheels, seen = [], {}
    for dep in ctx.attrs.wheels + (ctx.attrs.wheels_query or []):
        if PythonLibraryInfo not in dep:
            continue
        for whl in dep[DefaultInfo].default_outputs:
            if whl.basename not in seen:
                seen[whl.basename] = True
                wheels.append(whl)

    wheels = sorted(wheels, key = lambda w: w.basename)  # deterministic manifest
    for whl in wheels:
        tree[paths.join("wheels", whl.basename)] = whl

    # blender_manifest.toml. TOML string arrays are JSON-compatible.
    tree["blender_manifest.toml"] = ctx.actions.write(
        "blender_manifest.toml",
        [
            'schema_version = "1.0.0"',
            'id = "{}"'.format(ctx.attrs.extension_id),
            'version = "{}"'.format(ctx.attrs.version),
            "name = {}".format(json.encode(ctx.attrs.extension_name)),
            "tagline = {}".format(json.encode(ctx.attrs.tagline)),
            "maintainer = {}".format(json.encode(ctx.attrs.maintainer)),
            'type = "add-on"',
            'blender_version_min = "{}"'.format(ctx.attrs.blender_version_min),
            "license = {}".format(json.encode(ctx.attrs.license)),
            "wheels = {}".format(json.encode(["./wheels/" + w.basename for w in wheels])),
        ],
    )

    staged = ctx.actions.copied_dir("__staged__", tree)
    out = ctx.actions.declare_output("{}.zip".format(ctx.attrs.extension_id))
    ctx.actions.run(
        cmd_args(
            ctx.attrs._mkzip[RunInfo],
            cmd_args(out.as_output(), format = "--output={}"),
            cmd_args(staged, format = "--root={}"),
        ),
        category = "blender_extension",
    )
    return [DefaultInfo(default_output = out)]

blender_extension = rule(
    impl = _blender_extension_impl,
    attrs = {
        "blender_version_min": attrs.string(default = "4.2.0"),
        "deps": attrs.list(attrs.dep(providers = [PythonLibraryInfo]), default = []),
        "extension_id": attrs.string(),
        "extension_name": attrs.string(),
        "license": attrs.list(attrs.string(), default = ["SPDX:GPL-3.0-or-later"]),
        "maintainer": attrs.string(),
        "tagline": attrs.string(),
        "version": attrs.string(),
        "wheel_labels": attrs.list(attrs.string(), default = []),
        "wheels": attrs.list(attrs.dep(), default = []),
        "wheels_query": attrs.option(attrs.query(), default = None),
        "_mkzip": attrs.default_only(attrs.exec_dep(default = "//tools:mkzip")),
    },
)

# tools/mkzip.py
import argparse, os, zipfile

p = argparse.ArgumentParser()
p.add_argument("--output", required=True)
p.add_argument("--root", required=True)
args = p.parse_args()

with zipfile.ZipFile(args.output, "w", zipfile.ZIP_DEFLATED) as z:
    for dirpath, dirnames, filenames in os.walk(args.root):
        dirnames.sort()
        for name in sorted(filenames):
            path = os.path.join(dirpath, name)
            info = zipfile.ZipInfo(os.path.relpath(path, args.root))
            info.date_time = (1980, 1, 1, 0, 0, 0)   # reproducible output
            info.external_attr = 0o644 << 16
            info.compress_type = zipfile.ZIP_DEFLATED
            with open(path, "rb") as f:
                z.writestr(info, f.read())

# BUCK
load("//tools:blender_extension.bzl", "blender_extension")

native.python_bootstrap_binary(name = "mkzip", main = "tools/mkzip.py")

http_file(
    name = "requests-whl",
    urls = ["https://files.pythonhosted.org/.../requests-2.32.3-py3-none-any.whl"],
    sha256 = "...",
    out = "requests-2.32.3-py3-none-any.whl",
)

prebuilt_python_library(name = "requests", binary_src = ":requests-whl")

python_library(
    name = "my_addon_lib",
    srcs = glob(["my_addon/**/*.py"]),
    base_module = "",          # so files land at the zip root, not under the package path
    deps = [":requests"],
)

blender_extension(
    name = "my_addon",
    deps = [":my_addon_lib"],
    wheels = [":requests"],
    wheel_labels = ["root//:requests"],
    extension_id = "my_addon",
    extension_name = "My Addon",
    tagline = "Does a thing",
    maintainer = "You <you@example.com>",
    version = "1.0.0",
)

buck2 build :my_addon gives you an installable zip: first-party code at the root, pristine wheels under wheels/, manifest enumerating them.

Note base_module = "" — dest paths in the tset default to the dep's package path (python_library.bzl:68-85), and Blender wants __init__.py and blender_manifest.toml at the zip root.

The one awkward part is wheel_labels: the tset mixes first-party and prebuilt nodes with nothing to distinguish them. A small provider propagating (wheel, metadata) would remove that filter and is probably the real feature request — it'd cover the astral-sh/uv#1681 cases too.