Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe change adds first-party Python bytecode modes, compilation providers, runtime packaging, OCI image support, rules_python interoperability, and broad integration coverage for ChangesFirst-party bytecode support
Priority: ➖ Normal Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant BazelTarget
participant PycAspect
participant PycCompiler
participant RuntimeLauncher
participant OCIImageLayer
BazelTarget->>PycAspect: collect dependency sources
PycAspect->>PycCompiler: compile compatible Python sources
PycCompiler->>RuntimeLauncher: provide PycInfo artifacts
RuntimeLauncher->>OCIImageLayer: expose selected source or bytecode files
OCIImageLayer->>OCIImageLayer: validate mode and rewrite layers
Merge Risk: 🔵 Low · up to The bytecode feature is broadly mergeable, but the Starlark dependency metadata and protobuf pyc-only regression assertion should be corrected to keep release tooling and CI coverage accurate. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 40.82% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 49 functions across 50 files. (14 skipped: 12 unsupported, 2 over the file limit.) ✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| layer: 0 | ||
| files: | ||
| - -rwxr-xr-x 0 0 0 42 Jan 1 2023 ./app.runfiles/_main/oci/py_image_layer/branding/__init__.py | ||
| - -rwxr-xr-x 0 0 0 276 Jan 1 2023 ./app.runfiles/_main/oci/py_image_layer/branding/__pycache__/__init__.cpython-311.pyc |
There was a problem hiding this comment.
pyc = "pyc": source .py files having .pyc in __pycache__
| --- | ||
| layer: 0 | ||
| files: | ||
| - -rwxr-xr-x 0 0 0 276 Jan 1 2023 ./app.runfiles/_main/oci/py_image_layer/branding/__init__.pyc |
There was a problem hiding this comment.
pyc = "pyc_only": no source .py files and only .pyc in its place
8552c7c to
b5dc5a4
Compare
✨ Aspect Workflows Tasks📅 Thu Sep 17 00:09:04 UTC 2026 ✅ 44 successful tasks
⏱ Last updated Thu Sep 17 00:26:41 UTC 2026 · 📊 GitHub API quota 0/7,700 (0% used, resets in 59m) |
py_binary startup benchmark
sys.path quality
Bazel analysis benchmark
py_image_layer benchmark
|
b5dc5a4 to
c572f70
Compare
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
72b09f0 to
addd93c
Compare
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
a939918 to
acba6e7
Compare
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
87748f3 to
4a31404
Compare
4a31404 to
dbc5a45
Compare
de86acc to
d733626
Compare
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d733626e41
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| implementation = _pyc_aspect_impl, | ||
| attr_aspects = ["deps"], | ||
| attrs = PYC_ATTRS, | ||
| required_providers = [[RulesPythonPyInfo]], |
There was a problem hiding this comment.
When --@aspect_rules_py//py:emit_rules_python_providers is enabled, every rules_py py_library intentionally emits RulesPythonPyInfo, so this predicate also applies the aspect to rules_py dependencies. Those targets already emit PycInfo, while _pyc_aspect_impl returns another PycInfo, causing Bazel's “provider provided twice” analysis failure for the supported incremental-migration configuration (for example, the direct rules_py_consumer_test -> :lib edge in e2e/rules-python-provider-compat). Guard targets that already carry PycInfo before returning the aspect provider.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
I (and the robots) think this is incorrect because py_library does not publicly declare it is outputting RulesPythonPyInfo so the aspect won't apply. Tests should verify this though, and this is a discouraged flag only for repos in transition so performance is not as important.
27ec962 to
0be292e
Compare
|
@tamird please review the first-party PYC compilation API and implementation proposed in this PR |
| """, | ||
| ), | ||
| "_pex": attr.label(executable = True, cfg = "exec", default = "//py/tools/pex"), | ||
| "_allowlist_function_transition": attr.label( |
| compile_args.add(src) | ||
| compile_args.add(outputs[0]) | ||
| compile_args.add(src.short_path) | ||
| ctx.actions.run( |
There was a problem hiding this comment.
you're spawning an action for every .py? That's bold
There was a problem hiding this comment.
If we don't then a .py file can not appear in multiple targets and I think there are many cases where things like multiple py_binary targets include the same .py file. Yes the optimal/proper way is for them to share a py_library, but not everyone does.
I did make the pyc compiler a worker for this reason, just to avoid the python spawn overhead per-file, but I'm thinking we should add a config that willing participants (large repos like yours) can opt-in to have a single action that compiles all py_*(srcs) at once. WDYT? Is that a requirement for you?
zbarsky-openai
left a comment
zbarsky-openai
left a comment
There was a problem hiding this comment.
Requesting changes on three independently reproduced correctness problems: the compiler does not start on supported Python 3.8/3.9, a rules_python dependency can silently run optimized bytecode under a non-optimized launcher, and sourceless image dependencies make bazel coverage abort before tests. The inline notes also cover source stripping, source ownership and shared launchers, worker failures/diagnostics, and avoidable analysis/I/O costs.
On the existing action-fanout point: I would prefer bounded compiler batches with cache reuse at Python dependency boundaries instead of relying on a worker to make a per-file action graph cheap. Bazel gives remote execution precedence over the local persistent-worker strategy, and the published image benchmark runs with the default source mode, not these bytecode paths. I’d want source/pyc/pyc_only analysis plus cold and incremental bytecode builds (including the non-worker strategy) before treating the scaling as covered.
-zbarskybot
| for action in target.actions: | ||
| for out in action.outputs.to_list(): | ||
| if out.extension == "pyc": | ||
| existing[out.short_path] = out |
There was a problem hiding this comment.
[P1] Do not reuse rules_python bytecode without checking optimization. precompile_optimize_level=1 produces a file at the same natural path, so this blindly adopts optimized code while the launcher and this feature promise level 0. With Bazel 9.2.0, a dependency reported __debug__ == False and discarded assert False under a launcher with __debug__ == True in both pyc (keep source) and pyc_only (omit source); otherwise identical level-0 controls passed. Reject incompatible reuse or generate level-0 files under distinct artifact paths and map them to the intended runfiles destinations. The rules_python precompilation docs explain why the output filename cannot identify the optimization level.
-zbarskybot
| for binary in binaries: | ||
| binary_mode = binary[PycModeInfo].mode if PycModeInfo in binary else "source" | ||
| if binary_mode != effective_pyc: | ||
| fail("{}: binary {} has pyc={} but the image requires pyc={}; drop the binary's explicit pyc attribute or align it with the image".format( |
There was a problem hiding this comment.
[P1] Keep an explicit sourceless image buildable under coverage. bazel coverage changes the binary’s effective mode to source, but this image still requires pyc_only and aborts analysis, even when both attributes already say pyc_only. An otherwise identical py_test(data = [":image"]) passed under bazel test and failed before any test under bazel coverage; the existing //oci/py_image_layer:pyc_source_data_layers_test in the e2e workspace has this same shape. Keep an explicitly configured image’s dependency isolated from the launcher’s coverage fallback, or reconcile the effective modes without quietly changing the packaged artifact an image-verification test is meant to inspect.
-zbarskybot
| pass | ||
|
|
||
|
|
||
| def parse_args(argv: list[str]) -> tuple[str | None, bool, list[str]]: |
There was a problem hiding this comment.
[P1] Make the compiler itself runnable on supported Python 3.8/3.9. These annotations are evaluated when the script starts. Running the unchanged compiler with the Bazel startup flags and a valid source on real PBS CPython 3.8.20 and 3.9.25 exited before producing any output: 3.8 raises TypeError: 'type' object is not subscriptable; 3.9 raises TypeError: unsupported operand type(s) for |. The compile action intentionally runs a compatible or target interpreter, so this breaks both bytecode modes for the documented older runtimes. from __future__ import annotations avoids evaluating the annotations; please also exercise an older supported runtime.
-zbarskybot
| "PYTHONOPTIMIZE" in ctx.attr.env or | ||
| "PYTHONOPTIMIZE" in ctx.attr.env_inherit or | ||
| "PYTHONOPTIMIZE" in passed_env or | ||
| "PYTHONOPTIMIZE" in inherited_env |
There was a problem hiding this comment.
[P2] Prevent mixed optimization from the invoking shell. The guard covers declared and explicitly inherited settings, but isolated = False still lets an ambient PYTHONOPTIMIZE=1 optimize retained source dependencies while colocated first-party bytecode remains level 0. A CPython fixture yielded sys.flags.optimize == 1, __debug__ == True in the sourceless module, and __debug__ == False in a source-backed module; assertions therefore depend on which kind of dependency supplied the module. Clear the variable before invoking a sourceless launcher, or fail clearly at runtime, instead of allowing two optimization semantics in one process. Python documentation.
-zbarskybot
| continue | ||
| fs = target[DefaultInfo].files.to_list() | ||
| if len(fs) == 1 and _is_file_target(target.label, fs[0]): | ||
| files.append(fs[0]) |
There was a problem hiding this comment.
[P2] Compile local Python files regardless of how the srcs label is spelled. A same-package genrule output compiles if srcs names generated.py but stays source or makes pyc_only fail if it names :generator; a filegroup similarly hides its underlying Python files. The PR’s main_from_genrule fixtures deliberately preserve this distinction even though Bazel hands the rule the same declared file. Consumers commonly use the generating target or group as the source API. Resolve DefaultInfo.files, deduplicate by file, and decide compilation from the artifact/ownership rather than reconstructing its label spelling.
-zbarskybot
| compile_runtime = exec_runtime | ||
| tool_toolchain = EXEC_TOOLS_TOOLCHAIN | ||
| elif target_runtime.interpreter != None: | ||
| compile_runtime = target_runtime |
There was a problem hiding this comment.
[P2] Do not execute the target interpreter on the exec platform when no compatible exec runtime was found. The native-rules_python interop fixture provisions Python 3.11 only through rules_python. With Bazel 8.6 on x86_64, building its rules_python_dep_pyc_311_test for Linux ARM64 caused this PyCompile worker to execute python_3_11_aarch64-unknown-linux-gnu/bin/python3 and quit because it is an ARM ELF, not a host executable. The same target built successfully on the default host platform; a Windows cross-target has the same kind of mismatch. Resolve a compatible native exec tool/precompiler, or fail analysis clearly unless the fallback is known to be runnable on the execution platform. The existing cross-build test uses a matching Aspect-provisioned runtime and misses this migration configuration.
-zbarskybot
| compile_args.add("--expect-version", expected_version) | ||
| if len(outputs) == 2: | ||
| compile_args.add("--legacy") | ||
| compile_args.add(src) |
There was a problem hiding this comment.
[P2] Protect literal @ source paths in both worker and one-shot transports. The positional source/dfile enter a flagfile unframed. @ is valid in Bazel package and target names; real Bazel 8.6 built //control_scope/pkg:tool but failed the otherwise identical //@scope/pkg:tool, because the worker’s flagfile expansion tried to open scope/pkg/mod.py rather than the declared @scope/pkg/mod.py. This compiler’s one-shot parser similarly recurses on a real @script.py. Use unambiguous option framing such as --src=<path> and --dfile=<path>, or another encoding that preserves literal paths through both Bazel’s and the compiler’s expansion.
-zbarskybot
| srcs = [ | ||
| src | ||
| for src in get_py_info(target).transitive_sources.to_list() | ||
| if src.owner.package == ctx.label.package and src.owner.workspace_name == ctx.label.workspace_name |
There was a problem hiding this comment.
[P2] Do not report native bytecode complete after losing forwarded sources. This reads only native transitive_sources; when a rules_python library uses precompile_source_retention = "omit_source", the source can exist only in transitive_implicit_pyc_source_files (the new get_transitive_sources helper accounts for it elsewhere). A legal srcs-less target forwarding native PyInfo through an actual attribute built successfully here with complete=True and no sourceless files, then its pyc_only launcher failed ModuleNotFoundError; source mode and an otherwise identical forwarder through deps both passed under Bazel 8.6. Account for all native provider sources and propagate/reuse mapped artifacts, or at minimum mark unmapped sources incomplete so analysis fails before shipping a broken executable.
-zbarskybot
| toolchain supplies a custom `pyc_compile_tool`. | ||
|
|
||
| Compilation runs one `PyCompile` action per source, served by a Bazel | ||
| persistent worker so the interpreter starts once per worker rather than per |
There was a problem hiding this comment.
[P2] Benchmark the bytecode modes before drawing performance conclusions. The published analysis, startup, and image jobs never set the new pyc flag or attribute, so all three run the default source mode. They do not measure enabled compile cost, startup benefit, worker versus non-worker execution, or the new bytecode image traversal. Please benchmark source, pyc, and pyc_only on an import-heavy first-party graph, including cold builds and a one-source incremental rebuild; include the non-worker strategy rather than assuming the local worker represents remote execution. Published benchmark results.
-zbarskybot
| retained_source_short_paths = {} | ||
| if effective_pyc == "pyc_only": | ||
| for binary in binaries: | ||
| for f in binary[DefaultInfo].default_runfiles.files.to_list(): |
There was a problem hiding this comment.
[P2] Include explicit runfiles symlink targets in image source retention. default_runfiles.files does not contain files added only through Runfiles.symlinks or root_symlinks. A Bazel fixture adding pkg/foo.py through ctx.runfiles(root_symlinks = {"_main/pkg/foo.py": src}) produced files=[] and the expected root symlink. A sourceless launcher still retains this explicit data through the merged runfiles, but this image’s retention set never sees it; when the Python library also owns that source, it is later added to the tar skip list and the image drops a file the launcher includes. Account for both symlink collections and preserve explicit data targets/destinations consistently.
-zbarskybot
zbarsky-openai
left a comment
zbarsky-openai
left a comment
There was a problem hiding this comment.
Following up on the same head with additional, non-overlapping findings. The earlier request for changes still stands. In particular, the source-only default still pays for a bytecode action graph, and PEX unnecessarily rejects a binary that retains all of the source it needs. The inline comments include controls where I could run them.
-zbarskybot
| def _py_python_pex_impl(ctx): | ||
| binary = ctx.attr.binary | ||
| binary = _single_target(ctx.attr.binary) | ||
| if PycModeInfo in binary and binary[PycModeInfo].mode != "source": |
There was a problem hiding this comment.
[P2] Allow a source-retaining pyc binary to feed PEX. This guard rejects py_binary(pyc = "pyc"), although that mode still provides the .py entrypoint and all first-party sources; only pyc_only removes them. With Bazel 8.6.0, an otherwise identical PEX whose binary inherited global --@aspect_rules_py//py:pyc=pyc built, while the PEX wrapping the explicitly pyc binary failed here. That makes an incremental rollout break existing PEX wrappers simply because bytecode was opted into at the target instead of globally. Allow pyc, filtering out the provider-known first-party caches from PEX if it is intended to stay source-only; keep any necessary rejection specific to pyc_only.
-zbarskybot
| instrumented_files_info, | ||
| ] | ||
|
|
||
| compiled = compile_pycs(ctx, own_compile_sources(ctx.attr.srcs)) |
There was a problem hiding this comment.
[P2] Keep bytecode actions out of default source-only analysis. Both this call and py_venv unconditionally register compilation, and the venv transition normalizes the flag to source; not requesting the outputs prevents execution, not action/provider construction. On a source-only one-main fixture, Bazel 8.6.0 aquery exposed one PyCompile action even though the build executed none; the new e2e test explicitly asserts that this happens. Thus even repositories that never opt into bytecode pay for a per-file configured action graph and both output artifacts on ordinary source-mode analysis. This is separate from batching enabled builds: make requesting bytecode an opt-in dependency/aspect so the source-only graph does not construct the compiler actions, while still sharing enabled compilations across launchers.
-zbarskybot
| if prev != None: | ||
| if prev.pycache == entry.pycache: | ||
| continue | ||
| if effective_pyc == "pyc_only" and prev.pycache.basename != entry.pycache.basename: |
There was a problem hiding this comment.
[P2] Honor an explicitly retained source before rejecting mixed-runtime bytecode. Two pyc_only binaries with distinct mains and Python versions can legitimately share a module if both keep shared.py through data, as the documented data-precedence rule allows. This rejects the module on its differing bytecode tags before consulting the retained-source set. CPython imports the retained .py instead of the colocated .pyc (a control even ignored deliberately invalid bytecode), so the shared source supports both interpreters; each distinct main can remain sourceless. Omit conflicting colocated bytecode for explicitly retained paths and only reject collisions for paths that actually have to remain sourceless. PEP 3147.
-zbarskybot
| entrypoint = main, | ||
| info = info, | ||
| mode = mode, | ||
| venv_files = depset(transitive = [vinfo.transitive_sources, info.pycache_files]), |
There was a problem hiding this comment.
[P3] Surface cache-prefix settings that make all packaged bytecode unused. pyc always requests in-tree __pycache__ files, but this launcher accepts interpreter_options = ["-X", "pycache_prefix=/other/path"]; with isolated = False, PYTHONPYCACHEPREFIX can do the same. Python ignores every in-tree cache when a prefix is set, not just the direct main. With this PR’s compiler and -I -B, an imported control used the built cache normally but recompiled source when I supplied a different prefix. Either reject the incompatible declared option, make that combination fall back to source without requesting caches, or explicitly document the unsupported configuration rather than charging for artifacts no import can use. Python’s cache-prefix documentation.
-zbarskybot
| ), | ||
| PycModeInfo(mode = pyc.mode), | ||
| ] | ||
| if pyc.info != None: |
There was a problem hiding this comment.
[P3] Preserve bytecode information across Python-code dependencies that are launchers. A source-mode launcher still exports PyInfo, and py_library.deps accepts Python-code targets, but this condition withholds the bytecode mapping its venv already has. In a Bazel 8.6.0 control, an outer source binary depended on another source binary, imported its Python dependency and ran successfully; switching only the outer binary to pyc_only failed analysis with could not compile all first-party sources and no filenames. The launcher’s venv was normalized to source, so a caller cannot rely on the global mode fixing the inner edge either. Forward the existing mapping when used as Python code (independently of which artifacts the inner launcher packages), or explicitly disallow this dependency shape with an actionable diagnostic.
-zbarskybot
zbarsky-openai
left a comment
zbarsky-openai
left a comment
There was a problem hiding this comment.
Additional findings from a deeper pass on this same head; the earlier request for changes still stands. I would keep a single source-retention policy at the owning Python dependency boundary, preserve the global mode on executable data edges, and avoid asking sourceless package actions to stage inputs they discard. The inline findings include exact-head Bazel or Python controls where available, plus one proposal to replace an opaque compiler golden with a small behavioral test.
-zbarskybot
| def _pyc_aspect_impl(target, ctx): | ||
| # Srcs-less producers such as py_proto_library expose only transitive sources. | ||
| if hasattr(ctx.rule.files, "srcs"): | ||
| srcs = ctx.rule.files.srcs |
There was a problem hiding this comment.
[P2] Do not make unused third-party wheel source block first-party bytecode builds. This aspect treats the native pip hub’s py_library like first-party code, but its generated rule lists all site-packages/**/*.py, including tests/support code the application never imports. With Bazel 8.6, an Aspect-provisioned Python 3.13, and an actual synthetic wheel emitted through the rules_python pip target generator, the source-mode application imported vendor_sample and printed 1; both pyc and pyc_only failed its build on an unused external vendor_sample/tests/unused_python2.py containing Python-2 syntax. Large wheels also contribute an action per such file. Keep first-party compilation strict, but give native wheels a separate explicit policy/opt-in or let the wheel owner decide what it precompiles, so unrelated vendor fixtures cannot block an application rollout.
-zbarskybot
| pycache = ctx.actions.declare_file("__pycache__/{}".format(pycache_basename), sibling = src) | ||
| outputs.append(pycache) | ||
| if pyc == None: | ||
| pyc = ctx.actions.declare_file(stem + ".pyc", sibling = src) |
There was a problem hiding this comment.
[P2] Reject explicit runfiles that collide with compiler-owned bytecode. These natural module.pyc destinations can also be supplied through data; the launcher currently merges those runfiles without rejecting the overlap. With Bazel 8.6, a library’s collision.py returned 1, and the source-mode control returned 1 even with a data genrule supplying a valid unrelated collision.pyc. The otherwise identical pyc_only build succeeded without a collision warning and returned 99 from the data file instead of the library. Validate compiler-owned logical bytecode destinations against explicit runfiles and fail a conflict, or map output artifacts through one centralized precedence policy; declared runtime data should not silently substitute executable library code.
-zbarskybot
| "Python must exactly match the target Python version.").format(ctx.label, main)) | ||
|
|
||
| return struct( | ||
| entrypoint = main_entry.pyc, |
There was a problem hiding this comment.
[P2] Keep the main’s public CLI identity stable across packaging modes. Directly executing the physical .pyc changes sys.argv[0]; default argparse.ArgumentParser().prog, help/error prefixes, and programs that dispatch by executable basename therefore change solely when the global pyc_only flag is enabled. With this compiler and Python’s -I -B, the same main reported runme.py in source mode and runme.pyc when run sourceless. Preserve the original logical entrypoint in sys.argv[0] via a small bootstrap, or set the Bazel executable identity consistently in both modes. Source stripping should not change the public command name or CLI behavior.
-zbarskybot
| "pyc": attr.string( | ||
| default = "", | ||
| values = [""] + PYC_MODES, | ||
| doc = "First-party bytecode mode for the image; empty inherits the `//py:pyc` flag. Read by a transition, so not configurable.", |
There was a problem hiding this comment.
[P2] Make an image composable with a binary’s supported configurable bytecode mode. py_binary(pyc = select(...)) is explicitly supported, but this image attribute cannot use select, and an unset image assumes the global flag rather than the binary’s resolved choice. In Bazel 8.6, the same binary selected source for -c dbg and pyc_only for -c opt and analyzed in both. The unset and source-pinned images passed debug but failed release; the pyc_only-pinned image passed release but failed debug. Thus no single fixed or unset image target can package that valid binary across both configurations. Let an unset image derive and validate the resolved binary mode (including agreement among binaries), or expose a configurable image policy that is not read as an incoming transition attribute; keep explicit disagreement errors.
-zbarskybot
| # configuration. | ||
| def _reset_python_flags_transition_impl(settings, _attr): | ||
| acc = {} | ||
| acc = {PYC_FLAG: "source"} |
There was a problem hiding this comment.
[P2] Preserve the global bytecode mode for executable data dependencies. This hard-resets pyc to source even when the outer target inherited the command-line setting; an explicit terminal attribute already does not change that setting. With Bazel 8.6, building parent(data = [":child"]) and :child under global pyc_only produced two versions of the same child: the directly built child executed child.pyc, but invoking it from the parent’s runfiles executed child.py (the parent runfiles contained no child.pyc). Subprocesses therefore silently run different code layouts, and aggregates build duplicate child configurations. Preserve the caller’s global value on data edges, or restore a real captured baseline rather than the literal source mode.
-zbarskybot
| data_sources = [ | ||
| get_py_info(target).transitive_sources | ||
| get_transitive_sources(target) | ||
| for target in ctx.attr.data |
There was a problem hiding this comment.
[P2] Preserve transitive Python sources declared as data below the launcher. This adds a Python data target’s transitive sources only when data is attached directly to the terminal. With Bazel 8.6, owner(data = [":middle"]), middle(deps = [":leaf"]), and a pyc_only binary depending on both owner and middle retained middle.py but only leaf.pyc; code reading the explicitly declared transitive data source failed. Moving the same data = [":middle"] to the binary retained both sources and succeeded, as did the source-mode control. The documented transitive data/source-precedence contract should not depend on which graph node declares the data edge. Include Python data targets’ transitive source depsets in the shared library/venv runfiles construction, not only in this terminal comprehension.
-zbarskybot
| rule_group_files = [files for _, files in rule_groups] | ||
|
|
||
| source_files = depset(transitive = [info.source_files for info in infos]) | ||
| source_files = depset(transitive = [info.source_files for info in infos] + pyc_files) |
There was a problem hiding this comment.
[P2] Remove stripped Python sources from the sourceless image action inputs. source_files still contains every raw first-party source; only the later awk skip list removes its tar row. Both the mtree and final tar action declare that unchanged depset as real input. On the existing pyc_rule_group_layers fixture, Bazel 8.6 aquery showed both actions take server.py plus server.pyc, and direct_source_helper.py plus its .pyc, although those raw sources are not supposed to be packaged. Remote executors must stage inputs that the artifact discards, increasing input-tree/materialization cost for every sourceless image. Separate path-only skip/symlink metadata from the actual packageable depset, and pass only bytecode, explicitly retained data and the other genuinely packaged files to the tar action.
-zbarskybot
| try: | ||
| code = compile(source, dfile, "exec", dont_inherit=True, optimize=0) | ||
| except SyntaxError as exc: | ||
| raise CompileError("{}: {}".format(dfile, exc)) from exc |
There was a problem hiding this comment.
[P3] Preserve the compiler’s syntax-error source and caret. Stringifying SyntaxError strips the source line, column/range and exception type in both one-shot and worker modes. With this exact script, return (1 + ) produced only pkg/broken.py: invalid syntax (broken.py, line 2); the standard compiler gave the same logical filename plus the source, caret under ), and SyntaxError. These details matter especially for generated or long lines. Prefer stdlib py_compile.compile, which also avoids manually serializing the header (a valid-source control produced identical bytes), or retain the underlying diagnostic with traceback.format_exception_only.
-zbarskybot
| write_source_files( | ||
| name = "snapshots", | ||
| files = { | ||
| "snapshots/version_check_test.pyc": ":snapshot_pyc", |
There was a problem hiding this comment.
[P3] Replace the opaque compiler snapshot with a small semantic fixture. This checks in roughly 13 KB of bytecode compiled from the entire version-check unittest driver, with a private venv, launcher and shell extraction just to produce it. Changing the test driver or the pinned interpreter can require updating an opaque binary without showing reviewers which bytecode guarantee changed. The focused tests currently check output existence and layout equality, but not the runtime magic, selected invalidation flags/hash, stable co_filename (including nested code), or whether valid cached code actually executes. Assert those contracts on a tiny module in the existing compiler test and remove the snapshot pipeline; that would give smaller, more diagnostic coverage.
-zbarskybot
| "abi_flags": attr.string( | ||
| doc = "CPython ABI flag suffix, e.g. \"t\" for freethreaded.", | ||
| ), | ||
| "pyc_compile_tool": attr.label( |
There was a problem hiding this comment.
[P3] Drop the unused custom compiler contract, or make it a real provisioning option. This adds a second toolchain compiler API with its own executable/argument behavior and a version check the documentation says the tool may ignore. The interpreter hub’s only py_runtime_toolchain instantiation never passes it, the public interpreter-extension toolchain tag has no attribute to supply it, and no test configures it. Consequently the documented workaround for a third-party runtime is not selectable through the supported provisioning API, while the normal path still maintains this branch. Prefer removing it until there is a concrete supported consumer; if it is needed now, wire it through provisioning and prove it emits compatible bytecode and is actually selected.
-zbarskybot
zbarsky-openai
left a comment
zbarsky-openai
left a comment
There was a problem hiding this comment.
Four more non-overlapping findings from an explicitly requested pass on the same head. The earlier request for changes still stands. The highest-impact issue is that changing the compiler can rerun a bytecode action using stale worker code and produce incorrect output under the new cache key. The other notes cover non-isolated import precedence/hermeticity, misleading sourceless tracebacks, and an experimentally confirmed cross-configuration cache miss that one Bazel execution requirement fixes. The reproductions used Bazel 8.6 and CPython where applicable.
-zbarskybot
| execution_requirements = execution_requirements, | ||
| inputs = depset( | ||
| direct = [src], | ||
| transitive = [pyc_compile_tool.inputs], |
There was a problem hiding this comment.
[P1] Declare the compiler and interpreter support files as worker tools, not ordinary inputs. Bazel hashes Spawn.getToolFiles() to decide whether an existing worker can be reused; here only the bare interpreter is a tool, while the Python compiler script and runtime files are ordinary action inputs. With Bazel 8.6, I built a target that printed old, changed only the compiler script to emit new, and rebuilt on the same server: Bazel executed the compile action again via a worker, but the result still printed old. A fresh local-strategy control printed new; a sandboxed worker also reused its previous compiler after a second edit. This can produce incorrect bytecode under the updated action-cache key. Put the compiler/runtime closure in tools and keep the actual source in inputs, so compiler changes restart workers. Bazel worker hashing.
-zbarskybot
| compile_args.add("--legacy") | ||
| compile_args.add(src) | ||
| compile_args.add(outputs[0]) | ||
| compile_args.add(src.short_path) |
There was a problem hiding this comment.
[P2] Do not let sourceless tracebacks load arbitrary working-directory files. src.short_path is stored as an ordinary relative co_filename. I compiled a module that raises RuntimeError, packaged only its colocated .pyc, and ran it from a different directory containing an unrelated pkg/module.py; the traceback printed a line and caret from that unrelated file. With an empty working directory it printed no source. This can mislead debugging and copy incidental working-directory text into logs. Using a pseudo filename such as <pkg/module.py> fixes both modes in a control: sourceless imports print no unrelated source, while CPython’s source-retaining loader rewrites the cached filename to the actual .py path and prints the genuine raising line. Python traceback source lookup.
-zbarskybot
| ) | ||
| embedded_args, transformed_args = launcher.append_runfile( | ||
| file = main, | ||
| file = pyc.entrypoint, |
There was a problem hiding this comment.
[P2] Keep a non-isolated sourceless launcher’s first import directory in logical runfiles. Directly executing main.pyc lets CPython follow its symlink and prepend the physical generated-artifact directory to sys.path. With Bazel 8.6/Python 3.12, isolated=False, srcs=["main.py", "helper.py"], data=["helper.py"], and imports=["."], source mode imported the explicitly retained helper.py; sourceless mode chose physical helper.pyc and inspect.getsource() failed. The isolated sourceless control used retained source. This reproduced with directory and manifest runfiles. In another control, neither consumer declared an importable generated module, but after building an unrelated target in the package—without rebuilding either consumer—only the sourceless one could import that module; neither runfiles manifest contained it. Suppress the automatically prepended physical artifact directory and prepend the logical runfiles main directory for non-isolated launchers, so source/data precedence and imports do not depend on previous unrelated builds.
-zbarskybot
| tool_args.add_all(pyc_compile_tool.arguments) | ||
| execution_requirements = {} | ||
| if getattr(pyc_compile_tool, "supports_workers", False): | ||
| execution_requirements = {"supports-workers": "1", "requires-worker-protocol": "json"} |
There was a problem hiding this comment.
[P2] Opt PyCompile into Bazel path mapping. These requirements omit supports-path-mapping, even though source and output are passed as Bazel Files and the embedded source name is stable. With Bazel 8.6, --experimental_output_paths=strip, sandboxed compilation, and a shared disk cache, two unrelated configurations compiling the same checked-in source produced byte-identical bytecode but both executed and missed the cache; their flagfile/output paths still contained different configuration hashes. In an otherwise equivalent control, adding just supports-path-mapping produced the same bazel-out/cfg/... paths in both configurations and the second compile was a disk-cache hit. Set that requirement independently of worker capability so irrelevant configurations do not multiply this per-file compile cost; check the supported sandboxed worker and non-worker strategies.
-zbarskybot
zbarsky-openai
left a comment
zbarsky-openai
left a comment
There was a problem hiding this comment.
Two final non-overlapping findings from the same explicitly requested pass: the full interpreter distribution is sent through every per-file compiler action, and the fallback cache-tag syntax is not valid for PyPy. The runtime input counts came from exact-head Bazel 8.6 action graphs for two interpreter provisions; the PyPy behavior is grounded in its upstream implementation rather than a local PyPy execution. The earlier request for changes still stands.
-zbarskybot
| arguments = ["-S", "-s", "-B", ctx.file._pyc_compiler], | ||
| inputs = depset( | ||
| [compile_runtime.interpreter, ctx.file._pyc_compiler], | ||
| transitive = [compile_runtime.files], |
There was a problem hiding this comment.
[P2] Give the compiler a purpose-built runtime closure rather than the complete interpreter distribution. An exact-head Bazel 8.6 aquery for one 27-byte source found 1,888 runtime input artifacts under Aspect-provisioned Python 3.13 (including 264 C headers, 90 IDLE files, Tcl/Tk and turtle demos) and 2,286 under the native Python 3.11 fallback. Bazel enumerates and sends every declared input’s path/digest in every WorkRequest, even though this worker ignores that field; remote actions also inherit the oversized input tree, although I did not measure bytes transferred. That cost repeats for each per-source action. Expose a lean bytecode-execution tool/runtime containing the executable and required startup libraries/modules, rather than unconditionally adding all compile_runtime.files. Declaring the current distribution as tools fixes worker invalidation but does not remove its per-request enumeration. Bazel’s WorkRequest construction.
-zbarskybot
| version = getattr(runtime, "interpreter_version_info", None) | ||
| if not implementation_name or version == None: | ||
| return None | ||
| return "{}-{}{}".format(implementation_name, version.major, version.minor) |
There was a problem hiding this comment.
[P2] Do not invent a CPython-shaped cache tag for other implementations. For a third-party runtime with implementation_name="pypy" and no supplied pyc_tag, this produces pypy-311 (the new test even expects pypy-312). PyPy’s implementation actually exposes pypy311, without a hyphen. The source-retaining mode therefore builds and ships module.pypy-311.pyc, while the runtime looks for module.pypy311.pyc; those caches cannot be used. Limit the fallback to known CPython or require the actual target tag for other implementations, and consider checking that the selected compiler’s real sys.implementation.cache_tag matches the declared value. I verified the mismatch against PyPy’s cache-tag definition and its sys.implementation; I did not run a PyPy binary locally.
-zbarskybot
zbarsky-openai
left a comment
zbarsky-openai
left a comment
There was a problem hiding this comment.
Five additional findings from the requested exact-head pass; the earlier request for changes still stands. The standard/free-threaded Python 3.13 image failure and the public-provider wrapper that silently drops importable first-party code are both reproduced with Bazel 8.6. I would keep the design smaller: do not construct compilation in source mode, use one compiler per genuinely compatible bytecode format, and make one source-bearing dependency contract own compilation completeness.
There is also new correctness evidence for the previously reported default-mode action registration, without opening a duplicate inline thread: a binary with no pyc setting and an existing genrule that emits module.pyc built and ran at the PR merge-base, but at this head fails because the unrequested PyCompile action claims the same output.
I reran four upstream compiler/analysis tests without cached test results; all four passed. The native PyPy interpreters were not run; that finding is supported by immutable release source and a Bazel test of the actual compatibility helper.
-zbarskybot
| exec_toolchain = ctx.toolchains[EXEC_TOOLS_TOOLCHAIN] | ||
| exec_runtime = getattr(exec_toolchain, "exec_runtime", None) if exec_toolchain != None else None | ||
| compile_runtime = None | ||
| if exec_runtime != None and getattr(exec_runtime, "interpreter", None) != None and bytecode_compatible(exec_runtime, target_runtime): |
There was a problem hiding this comment.
[P1] Use one compiler for compatible standard and free-threaded CPython targets. This generic exec toolchain chooses different Python 3.13 compiler binaries for the two GIL modes. With the same checked-in source (def value(): return {i for i in (1, 2, 3)}), Bazel 8.6 produced different .pyc bytes even though the magic, source hash and code matched, and each interpreter successfully loaded both outputs. Both standalone sourceless binaries ran; py_image_layer(binaries = [":gil", ":free"], pyc = "pyc_only") instead failed PyImageLayerValidate on shared.pyc; the source-retaining image also failed on its shared __pycache__ file. The two shared inputs also ran four compilation actions through two worker pools. Use a canonical compiler for these compatible runtime variants, while continuing to reject genuinely conflicting generated sources.
-zbarskybot
| transitive_pycache_files.append(dep_pyc.pycache_files) | ||
| transitive_sourceless_files.append(dep_pyc.sourceless_files) | ||
| complete = complete and dep_pyc.complete | ||
| elif has_py_info(dep) and PyWheelsInfo not in dep: |
There was a problem hiding this comment.
[P2] Do not use the transitive wheel provider as proof a Python-code dependency contains no first-party sources. A custom wrapper forwarding the publicly exported PyInfo, PyWheelsInfo and DefaultInfo from an ordinary Aspect py_library with zero wheels hits this exception and is reported complete without any bytecode mapping. Under Bazel 8.6 the source binary printed the expected value; the identical pyc_only binary built successfully, shipped neither the wrapped module’s .py nor .pyc, and failed ModuleNotFoundError. Removing only the empty wheel provider from the wrapper correctly made analysis fail. Limit the exemption to actual wheel-owned sources/install trees, or propagate bytecode through source-bearing public Python providers; unrelated wheel metadata must not disable completeness checks.
-zbarskybot
| args = iter(argv) | ||
| for arg in args: | ||
| if arg.startswith("@"): | ||
| with open(arg[1:]) as f: |
There was a problem hiding this comment.
[P2] Decode both compiler transports explicitly as UTF-8. The flagfile here and sys.stdin in worker_loop are decoded using the host’s default text encoding, but Bazel writes UTF-8 for both. With Python 3.12 under a non-UTF-8 locale, a UTF-8 flagfile for pkg/café.py raised UnicodeDecodeError before producing output; the raw UTF-8 JSON worker request returned success but wrote a corrupted co_filename. The ASCII-escaped JSON control preserved the correct name. An actual Bazel 8.6 one-shot flagfile for naïve.py contains UTF-8; Python’s Windows filesystem encoding change does not make text-file decoding UTF-8. Specify encoding="utf-8" here and parse worker stdin as bytes with json.loads, or explicitly reconfigure it to UTF-8.
-zbarskybot
| prereleases require an exact version match. | ||
| """ | ||
| target_key = _bytecode_key(target_runtime) | ||
| return target_key != None and target_key == _bytecode_key(exec_runtime) |
There was a problem hiding this comment.
[P2] Do not infer non-CPython bytecode compatibility from the cache tag and Python language version. Stable PyPy 7.3.22 and 7.3.23 both expose Python 3.11.15 final and the correct pypy311 cache tag, but the releases deliberately use incompatible bytecode magic: 416 in 7.3.22 and 432 in 7.3.23. An exact-head Bazel test showed this helper calls them compatible; the compiler’s --expect-version check also cannot distinguish them. A sourceless consumer consequently cannot load the output; a source-backed cache is unusable. Require actual magic/build identity, or conservatively restrict cross-runtime matching to CPython until third-party toolchains expose that identity. This persists even when the separate fallback cache-tag spelling is fixed.
-zbarskybot
| if compile_runtime != None: | ||
| pyc_compile_tool = struct( | ||
| executable = compile_runtime.interpreter, | ||
| arguments = ["-S", "-s", "-B", ctx.file._pyc_compiler], |
There was a problem hiding this comment.
[P2] Carry compile-affecting launcher policies into precompilation. An explicit interpreter_options = ["-X", "int_max_str_digits=0"] makes a 5,000-digit decimal literal valid. With Bazel 8.6/Python 3.12, the source target ran; both bytecode modes failed in PyCompile at its default limit of 4,300. The opposite failure occurs with interpreter_options = ["-Werror::SyntaxWarning"]: with Bazel 8.6/Python 3.13, an imported return value is 1000 failed from source, while both bytecode modes built and ran successfully. Python performs both checks while parsing source; strict warnings cannot be recovered at launch from compiled code. Define an explicit build-side policy for these options, or reject settings the shared compiler cannot honor; merely forwarding them to the launcher changes target behavior.
-zbarskybot
zbarsky-openai
left a comment
zbarsky-openai
left a comment
There was a problem hiding this comment.
Four additional, non-overlapping findings from the requested pass on this exact head. The earlier request for changes still stands. The most concerning issue is that both bytecode modes can execute a native dependency’s unadvertised private action output in place of its declared Python source. The other notes cover a strategy-dependent filename failure, a global bytecode-compilation barrier for otherwise independent image groups, and a redundant provider graph that can be deleted.
The four focused upstream compiler/analysis tests passed uncached. The native substitution and filename cases were reproduced; the image dependency was confirmed with Bazel’s action graph, but end-to-end image timing was not measured.
-zbarskybot
| existing = {} | ||
| for action in target.actions: | ||
| for out in action.outputs.to_list(): | ||
| if out.extension == "pyc": |
There was a problem hiding this comment.
[P2] Reuse only bytecode advertised by the native Python provider. This scans every action for .pyc, including private outputs that the dependency does not declare as Python bytecode, default files, or runfiles. With Bazel 8.6/CPython 3.11, a native PyInfo rule advertised victim.py returning 1 and happened to have a separate hidden action producing valid victim.pyc returning 99. The source control ran 1; pyc_only ran 99. The same happened in source-retaining pyc with a hidden __pycache__ output, even though the real .py was still present. This is distinct from checking the optimization of genuine native bytecode: here the provider declares no bytecode at all. Use the native provider’s bytecode fields and reject overlaps with unadvertised outputs, or compile the declared sources under separately owned artifacts and map them into runfiles.
-zbarskybot
| if arg.startswith("@"): | ||
| with open(arg[1:]) as f: | ||
| nested_version, nested_legacy, nested_files = parse_args( | ||
| f.read().splitlines() |
There was a problem hiding this comment.
[P2] Split Bazel flagfiles on physical line endings only. Python’s str.splitlines() also splits Unicode U+2028/U+2029 and NEL, which can be literal characters in a valid Bazel source filename. On a UTF-8 Linux host with Bazel 8.6, a filename containing U+2028 failed under --strategy=PyCompile=local because this code tried to open the truncated prefix; the exact same target succeeded with the worker and emitted the correct logical filename. The materialized flagfile had three literal U+2028 characters and six real LF delimiters. This still fails with explicitly correct UTF-8 decoding. Iterate physical lines and remove only their actual CR/LF terminators so the worker and one-shot strategies handle the same input.
-zbarskybot
| rule_group_files = [files for _, files in rule_groups] | ||
|
|
||
| source_files = depset(transitive = [info.source_files for info in infos]) | ||
| source_files = depset(transitive = [info.source_files for info in infos] + pyc_files) |
There was a problem hiding this comment.
[P2] Do not make independent image groups wait for all generated bytecode. Adding the entire compiled closure here also feeds it to the one shared symlink-mapping action, which every group’s mtree consumes. Bazel 8.6 aquery for separate left and right libraries/groups showed the shared mapping takes left.pyc, main.pyc, and right.pyc; the left mtree directly takes only left.pyc plus that mapping, and the right mtree takes only right.pyc plus the same mapping. Neither group can start until the unrelated group’s compile finishes. The awk consumer only records source/destination paths for mapping-only rows and does not open those bytecode files. Write ordinary generated-bytecode registrations from their paths without depending on their contents; reserve the shared dynamic dependency for artifacts whose expansion actually requires it. Removing stripped raw .py inputs alone does not remove this dependency. The action graph was verified; end-to-end timing was not measured.
-zbarskybot
| direct = direct_entries, | ||
| transitive = transitive_entries, | ||
| ), | ||
| legacy_files = depset( |
There was a problem hiding this comment.
[P3] Remove the transitive legacy_files provider projection. This builds another depset graph through every configured library, venv, and native aspect. Its only production consumer is image assembly, immediately before that code already walks PycInfo.entries, and every entry carries its exact colocated .pyc. The sourceless launcher already uses sourceless_files, which separately preserves non-Python source artifacts. Collect entry.pyc during the image’s existing entry traversal and remove the private provider field and its transitive plumbing; this deletes an entire parallel graph without another closure walk or changing the launcher’s retained-file behavior. Source inspection found one consumer; I have not measured memory savings.
-zbarskybot
zbarsky-openai
left a comment
zbarsky-openai
left a comment
There was a problem hiding this comment.
Seven additional findings from the explicitly requested pass on this head; the earlier request for changes still stands. The highest-impact result is that a fresh source-retaining build can execute generated bytecode that disagrees with its packaged source. Default-mode native source retention and sourceless unittest artifact selection also change which code runs. I would select one runtime source before deriving its bytecode, resolve unittest inputs only from runfiles, pass both declared compiler outputs explicitly, and union image group inputs before rewriting them.
The generated-source and native default-mode controls used Bazel 8.6; the unittest substitution used Bazel 9.2. Two existing Bazel 8.6 coverage targets passed and reported hits for their shared library, but a synthetic LCOV control demonstrates that the new regression assertion also accepts zero hits for that library. The Windows-path result used Windows path semantics, not a native Windows host; the image analysis counts are file visits, not wall-clock benchmarks.
-zbarskybot
| entrypoint = main, | ||
| info = info, | ||
| mode = mode, | ||
| venv_files = depset(transitive = [vinfo.transitive_sources, info.pycache_files]), |
There was a problem hiding this comment.
[P1] Compile the same generated source that wins runtime runfiles. A generated .py can be present in both Python-transitioned srcs and explicit caller-configured data; those configurations can produce different contents at the same logical path. In a fresh Bazel 8.6 / Python 3.13 build of all three modes, source and pyc_only both imported VALUE = 0, the visible data source; pyc imported VALUE = 313 although opening its packaged generated.py still read VALUE = 0. The runfiles manifest selected the inherited configuration for the .py and the transitioned one for its unchecked cache. This happens on the first build, without editing anything or using a stale launcher. Derive the cache from the selected runtime source, omit it when source identity conflicts, or validate the retained source before using it; otherwise simply enabling the performance mode silently changes program behavior.
-zbarskybot
| info = get_py_info(target) | ||
| return depset(transitive = [ | ||
| info.transitive_sources, | ||
| getattr(info, "transitive_implicit_pyc_source_files", depset()), |
There was a problem hiding this comment.
[P2] Keep recovered native compile sources out of default runtime runfiles when the dependency omitted them. This unconditionally merges transitive_implicit_pyc_source_files into the same source set used for Aspect runtime packaging. With Bazel 8.6 / Python 3.12, a native library explicitly used precompile="enabled", precompile_source_retention="omit_source" and level 1; a native provider wrapper correctly published its required sourceless .pyc in runfiles. An Aspect binary with no pyc setting worked on the merge base: the source was absent, Python used SourcelessFileLoader, and __debug__ was false. On this head the same binary gained the omitted .py, Python chose SourceFileLoader, and __debug__ became true. That both exposes explicitly omitted source and changes existing default-mode behavior. Separate source recovery needed by the compiler or coverage from runtime source selection; do not inject a matching omitted .py when the native bytecode is already packaged.
-zbarskybot
| rel = rel[len("../"):] | ||
| mod_name = rel[:-len(".py")].replace("/", ".") | ||
| loader = importlib.machinery.SourceFileLoader(mod_name, path) | ||
| if os.path.exists(path): |
There was a problem hiding this comment.
[P2] Resolve unittest sources and bytecode from runfiles, not the caller’s working directory. path is a baked relative filename, so this existence check lets an arbitrary matching .py override the declared sourceless test. With Bazel 9.2, bazel test passed and the executable’s runfiles contained only pkg/test_safety.pyc. Without rebuilding, invoking the same executable from its runfiles ran the compiled test, invoking it from the workspace ran an edited source instead, and invoking it from an unrelated directory ran that directory’s matching source; all reported success. Resolve both candidate artifacts through the Bazel runfiles directory/manifest before choosing a loader. A pyc_only executable should run its packaged tests regardless of the caller’s working directory.
-zbarskybot
|
|
||
| def legacy_path(src: str, out: str) -> str: | ||
| """``pkg/__pycache__/mod.<tag>.pyc`` beside ``mod.py`` -> ``pkg/mod.pyc``.""" | ||
| cache_dir = out.rpartition("/")[0] |
There was a problem hiding this comment.
[P2] Stop reconstructing the second compiler output with POSIX-only separators. The new version_check_test.py builds its paths with os.path.join; on native Windows its positive --legacy case passes ordinary backslash paths, which this check rejects before writing the colocated file. Running the exact helper with Windows-path semantics rejected C:\work\pkg\__pycache__\module.cpython-313.pyc; the equivalent forward-slash input succeeded. I did not run a native Windows host, and Bazel-generated action arguments may use /; the direct compiler contract and the new untagged test do use native paths. Passing both already-declared outputs explicitly would remove this reconstruction entirely; otherwise use platform-aware filesystem path operations.
-zbarskybot
| if mode == "source" or not pyc_by_source_path: | ||
| return files | ||
| out = [] | ||
| for f in files.to_list(): |
There was a problem hiding this comment.
[P2] Union each first-party tier group before rewriting it for bytecode. This helper flattens every tier entry separately, while entries assigned to the same output group are only unioned later. Instrumenting this exact loop under Bazel 8.6, one pyc image with 20 libraries in the same tier group, each sharing one opaque filegroup of 200 Python data files, performed 20 scans of 201 files: 4,020 visits for 220 distinct files. The otherwise identical source image did not scan here. This cost persists with just one binary, independently of the per-binary closure flattening reported earlier. Preserve the original depsets until the group boundary and rewrite their union once, or memoize the file-level conversion if individual ownership still has to be retained. I measured file visits, not wall-clock time.
-zbarskybot
| "PYTHONOPTIMIZE" in passed_env or | ||
| "PYTHONOPTIMIZE" in inherited_env | ||
| ): | ||
| fail("{}: pyc_only does not support PYTHONOPTIMIZE in env or env_inherit, even when set to 0. Remove it or use pyc = \"pyc\" or \"source\".".format(ctx.label)) |
There was a problem hiding this comment.
[P3] Reject only environment optimization that can actually affect this interpreter. This fails pyc_only for an explicit PYTHONOPTIMIZE=0, and for any declared value under the default isolated=True, even though CPython’s -I ignores it. With Bazel 8.6 an isolated source control with value 1 printed isolated=1, optimize=0, __debug__=True; sourceless variants with value 0, or value 1 under that same isolation, both failed analysis. Non-isolated CPython also runs at level zero for an explicit value 0. This unnecessarily prevents shared configuration that pins level zero or passes a setting only to child tools from adopting sourceless packaging. Skip this environmental guard for isolated launchers; in non-isolated mode allow a known declared zero and handle unknown inherited/ambient values separately.
-zbarskybot
| check_coverage //coverage-drivers:coverage_pytest_codegen_test bazel-testlogs/coverage-drivers/coverage_pytest_codegen_test/coverage.dat | ||
| check_coverage //coverage-drivers:coverage_pytest_chdir_test bazel-testlogs/coverage-drivers/coverage_pytest_chdir_test/coverage.dat | ||
| check_coverage //coverage-drivers:coverage_unittest_test bazel-testlogs/coverage-drivers/coverage_unittest_test/coverage.dat | ||
| check_coverage //coverage-drivers:coverage_pyc_only_test bazel-testlogs/coverage-drivers/coverage_pyc_only_test/coverage.dat |
There was a problem hiding this comment.
[P3] Check that the new coverage regression actually records a hit for foo.py. The reused helper checks for an SF:...foo.py anywhere and a nonzero DA anywhere, not in that file’s record. An LCOV file with only DA:1,0 for foo.py and DA:1,1 for an unrelated file passes both exact checks. That means the new pyc_only coverage test can go green when coverage of the shared Python dependency is lost, even though that is the behavior this test promises to protect. Scope the nonzero-line assertion to the foo.py record ending at end_of_record; no extra fixture matrix is needed.
-zbarskybot
@zbarsky-openai is python 3.8/3.9 support required? Currently 3.10 is the minimum as you've noted... |
zbarsky-openai
left a comment
zbarsky-openai
left a comment
There was a problem hiding this comment.
Re-reviewed 1a8b158. Removing the explicit transition allowlist is safe: supported Bazel 8.6 supplies it automatically. I ran //pyc:pex_no_bytecode_test on this exact checkout, including a forced rerun; it passed. The shared-source discussion did uncover a new, reproducible default-mode action-conflict blocker in the cumulative implementation; I’ve left the exact reproduction inline. The three-line allowlist deletion is not its cause.
On the Python 3.8/3.9 question: 3.10 is indeed the bundled floor. I called older versions supported because the current interpreter docs explicitly advertise opting into them, including a working-configuration example for 3.8, and the new bytecode docs do not declare a separate floor. I can’t set anyone’s adoption requirement. If first-party bytecode intentionally starts at 3.10, I’d withdraw that blocker once that feature boundary is documented and selecting an older runtime reports a clear unsupported-version error rather than crashing at compiler startup. Otherwise, postponing annotation evaluation appears to be the smallest fix.
On the batching question: I’d keep per-source compatibility as the default and consider opt-in batching for large targets only if measurements justify it. The shared-source concern is valid with the current output paths. The existing partial-overlap fixture puts the same raw sources in targets with different complete source sets. One action per target declaring those same sibling bytecode outputs would give Bazel different producers for the same output; shared actions must have identical inputs, outputs and command lines. A persistent worker saves local interpreter starts, but still leaves one action/cache unit per source; it does not remove remote per-action scheduling when remote execution is selected.
The simplest optional design I see is to batch only where the repository can guarantee a single compilation owner for each source (shared raw files move into a common py_library), and keep per-source mode for arbitrary overlaps. The compiler already accepts multiple SRC OUT DFILE triples; actions can group files according to whether they need both outputs or only one. I’d compare configured action counts, a clean build and a one-source edit in both local-worker and actual remote execution before changing the default: a change to one file invalidates the entire batch. This is my review recommendation, not a statement of anyone else’s adoption requirements. Earlier signed findings remain on the PR; this commit doesn’t change them.
-zbarskybot
|
|
||
| outputs = [] | ||
| if pycache == None: | ||
| pycache = ctx.actions.declare_file("__pycache__/{}".format(pycache_basename), sibling = src) |
There was a problem hiding this comment.
[P1] Preserve raw-source sharing across rules_py and rules_python. On the supported Bazel 8.6 / CPython 3.12, an Aspect py_library and a rules_python py_library(precompile = "enabled", precompile_source_retention = "keep_source") both directly listed the same checked-in shared.py. An Aspect bytecode consumer of either library separately passed. An Aspect consumer depending on both failed analysis with conflicting actions for both shared.pyc and __pycache__/shared.cpython-312.pyc; an otherwise identical consumer in default source mode also failed. The identical default-source fixture passes on the PR merge base (c978b80). The direct rules_py rule claims both natural outputs in one action, while the foreign precompilation/interoperability path claims them in separate actions with different action keys; inspecting only the foreign target’s own actions cannot prevent the direct declaration. Coordinate a single producer, or give independent intermediate compilations distinct artifact paths and map/deduplicate their logical bytecode outputs. Per-source actions alone currently don’t preserve cross-ruleset source sharing.
-zbarskybot
1a8b158 to
ed8f20b
Compare
zbarsky-openai
left a comment
zbarsky-openai
left a comment
There was a problem hiding this comment.
Re-reviewed the rewritten ed8f20b. Comparing both immutable archive trees directly, only README.md, docs/interpreter.md, and the Ruff target in pyproject.toml changed; the bytecode implementation and tests are byte-identical.
The new ruleset-wide Python 3.10 minimum and interpreter-policy clarification address the ambiguity behind my earlier Python 3.8/3.9 finding. I no longer classify inability to run the compiler on those versions as a supported-version correctness blocker. An older PBS interpreter can still be downloaded/configured and the unchanged compiler still fails with a generic annotation TypeError rather than a clear unsupported-version error; I consider that a nonblocking diagnostics follow-up under the clarified policy.
The previously reported cross-ruleset source-sharing action conflict remains: its entire implementation is unchanged. The previous supported Bazel 8.6 controls passed with either owner individually and on the merge base, but failed even in default source mode when the two owners were combined. I’m not repeating that or the other unchanged findings inline. I found no new issue in this three-file rewrite.
-zbarskybot
zbarsky-openai
left a comment
zbarsky-openai
left a comment
There was a problem hiding this comment.
Re-reviewed the five-file e312809 update. I found one new issue in the OCI coverage regression: it can pass on query/archive errors and can miss a forbidden source even in a valid archive. I reproduced those cases with the exact shell block and GNU tar; a short forbidden-source archive failed as expected, and a .pyc-only control passed. The existing image test can replace the new shell inspection; details are inline.
The image transition now explicitly disables coverage instrumentation for the packaged image, and the compiler now postpones annotation evaluation. The previously reported cross-ruleset source-sharing action conflict is not changed by moving compiler inputs to tools: this increment leaves the output declarations and foreign-output reuse unchanged. I’m not duplicating that finding. I did not run the actual OCI coverage target locally.
-zbarskybot
| fi | ||
| coverage_layers="$("$BAZEL" cquery --collect_code_coverage --output=files //oci/py_venv_image_layer:pyc_only_amd64_layers 2>/dev/null)" | ||
| for layer in $coverage_layers; do | ||
| if tar tzf "$layer" | grep -q '/_main/oci/py_venv_image_layer/__main__\.py$'; then |
There was a problem hiding this comment.
[P2] This regression test can report PASS without checking an image—and even when a valid image contains the forbidden source. With this exact block, a successful build followed by cquery exiting 17, an empty successful query, or a missing tar all return 0 and print PASS. I also made a valid GNU tar archive with __main__.py first and 10,000 later entries: grep -q exits on the match, tar then returns nonzero because its output pipe closes, and pipefail makes the if false; this archive also prints PASS. A short archive with the same source correctly fails. Rather than adding more shell error handling, please run the existing //oci/py_venv_image_layer:my_app_pyc_only_amd64_layers_test with bazel coverage (or bazel test --collect_code_coverage): it already checks that .pyc exists and .py does not, and exercises the image through a test dependency.
-zbarskybot
2 participants
PYC compilation as an action in
py_libraryfor first-partypy_library(srcs)as well as an aspect applied topy_library(deps)for legacyrules_pythonproviders.Changes are visible to end-users: yes
Add
pyc = "source|pyc|pyc_only"flag alongsidepython_version,freethreadedflags in public APIs as well as a default via--config=@aspect_rules_py//py:pyc=source|pyc|pyc_only.Test plan