toolchain: defer driver materialization for runtime-only builds

DESIGN.md

@@ -10,11 +10,15 @@ flags.
 
 ## Bundle repos and exported targets
 
-Each `xls.toolchain(...)` call materializes two repos. `@<name>_runtime`
-contains the selected tool binaries, the DSLX stdlib tree, the matching
-`libxls` shared library, and the runtime-facing exports. `@<name>_toolchain`
-contains the local `xlsynth-driver`, the `:bundle` target, and the registered
-toolchain target. The public split is:
+Each `xls.toolchain(...)` call exposes one runtime repo and one toolchain repo.
+`@<name>_runtime` contains the selected tool binaries, the DSLX stdlib tree,
+the matching `libxls` shared library, and the runtime-facing exports.
+`@<name>_toolchain` contains the `:bundle` target, the registered toolchain
+target, and a declared `:xlsynth-driver` target. Loading or registering the
+toolchain repo is metadata-only; the driver target copies, validates,
+downloads, or installs `xlsynth-driver` only when `:xlsynth-driver` is built
+directly or a rule action consumes the driver from `:bundle`. The public split
+is:
 
 - `@<name>_runtime//:libxls`
 - `@<name>_runtime//:libxls_link`
@@ -26,8 +30,17 @@ toolchain target. The public split is:
 - `@<name>_runtime//:xlsynth_sys_runtime_files`
 - `@<name>_runtime//:xlsynth_sys_link_dep`
 - `@<name>_runtime//:libxls_runtime_files`
+- `@<name>_toolchain//:xlsynth-driver`
 - `@<name>_toolchain//:bundle`
-- `@<name>_toolchain//:all`
+- `@<name>_toolchain//:toolchain`
+
+Workspaces still register the selected default with
+`register_toolchains("@<name>_toolchain//:all")`; that is the registration
+pattern for the package, not a separate exported target. When a local or
+installed driver path is configured, the module extension creates a private
+generated repo that exposes that host driver file as a declared input to
+`@<name>_toolchain//:xlsynth-driver`. Downstream workspaces do not use that
+repo directly.
 
 The `xlsynth_sys_*` exports are the intended downstream contract for
 `rules_rust` `crate_extension.annotation(...)` wiring. The preferred modern
@@ -48,12 +61,19 @@ generic bundle internals.
 - `local_paths` uses explicit local paths and is the documented escape hatch
   for `/tmp/xls-local-dev/` style setups.
 
-For the installed-layout modes, the provider derives the concrete paths from
-the toolchain declaration instead of hard-coding a repository-global install
-root: `<installed_tools_root_prefix>/v<xls_version>` for the tools tree and
+For the installed-layout modes, runtime materialization derives the concrete
+tools and library paths from the toolchain declaration instead of hard-coding a
+repository-global install root: `<installed_tools_root_prefix>/v<xls_version>`
+for the tools tree, DSLX stdlib, and `libxls`. Driver materialization derives
 `<installed_driver_root_prefix>/<xlsynth_driver_version>/bin/xlsynth-driver`
-for the driver binary. The provider owns the version-derived suffixes; the
-consumer workspace owns the root prefixes.
+inside the declared driver target. The provider owns the version-derived
+suffixes; the consumer workspace owns the root prefixes.
+
+For `local_paths`, runtime materialization uses `local_tools_path`,
+`local_dslx_stdlib_path`, and `local_libxls_path`. The local driver path is
+needed only by driver-backed actions. This lets runtime consumers depend on
+`@<name>_runtime` or register `@<name>_toolchain` without materializing,
+probing, downloading, or compiling `xlsynth-driver`.
 
 ## Default bundles and explicit overrides
 

README.md

@@ -47,8 +47,7 @@ register_toolchains("@workspace_xls_toolchain//:all")
   release artifacts.
 - `installed_only` requires the matching installed layout.
 - `download_only` always downloads the release artifacts.
-- `local_paths` uses `local_tools_path`, `local_dslx_stdlib_path`,
-  `local_driver_path`, and `local_libxls_path`.
+- `local_paths` uses explicit paths supplied by the consumer workspace.
 
 For the installed-layout modes, `rules_xlsynth` derives exact-version paths as:
 
@@ -59,27 +58,46 @@ For the installed-layout modes, `rules_xlsynth` derives exact-version paths as:
 - `<installed_driver_root_prefix>/<xlsynth_driver_version>/bin/xlsynth-driver`
   for the driver binary
 
-The attributes accepted by each mode are strict:
-
-- `local_paths` requires all four `local_*` attrs and does not accept
-  `xls_version` or `xlsynth_driver_version`.
-- `auto` and `installed_only` require `xls_version`,
-  `xlsynth_driver_version`, `installed_tools_root_prefix`, and
-  `installed_driver_root_prefix`, and do not accept any `local_*` attrs.
-- `download_only` requires `xls_version` and `xlsynth_driver_version`, and
-  does not accept any `local_*` or `installed_*` attrs.
-
-Download-backed modes also have one host prerequisite: when `auto` falls back
-to downloading, or when `download_only` is selected, the repository rule
-installs `xlsynth-driver` with `rustup run nightly cargo install`. The host
-running module resolution must have `rustup` available. If the nightly
-toolchain is missing, `rules_xlsynth` bootstraps a repo-local `rustup` home
-before installing the driver.
-
-Each `xls.toolchain(...)` call now exports two repos:
+The attributes accepted by each mode are strict, but runtime and driver inputs
+are resolved at different times:
+
+- The runtime repo for `local_paths` requires `local_tools_path`,
+  `local_dslx_stdlib_path`, and `local_libxls_path`. `local_driver_path` is
+  required only when a driver-backed bundle/action is built.
+- The runtime repo for `auto` and `installed_only` requires `xls_version` and
+  `installed_tools_root_prefix`. `xlsynth_driver_version` and
+  `installed_driver_root_prefix` are required only when a driver-backed
+  bundle/action is built.
+- `download_only` requires `xls_version` for the runtime repo.
+  `xlsynth_driver_version` is required only when the driver action has to
+  install the driver.
+- `local_paths` does not accept `xls_version` or `xlsynth_driver_version`;
+  the other modes do not accept any `local_*` attrs.
+- `download_only` does not accept any `installed_*` attrs.
+
+Registering or loading `@<name>_toolchain` is metadata-only: it defines the
+toolchain, bundle, and `xlsynth-driver` targets, but does not copy, execute,
+download, or compile the driver. Driver materialization is a declared Bazel
+action behind `@<name>_toolchain//:xlsynth-driver` and behind rule actions that
+consume the driver from `@<name>_toolchain//:bundle`.
+
+Download-backed driver actions have one host prerequisite: when `auto` falls
+back to downloading the driver, or when `download_only` is selected and a
+driver-backed action is built, that action installs `xlsynth-driver` with
+`rustup run nightly cargo install`. The execution host must have `rustup`
+available. If the nightly toolchain is missing, `rules_xlsynth` bootstraps a
+repo-local `rustup` home before installing the driver.
+
+Each `xls.toolchain(...)` call now exports two public repos:
 
 - `@<name>_runtime` for runtime files, `xlsynth-sys` wiring, tools, and `libxls`
-- `@<name>_toolchain` for `:bundle`, `:toolchain`, and `register_toolchains(...)`
+- `@<name>_toolchain` for `:bundle`, `:toolchain`, `:xlsynth-driver`, and
+  the `@<name>_toolchain//:all` registration pattern
+
+When a local or installed driver path is configured, the module extension
+creates a private generated repo that exposes that host driver file as a
+declared input to `@<name>_toolchain//:xlsynth-driver`. Downstream workspaces
+do not use that repo directly or publish it with `use_repo(...)`.
 
 The runtime repo exposes:
 

artifact_resolution_test.py

@@ -43,6 +43,47 @@ def test_auto_falls_back_to_download(self):
         self.assertEqual(plan["xls_version"], "0.38.0")
         self.assertEqual(plan["driver_version"], "0.33.0")
 
+    def test_runtime_surface_does_not_require_or_select_driver(self):
+        observed_paths = []
+
+        def exists_fn(path):
+            observed_paths.append(path)
+            return path != "/tools/xlsynth/v0.38.0/xlsynth-driver-sentinel"
+
+        plan = materialize_xls_bundle.resolve_artifact_plan(
+            artifact_source = "auto",
+            xls_version = "0.38.0",
+            driver_version = "",
+            surface = "runtime",
+            installed_tools_root_prefix = "/tools/xlsynth",
+            installed_driver_root_prefix = "",
+            exists_fn = exists_fn,
+        )
+        self.assertEqual(plan["mode"], "installed")
+        self.assertNotIn("driver", plan)
+        self.assertNotIn("driver_version", plan)
+        self.assertEqual(
+            observed_paths,
+            [
+                "/tools/xlsynth/v0.38.0",
+                "/tools/xlsynth/v0.38.0/xls/dslx/stdlib",
+                "/tools/xlsynth/v0.38.0/libxls.dylib" if sys.platform == "darwin" else "/tools/xlsynth/v0.38.0/libxls.so",
+            ],
+        )
+
+    def test_runtime_local_paths_does_not_require_local_driver_path(self):
+        plan = materialize_xls_bundle.resolve_artifact_plan(
+            artifact_source = "local_paths",
+            xls_version = "",
+            driver_version = "",
+            surface = "runtime",
+            local_tools_path = "/tmp/xls-local-dev/tools",
+            local_dslx_stdlib_path = "/tmp/xls-local-dev/stdlib",
+            local_libxls_path = "/tmp/xls-local-dev/libxls.so",
+        )
+        self.assertEqual(plan["mode"], "local_paths")
+        self.assertNotIn("driver", plan)
+
     def test_auto_requires_installed_prefixes(self):
         with self.assertRaises(ValueError):
             materialize_xls_bundle.resolve_artifact_plan(
@@ -95,6 +136,129 @@ def test_local_paths_bypass_versioned_selection(self):
             "/tmp/xls-local-dev",
         )
 
+    def test_resolve_driver_plan_prefers_installed_driver(self):
+        plan = materialize_xls_bundle.resolve_driver_plan(
+            artifact_source = "auto",
+            driver_version = "0.33.0",
+            installed_driver_root_prefix = "/tools/xlsynth-driver",
+            exists_fn = lambda path: True,
+        )
+        self.assertEqual(plan["mode"], "installed")
+        self.assertEqual(plan["driver"], Path("/tools/xlsynth-driver/0.33.0/bin/xlsynth-driver"))
+
+    def test_resolve_driver_plan_auto_falls_back_to_download(self):
+        plan = materialize_xls_bundle.resolve_driver_plan(
+            artifact_source = "auto",
+            driver_version = "0.33.0",
+            installed_driver_root_prefix = "/tools/xlsynth-driver",
+            exists_fn = lambda path: False,
+        )
+        self.assertEqual(plan, {"mode": "download", "driver_version": "0.33.0"})
+
+    def test_resolve_driver_plan_uses_declared_installed_driver_input(self):
+        plan = materialize_xls_bundle.resolve_driver_plan(
+            artifact_source = "auto",
+            driver_version = "0.33.0",
+            installed_driver_root_prefix = "/unavailable/xlsynth-driver",
+            driver_input = "external/toolchain/host_xlsynth-driver",
+            exists_fn = lambda path: False,
+        )
+        self.assertEqual(plan["mode"], "auto_driver_input")
+        self.assertEqual(plan["driver"], Path("external/toolchain/host_xlsynth-driver"))
+        self.assertEqual(plan["driver_version"], "0.33.0")
+        self.assertEqual(plan["installed_driver_root_prefix"], "/unavailable/xlsynth-driver")
+
+    def test_resolve_driver_plan_rejects_declared_local_driver_input_without_plan_path(self):
+        with self.assertRaisesRegex(ValueError, "local_paths driver materialization requires local_driver_path"):
+            materialize_xls_bundle.resolve_driver_plan(
+                artifact_source = "local_paths",
+                driver_version = "",
+                local_driver_path = "",
+                driver_input = "external/toolchain/host_xlsynth-driver",
+            )
+
+    def test_resolve_driver_plan_rejects_declared_local_driver_input_mismatch(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            root = Path(tempdir)
+            configured_driver = root / "configured" / "xlsynth-driver"
+            configured_driver.parent.mkdir()
+            configured_driver.write_text("#!/bin/sh\nexit 0\n", encoding = "utf-8")
+            configured_driver.chmod(0o755)
+            declared_driver = root / "declared" / "host_xlsynth-driver"
+            declared_driver.parent.mkdir()
+            declared_driver.write_text("#!/bin/sh\nexit 127\n", encoding = "utf-8")
+            declared_driver.chmod(0o755)
+
+            with self.assertRaisesRegex(ValueError, "local_paths declared driver input must match local_driver_path"):
+                materialize_xls_bundle.resolve_driver_plan(
+                    artifact_source = "local_paths",
+                    driver_version = "",
+                    local_driver_path = str(configured_driver),
+                    driver_input = str(declared_driver),
+                )
+
+    def test_resolve_driver_plan_uses_declared_local_driver_input_matching_plan_path(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            root = Path(tempdir)
+            configured_driver = root / "configured" / "xlsynth-driver"
+            configured_driver.parent.mkdir()
+            configured_driver.write_text("#!/bin/sh\nexit 0\n", encoding = "utf-8")
+            configured_driver.chmod(0o755)
+            declared_driver = root / "declared" / "host_xlsynth-driver"
+            declared_driver.parent.mkdir()
+            declared_driver.symlink_to(configured_driver)
+
+            plan = materialize_xls_bundle.resolve_driver_plan(
+                artifact_source = "local_paths",
+                driver_version = "",
+                local_driver_path = str(configured_driver),
+                driver_input = str(declared_driver),
+            )
+            self.assertEqual(
+                plan,
+                {
+                    "mode": "local_paths",
+                    "driver": declared_driver,
+                },
+            )
+
+    def test_auto_driver_input_materialization_falls_back_when_declared_input_fails(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            root = Path(tempdir)
+            declared_driver = root / "declared" / "host_xlsynth-driver"
+            declared_driver.parent.mkdir()
+            declared_driver.write_text("", encoding = "utf-8")
+            declared_driver.chmod(0o755)
+
+            fallback_driver = root / "installed" / "0.33.0" / "bin" / "xlsynth-driver"
+            fallback_driver.parent.mkdir(parents = True)
+            fallback_driver.write_text(
+                """#!/bin/sh
+if [ "${1:-}" = "--version" ]; then
+  printf 'xlsynth-driver 0.33.0 fallback\\n'
+  exit 0
+fi
+printf 'fallback body\\n'
+""",
+                encoding = "utf-8",
+            )
+            fallback_driver.chmod(0o755)
+
+            output_driver = root / "out" / "xlsynth-driver"
+            materialize_xls_bundle.materialize_driver_binary(
+                repo_root = root / "repo",
+                plan = {
+                    "mode": "auto_driver_input",
+                    "driver": declared_driver,
+                    "driver_version": "0.33.0",
+                    "installed_driver_root_prefix": str(root / "installed"),
+                },
+                driver_output = output_driver,
+                libxls_path = root / "runtime" / "libxls.so",
+                dslx_stdlib_path = root / "stdlib",
+            )
+            self.assertIn("fallback body", output_driver.read_text(encoding = "utf-8"))
+
     def test_download_only_rejects_installed_prefixes(self):
         with self.assertRaises(ValueError):
             materialize_xls_bundle.resolve_artifact_plan(
@@ -489,6 +653,32 @@ def test_materialize_toolchain_surface_records_driver_capabilities(self):
             self.assertEqual(metadata["driver_supports_sv_struct_field_ordering"], "false")
             self.assertTrue((repo_root / "xlsynth-driver").exists())
 
+    def test_materialize_driver_binary_copies_local_driver_output(self):
+        with tempfile.TemporaryDirectory() as tempdir:
+            repo_root = Path(tempdir)
+            source_driver = repo_root / "input-driver"
+            source_driver.write_text("#!/bin/sh\nexit 0\n", encoding = "utf-8")
+            source_driver.chmod(0o755)
+            driver_output = repo_root / "out" / "xlsynth-driver"
+            libxls_path = repo_root / "libxls.so"
+            libxls_path.write_text("xls\n", encoding = "utf-8")
+            stdlib_root = repo_root / "stdlib"
+            stdlib_root.mkdir()
+
+            materialize_xls_bundle.materialize_driver_binary(
+                repo_root,
+                {
+                    "mode": "local_paths",
+                    "driver": source_driver,
+                },
+                driver_output,
+                libxls_path,
+                stdlib_root,
+            )
+
+            self.assertEqual(driver_output.read_text(encoding = "utf-8"), "#!/bin/sh\nexit 0\n")
+            self.assertTrue(os.access(driver_output, os.X_OK))
+
 
 if __name__ == "__main__":
     unittest.main()