From 2bf6fea5035f0351fd405b61a0cdc16aed615045 Mon Sep 17 00:00:00 2001 From: Ali Caglayan Date: Tue, 28 Apr 2026 01:45:12 +0200 Subject: [PATCH] feat(pkg): install layouts for package sets `(deps (package ...))` materializes a scoped install layout under `_build/install//.packages//` containing only the declared package dependencies. This replaces the old alias-based mechanism where `(deps (package foo))` populated the shared `_build/install/` staging area, making every package in the workspace discoverable regardless of what was declared. The layout includes only immediate deps (no transitive closure). Actions should declare what they need explicitly. Environment variables (PATH, OCAMLPATH, CAML_LD_LIBRARY_PATH, OCAMLFIND_IGNORE_DUPS_IN, OCAMLTOP_INCLUDE_PATH, MANPATH) are set from the layout directory and consed onto the directory env via extend_action. The layout only applies to workspace packages. Lock-dir packages and the site/plugin system are unaffected. See doc/dev/install-layouts.md for design details and known limitations. Signed-off-by: Ali Caglayan --- Follow-up work (not blocking): Pre-existing limitations preserved by this PR: - Lock-dir packages: (deps (package lockdir-pkg)) builds the package but does not set up layout env vars. This was already broken before this PR (the old `package` function returned only the build action); this PR documents it as a known TODO rather than fixing or worsening it. dune exec lockdir-tool continues to work via bin/exec.ml's unconditional staging cons. - Sandbox PATH non-relocation: layout dirs on PATH/OCAMLPATH point at un-sandboxed _build/. Works locally; would break remote execution. Same mechanism as the bin-layout limitation in #14432. New limitations from this PR's scope: - _root section collisions (lib_root, share_root, libexec_root) raise hard errors today. Opam-style override via dep DAG ordering is future work. - Install_layout.Key.reverse_table, Bin_layout.Key.reverse_table, and Ppx_driver.Key.reverse_table are three copies of the same digest+ reverse table pattern. Worth a Digest_key functor; non-blocking. Test coverage gaps: - Mixing workspace + lock-dir package deps in a single (deps ...). CR-someday Alizter: factor the Context.for_host fallback used in make_bin_env, combined_package_deps_builder and super_context.ml:make_root_env into a single helper. Signed-off-by: Ali Caglayan --- bin/exec.ml | 16 ++ doc/changes/changed/14373.md | 5 + doc/dev/install-layouts.md | 215 ++++++++++++++++++ .../blackbox-tests/pkg-config-quoting.t/run.t | 1 + otherlibs/dune-site/test/dune | 3 +- src/dune_rules/bin_layout.ml | 10 +- src/dune_rules/bin_layout.mli | 2 +- src/dune_rules/cram/cram_rules.ml | 31 ++- src/dune_rules/dep_conf_eval.ml | 183 +++++++++++---- src/dune_rules/dep_conf_eval.mli | 3 - src/dune_rules/gen_rules.ml | 4 +- src/dune_rules/install_layout.ml | 118 ++++++++++ src/dune_rules/install_layout.mli | 44 ++++ src/dune_rules/install_rules.ml | 76 ++++++- src/dune_rules/install_rules.mli | 9 + src/dune_rules/rocq/rocq_rules.ml | 78 +++++-- src/dune_rules/super_context.ml | 36 +-- src/install/context.ml | 5 +- src/install/roots.ml | 36 ++- src/install/roots.mli | 17 ++ .../test-cases/bin-pform/env-binaries.t | 4 +- .../test-cases/bin-pform/include-deps.t | 4 +- .../bin-pform/inline-tests-include.t | 1 - .../test-cases/bin-pform/inline-tests.t | 3 +- .../test-cases/bin-pform/multiple-bins.t | 4 +- .../test-cases/bin-pform/run-action.t | 4 +- .../test-cases/bin-pform/shared-layout.t | 2 - .../test-cases/bin-pform/test-stanza.t | 1 - .../bin-pform-context-expansion.t | 1 - .../package-deps-context-expansion.t | 39 ++++ test/blackbox-tests/test-cases/dune | 5 + test/blackbox-tests/test-cases/dune-site/dune | 5 + .../test-cases/dune-site/sites-plugin.t | 2 +- .../test-cases/inline-tests/dune | 8 +- test/blackbox-tests/test-cases/package-dep.t | 2 +- .../cram-integration.t | 69 ++++++ .../package-materialization/cycle.t | 65 ++++++ .../test-cases/package-materialization/dune | 7 + .../env/caml-ld-library-path.t | 29 +++ .../env/ocamlfind-ignore-dups-in.t | 23 ++ .../package-materialization/env/ocamlpath.t | 23 ++ .../env/ocamltop-include-path.t | 23 ++ .../package-materialization/env/path.t | 27 +++ .../package-materialization/helpers.sh | 4 + .../package-materialization/inline-tests.t | 44 ++++ .../installed-package.t | 43 ++++ .../package-materialization/mixed-deps.t | 30 +++ .../no-transitive-through-targets.t | 65 ++++++ .../nonexistent-package.t | 19 ++ .../package-materialization/ocamlfind.t | 63 +++++ .../overlapping-directory-targets.t | 51 +++++ .../package-materialization/single-package.t | 65 ++++++ .../strict-package-deps.t | 50 ++++ .../transitive-closure.t | 49 ++++ test/blackbox-tests/test-cases/pkg/dune | 6 + .../test-cases/pkg/sites-plugin.t | 1 + .../test-cases/reporting-of-cycles.t/run.t | 6 - .../test-cases/rocq/plugin-meta.t/run.t | 4 +- .../sandbox/sandboxing-package-deps.t | 14 +- .../stanzas/env/env-bins/nested-env.t/run.t | 3 - .../env/env-bins/private-bin-import.t/run.t | 2 - .../test-cases/toplevel-plugin/dune | 5 + test/blackbox-tests/test-cases/watching/dune | 14 ++ 63 files changed, 1608 insertions(+), 173 deletions(-) create mode 100644 doc/changes/changed/14373.md create mode 100644 doc/dev/install-layouts.md create mode 100644 src/dune_rules/install_layout.ml create mode 100644 src/dune_rules/install_layout.mli create mode 100644 test/blackbox-tests/test-cases/custom-cross-compilation/package-deps-context-expansion.t create mode 100644 test/blackbox-tests/test-cases/dune-site/dune create mode 100644 test/blackbox-tests/test-cases/package-materialization/cram-integration.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/cycle.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/dune create mode 100644 test/blackbox-tests/test-cases/package-materialization/env/caml-ld-library-path.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/env/ocamlfind-ignore-dups-in.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/env/ocamlpath.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/env/ocamltop-include-path.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/env/path.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/helpers.sh create mode 100644 test/blackbox-tests/test-cases/package-materialization/inline-tests.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/installed-package.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/mixed-deps.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/no-transitive-through-targets.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/nonexistent-package.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/ocamlfind.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/overlapping-directory-targets.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/single-package.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/strict-package-deps.t create mode 100644 test/blackbox-tests/test-cases/package-materialization/transitive-closure.t create mode 100644 test/blackbox-tests/test-cases/toplevel-plugin/dune diff --git a/bin/exec.ml b/bin/exec.ml index ef9a65d3133..311d875e2c9 100644 --- a/bin/exec.ml +++ b/bin/exec.ml @@ -108,6 +108,20 @@ let program_not_built_yet prog = ] ;; +(* Cons the workspace's install staging dir ([_build/install//]) onto + the env's path-like vars (OCAMLPATH, PATH, etc.) for [dune exec]. This + does NOT trigger any build; if the staging dir isn't populated (no + [@install] was run), the entries on the path simply don't resolve. The + point is to preserve the old [dune exec] contract that workspace + libraries are visible via OCAMLPATH to anything the binary delegates to + (findlib, ocamlfind, dune-site's plugin lookup, etc.). Per-action rule + envs are unaffected — only [dune exec]'s own env extension. *) +let extend_with_staging_env context base = + let staging = Install.Context.dir ~context in + let roots = Install.Roots.opam_from_prefix staging ~relative:Path.Build.relative in + Install.Roots.add_to_env roots base +;; + let build_prog ~no_rebuild ~prog p = if no_rebuild then @@ -176,6 +190,7 @@ let step ~prog ~args ~common ~no_rebuild ~context ~on_exit () = Memo.parallel_map args ~f:(Cmd_arg.expand ~root:(Common.root common) ~sctx) in let* env = Super_context.context_env sctx in + let env = extend_with_staging_env context env in Memo.of_non_reproducible_fiber @@ Dune_engine.Process.run_inherit_std_in_out ~dir:(Path.of_string Fpath.initial_cwd) @@ -305,6 +320,7 @@ let exec_building_directly ~common ~config ~context ~prog ~args ~no_rebuild = and* args = Memo.parallel_map ~f:(Cmd_arg.expand ~root:(Common.root common) ~sctx) args in + let env = extend_with_staging_env context env in Util.restore_cwd_and_execve (Common.root common) prog args env) ;; diff --git a/doc/changes/changed/14373.md b/doc/changes/changed/14373.md new file mode 100644 index 00000000000..b7d5df56381 --- /dev/null +++ b/doc/changes/changed/14373.md @@ -0,0 +1,5 @@ +- `(deps (package ...))` now exposes only the directly declared packages to + the action's environment (`OCAMLPATH`, `PATH`, etc.). Previously, other + packages in the workspace could be discoverable via the shared install + staging area. Actions that relied on undeclared packages being visible + must declare them explicitly. (#14373, @Alizter) diff --git a/doc/dev/install-layouts.md b/doc/dev/install-layouts.md new file mode 100644 index 00000000000..79e7669d72c --- /dev/null +++ b/doc/dev/install-layouts.md @@ -0,0 +1,215 @@ +# Install Layouts for Package Sets + +## Overview + +`(deps (package ...))` materializes a scoped install layout under +`_build/install//.packages//` containing only the declared +package dependencies. This replaces the old alias-based mechanism where +`(deps (package foo))` depended on the `.foo-files` install alias, which +populated the `_build/install/` staging area shared by all packages. + +## Motivation + +The `_build/install/` staging area shared by all packages causes several +problems: + +1. Actions can silently depend on packages they did not declare via the shared + environment variables (OCAMLPATH, PATH, etc.). Whether an action succeeds + can depend on what other packages happened to be built, making builds + non-deterministic. `(strict_package_deps)` validates that dependencies are + declared but does not prevent undeclared packages from being visible at + runtime. + +2. The shared staging area can cause rule collisions and dependency cycles. + Multiple packages installing directory targets to the same section root + crash ([#13307]). Computing install artifacts globally creates cycles in + monorepo builds ([#13500]) and Coq compositions ([#7908]). + +3. Lock-dir packages that depend on workspace packages need a scoped install + prefix to build against. The shared staging area does not provide this. This + is the "in-and-out" problem ([#8652]): when a lock-dir package depends on a + workspace package which depends on another lock-dir package, dune cannot + resolve the dependency without per-package install layouts. + +[#7908]: https://github.com/ocaml/dune/issues/7908 +[#8652]: https://github.com/ocaml/dune/issues/8652 +[#13307]: https://github.com/ocaml/dune/issues/13307 +[#13500]: https://github.com/ocaml/dune/issues/13500 + +## Design + +### Package kinds + +`Package_db.find_package` classifies packages into three kinds: + +- `Local`: workspace packages defined in `dune-project`. These get layout + file deps and env vars. + +- `Build`: lock-dir packages from `dune.lock`. `(deps (package foo))` runs + the package's build action but does not set up env vars for the consuming + action. This is a known bug. See the TODO section below. + +- `Installed`: externally installed packages (found via findlib). Depended + on directly. Already on the system OCAMLPATH. + +### Immediate deps only + +The layout includes only the immediate packages listed in `(deps (package +...))`. No transitive expansion is performed. Actions should declare what +they need explicitly. + +This is a deliberate design choice: + +1. Transitive closure cannot traverse lock-dir packages (they are not + workspace packages), so it gives incomplete results in mixed + workspace/lock-dir setups. Immediate deps avoid this inconsistency. + +2. Workspace package compilation is handled by dune internally via `Lib.DB`, + not via OCAMLPATH. The only consumer of OCAMLPATH in the layout is + user-written rule actions, where explicit deps are appropriate. + +3. Immediate deps keep the layout tractable for the "in-and-out" problem + ([#8652]): when a lock-dir package depends on a workspace package, the + layout for that workspace package can be provided to the lock-dir + package's build env without needing to chase transitive deps through + other lock-dir packages. + +### Environment variables + +The layout sets PATH, OCAMLPATH, CAML_LD_LIBRARY_PATH, +OCAMLFIND_IGNORE_DUPS_IN, OCAMLTOP_INCLUDE_PATH, and MANPATH from the +layout's directory structure. These are defined by `Install.Roots.path_vars` +and consed onto the directory environment in `extend_action` +(super_context.ml), so external/system paths remain visible. The env vars +are propagated to rule actions, cram tests, cinaps, mdx, and inline tests. + +### Staging area + +`(deps (package ...))` no longer populates the `_build/install/` staging +area. The staging area is only populated by `dune build @install` / +`dune install`. Code that hardcodes staging area paths must use `@install` +or `%{bin:foo}`. + +### Environment hermeticity + +`make_root_env` returns only `Context.installed_env` with no staging paths +attached. Actions only see declared dependencies via layout env vars consed +in `extend_action`. This is what makes the strict-deps property hold: +nothing in the workspace is reachable unless declared. + +### `dune exec` and the staging area + +`dune exec` runs binaries outside the rule machinery, so it cannot inherit +per-action env from `combined_package_deps_builder`. To preserve the +documented `dune exec` contract that workspace libraries are reachable via +OCAMLPATH (see the man page: "you will have access to the libraries defined +in the workspace using your usual directives"), `bin/exec.ml` conses +`_build/install//{lib,bin,...}` onto the path-like env vars of +the executed process. + +This cons is unconditional and does not trigger any build. If +`dune build @install` (or any rule that drags install entries in) was run +beforehand, the staging dir is populated and workspace libraries are +visible to anything the binary delegates to (findlib, ocamlfind, dune-site +plugin lookup, etc.). If not, the entries on the path simply don't +resolve. Per-action rule envs are unaffected; only `dune exec`'s own env +extension is consed. + +### Sites and plugins + +The dune-site mechanism uses `DUNE_DIR_LOCATIONS` to tell binaries where +their sites live. `Site_env.add_packages_env` walks workspace stanzas to +discover site/plugin packages and sets the env var to point at the +staging area (`_build/install//lib//
/`). Combined +with the `dune exec` staging cons above, this means: + +- A binary built locally and run via `dune exec foo.exe` sees plugins iff + the staging dir is populated. The dune-site / sites cram tests + consequently run `dune build @install` before `dune exec`. +- Cram tests that exercise dune-site libraries (`(libraries dune-site + dune-site.plugins)`) must declare both `(package dune-site)` and + `(package dune-private-libs)` in their cram-level `dune` setup. + `dune-site` re-exports `dune-private-libs.dune-section`, and the layout + does not auto-expand transitive package deps (see "Immediate deps only" + above). The same pattern applies to other re-exporting libraries (e.g. + `(package stdune)` requires `(package dyn) (package ordering) (package + pp) (package top-closure) (package csexp) (package fs-io)`). + +The current staging cons is sufficient for the existing test surface and +matches pre-install-layouts behaviour. + +### Package set structure and `_root` section collisions + +The layout merges all packages' install entries into a single directory tree. +For scoped sections (`lib`, `share`, `doc`, `etc`), each package installs +under its own subdirectory (`lib//`), so collisions are impossible +by construction. The unordered set is the correct data structure. + +Collisions can only occur in `_root` sections (`lib_root`, `share_root`, +`libexec_root`), which install directly to the section root without package +namespacing ([opam manual][opam-install-format], [opam#2153]). In opam, +`_root` sections serve specific use cases: + +- `ocamlfind` installs `topfind` into the compiler's stdlib directory via + `lib_root` ([opam#45]). +- Cross-compiled C libraries install `.pc` files to `lib/pkgconfig/` via + `lib_root`. +- `share_root` is used for shared data directories (e.g. emacs site-lisp). + +These are rare. Collisions in `_root` sections are currently errors. To +support opam-style override semantics in the future, the layout would +resolve `_root` collisions using dependency edges: if B depends on A, B's +`_root` entries take priority. Diamond collisions (incomparable packages) +would remain errors. This only affects `_root` handling and does not +require changing the core layout mechanism. + +[opam-install-format]: https://opam.ocaml.org/doc/Manual.html +[opam#2153]: https://github.com/ocaml/opam/issues/2153 +[opam#45]: https://github.com/ocaml/opam/issues/45 + +## Future work + +### Lock-dir build layouts + +The layout mechanism can support the "in-and-out" problem ([#8652]): when a +lock-dir package depends on a workspace package, a layout containing the +workspace package's artifacts can be provided to the lock-dir package's +build env. The opam-like directory structure matches what build systems +expect at an install prefix. + +Longer term, the layout overlay mechanism could replace isolated per-package +install directories for lock-dir builds entirely. Packages would build +against a layout that overlays all their dependencies' installed files into +a single virtual prefix, using dependency structure for overlay ordering. + +### Incremental layout construction + +If A depends on B depends on C, the layouts for their builds are {C}, {B,C}, +and {A,B,C}. Instead of computing each independently, layouts can be built +incrementally: {B,C} hardlinks from {C} and adds B's entries, {A,B,C} +hardlinks from {B,C} and adds A's entries. This makes `_root` overrides +natural: each step replaces entries from the smaller layout. The key +question is whether the digest/key scheme can encode this incrementality or +whether a structural (DAG-based) key is needed. + +## TODO + +- Lock-dir package deps don't set up env vars for the consuming action. + `Package_db.find_package` classifies lock-dir packages as `Build` and + runs their build action but does not set up layout env vars. `dune exec + lockdir-tool` continues to work via `bin/exec.ml`'s unconditional + staging cons. +- Sandbox PATH non-relocation: layout dirs on PATH/OCAMLPATH point at + un-sandboxed `_build/`. Works locally; would break remote execution. + Same mechanism as the bin-layout limitation in #14432. +- `Install_layout.Key.reverse_table`, `Bin_layout.Key.reverse_table`, and + `Ppx_driver.Key.reverse_table` are three copies of the same + digest+reverse-table pattern. Worth a `Digest_key` functor; non-blocking. +- Factor the `Context.for_host` fallback used in `make_bin_env`, + `combined_package_deps_builder`, and `super_context.ml:make_root_env` + into a single helper. +- Missing test: mixing workspace and lock-dir package deps in a single + `(deps ...)`. +- `overlapping-directory-targets.t` halves test different things: the old + lang section tests `dune build @install` (not package deps), while the + new lang section tests layout rule collision. Should be split. diff --git a/otherlibs/configurator/test/blackbox-tests/pkg-config-quoting.t/run.t b/otherlibs/configurator/test/blackbox-tests/pkg-config-quoting.t/run.t index 08ab88fa1f4..1dca35373fc 100644 --- a/otherlibs/configurator/test/blackbox-tests/pkg-config-quoting.t/run.t +++ b/otherlibs/configurator/test/blackbox-tests/pkg-config-quoting.t/run.t @@ -1,4 +1,5 @@ These tests show how various pkg-config invocations get quotes (and test specifying a custom PKG_CONFIG): + $ dune build @install $ PKG_CONFIG=$PWD/_build/install/default/bin/pkg-config dune build 2>&1 | awk '/run:.*bin\/pkg-config/{a=1}/stderr/{a=0}a' run: $TESTCASE_ROOT/_build/install/default/bin/pkg-config --print-errors gtk+-quartz-3.0 -> process exited with code 0 diff --git a/otherlibs/dune-site/test/dune b/otherlibs/dune-site/test/dune index 5f479b2ac32..7c6cb25098e 100644 --- a/otherlibs/dune-site/test/dune +++ b/otherlibs/dune-site/test/dune @@ -2,7 +2,8 @@ (applies_to :whole_subtree) (deps (package dune) - (package dune-site))) + (package dune-site) + (package dune-private-libs))) ; The test being broken on CI, we deactivate it when ; we detect that the CI environment variable is not diff --git a/src/dune_rules/bin_layout.ml b/src/dune_rules/bin_layout.ml index abf7934ab82..7eaad8c8f02 100644 --- a/src/dune_rules/bin_layout.ml +++ b/src/dune_rules/bin_layout.ml @@ -89,12 +89,10 @@ let make_dispatch ~dir subdirs f = let gen_rules context_name ~dir rest = match rest with - | [] -> Memo.return (make_dispatch ~dir Subdir_set.all (fun () -> Memo.return ())) + | [] -> make_dispatch ~dir Subdir_set.all (fun () -> Memo.return ()) | [ key ] -> - Memo.return - (make_dispatch ~dir Subdir_set.empty (fun () -> - symlink_rules_for_key context_name ~dir key)) + make_dispatch ~dir Subdir_set.empty (fun () -> + symlink_rules_for_key context_name ~dir key) | _ :: _ :: _ -> - Memo.return - (Build_config.Gen_rules.redirect_to_parent Build_config.Gen_rules.Rules.empty) + Build_config.Gen_rules.redirect_to_parent Build_config.Gen_rules.Rules.empty ;; diff --git a/src/dune_rules/bin_layout.mli b/src/dune_rules/bin_layout.mli index 27d929ea0d5..08dc96b97b7 100644 --- a/src/dune_rules/bin_layout.mli +++ b/src/dune_rules/bin_layout.mli @@ -16,4 +16,4 @@ val gen_rules : Context_name.t -> dir:Path.Build.t -> string list - -> Build_config.Gen_rules.result Memo.t + -> Build_config.Gen_rules.result diff --git a/src/dune_rules/cram/cram_rules.ml b/src/dune_rules/cram/cram_rules.ml index 4102f8f4913..1012a226625 100644 --- a/src/dune_rules/cram/cram_rules.ml +++ b/src/dune_rules/cram/cram_rules.ml @@ -8,7 +8,7 @@ module Spec = struct ; extra_aliases : Alias.Name.Set.t ; deps : unit Action_builder.t list ; sandbox : Sandbox_config.t Action_builder.t - ; bin_env : Env.t Action_builder.t + ; env : Env.t Action_builder.t ; enabled_if : (Expander.t * Blang.t) list ; locks : Path.Set.t Action_builder.t ; packages : Package.Name.Set.t @@ -26,7 +26,7 @@ module Spec = struct ; locks = Action_builder.return Path.Set.empty ; deps = [] ; sandbox = Action_builder.return Sandbox_config.needs_sandboxing - ; bin_env = Action_builder.return Env.empty + ; env = Action_builder.return Env.empty ; packages = Package.Name.Set.empty ; timeout = None ; conflict_markers = Ignore @@ -64,7 +64,7 @@ let test_rule ; deps ; locks ; sandbox - ; bin_env + ; env ; packages = _ ; timeout ; conflict_markers @@ -140,8 +140,8 @@ let test_rule let+ (_ : Path.Set.t) = Action_builder.dyn_memo_deps deps in () and+ () = Action_builder.paths setup_scripts - and+ sandbox = sandbox - and+ bin_env = bin_env + and+ sandbox + and+ env and+ locks = locks >>| Path.Set.to_list in Cram_exec.run ~src:(Path.build script) @@ -156,7 +156,7 @@ let test_rule ~setup_scripts shell |> Action.Full.make ~locks ~sandbox - |> Action.Full.add_env bin_env) + |> Action.Full.add_env env) |> Action_builder.with_file_targets ~file_targets:[ output ] |> Super_context.add_rule sctx ~dir ~loc in @@ -261,11 +261,11 @@ let rules ~sctx ~dir tests project = | false -> Memo.return (runtest_alias, acc) | true -> let+ expander = Super_context.expander sctx ~dir in - let deps, sandbox, bin_env = + let deps, sandbox, env = match stanza.deps with - | None -> acc.deps, acc.sandbox, acc.bin_env + | None -> acc.deps, acc.sandbox, acc.env | Some deps -> - let action_env, _, sandbox = + let env, _, sandbox = Dep_conf_eval.named ~expander Sandbox_config.no_special_requirements @@ -277,16 +277,13 @@ let rules ~sctx ~dir tests project = and+ sandbox = sandbox in Sandbox_config.inter acc sandbox in - let bin_env = + let env = let open Action_builder.O in - let+ acc = acc.bin_env - and+ env = action_env in + let+ acc = acc.env + and+ env in Env_path.extend_env_concat_path acc env in - (* [action_env]'s deps are registered when [bin_env] is - consumed at the rule site (it folds the env's - action_builder evaluation into [bin_env]). *) - acc.deps, sandbox, bin_env + acc.deps, sandbox, env in let locks = let open Action_builder.O in @@ -369,7 +366,7 @@ let rules ~sctx ~dir tests project = ; extra_aliases ; packages ; sandbox - ; bin_env + ; env ; timeout ; conflict_markers ; setup_scripts diff --git a/src/dune_rules/dep_conf_eval.ml b/src/dune_rules/dep_conf_eval.ml index 8d3fa07d25d..523ae16bec4 100644 --- a/src/dune_rules/dep_conf_eval.ml +++ b/src/dune_rules/dep_conf_eval.ml @@ -20,17 +20,6 @@ let make_alias expander s = Expander.expand_path expander s >>| Alias.of_user_written_path ~loc ;; -let package_install ~(context : Build_context.t) ~(pkg : Package.t) = - let dir = - let dir = Package.dir pkg in - Path.Build.append_source context.build_dir dir - in - let name = Package.name pkg in - sprintf ".%s-files" (Package.Name.to_string name) - |> Alias.Name.of_string - |> Alias.make ~dir -;; - type dep_evaluation_result = | Simple of Path.t list Memo.t | Other of Path.t list Action_builder.t @@ -45,7 +34,7 @@ let to_action_builder = function | Include_result pair -> Action_builder.map pair ~f:fst ;; -let include_bin_env = function +let include_action_env = function | Simple _ | Other _ -> None | Include_result pair -> Some (Action_builder.map pair ~f:snd) ;; @@ -156,6 +145,12 @@ let make_bin_env expander bin_names = Some (Bin.cons_path (Path.build layout_dir) ~_PATH))) ;; +let package_dep_swvs (dep : Dep_conf.t) = + match dep with + | Package p -> [ p, String_with_vars.loc p ] + | _ -> [] +;; + let add_sandbox_config acc (dep : Dep_conf.t) = match dep with | Sandbox_config cfg -> Sandbox_config.inter acc (make_sandboxing_config cfg) @@ -181,15 +176,19 @@ let rec dir_contents ~loc d = >>| List.concat ;; -let package loc pkg (context : Build_context.t) ~dune_version = - let pkg = Package.Name.of_string pkg in +let package loc pkg_name (context : Build_context.t) ~dune_version = Action_builder.of_memo (let open Memo.O in let* package_db = Package_db.create context.name in - Package_db.find_package package_db pkg) + Package_db.find_package package_db pkg_name) >>= function | Some (Build build) -> build - | Some (Local pkg) -> Alias_builder.alias (package_install ~context ~pkg) + | Some (Local _) -> + (* The named/unnamed paths skip [Package _] before reaching here so that + [combined_package_deps_builder] handles the whole package set at once. + This arm only fires from [unnamed_get_paths] (e.g. + [(public_headers (package foo))]) where a no-op is fine. *) + Action_builder.return () | Some (Installed pkg) -> if dune_version < (2, 9) then @@ -224,7 +223,7 @@ let package loc pkg (context : Build_context.t) ~dune_version = (fun () -> User_error.raise ~loc - [ Pp.textf "Package %s does not exist" (Package.Name.to_string pkg) ]) + [ Pp.textf "Package %s does not exist" (Package.Name.to_string pkg_name) ]) } ;; @@ -238,9 +237,9 @@ let rec dep expander : Dep_conf.t -> _ = function let* project = Action_builder.of_memo @@ Dune_load.find_project ~dir in expand_include ~dir ~project s in - let builder, _bindings, bin_env = named_paths_builder ~expander deps in + let builder, _bindings, action_env = named_paths_builder ~expander deps in let+ paths = builder - and+ env = bin_env in + and+ env = action_env in paths, env in Include_result (Action_builder.memoize "include-eval" pair) @@ -294,7 +293,7 @@ let rec dep expander : Dep_conf.t -> _ = function | Package p -> Other (let+ () = - let* pkg = Expander.expand_str expander p in + let* pkg_name = Expander.expand_str expander p >>| Package.Name.of_string in let context = Build_context.create ~name:(Expander.context expander) in let loc = String_with_vars.loc p in let* dune_version = @@ -304,7 +303,7 @@ let rec dep expander : Dep_conf.t -> _ = function Dune_load.find_project ~dir:(Expander.dir expander) >>| Dune_project.dune_version in - package loc pkg context ~dune_version + package loc pkg_name context ~dune_version in []) | Universe -> @@ -318,24 +317,97 @@ let rec dep expander : Dep_conf.t -> _ = function []) | Sandbox_config _ -> Other (Action_builder.return []) +and combined_package_deps_builder expander pkgs = + let open Action_builder.O in + (* Resolve packages and name the layout dir in the host context, same as + [make_bin_env] above. *) + let* host_name = + Action_builder.of_memo + (let open Memo.O in + Expander.host_context expander >>| Context.name) + in + let context = Build_context.create ~name:host_name in + let* package_db = Action_builder.of_memo (Package_db.create context.name) in + let* classified = + Action_builder.List.map pkgs ~f:(fun (swv, loc) -> + let* name = Expander.expand_str expander swv in + let pkg = Package.Name.of_string name in + let+ found = Action_builder.of_memo (Package_db.find_package package_db pkg) in + loc, pkg, found) + in + let local_package_names = + List.filter_map classified ~f:(fun (_, _, found) -> + match found with + | Some (Package_db.Local pkg) -> Some (Package.name pkg) + | _ -> None) + in + let* env = + match local_package_names with + | [] -> Action_builder.return Env.empty + | _ -> + let* files, layout_root = + Action_builder.of_memo + @@ + let open Memo.O in + let layout_root = + Install_layout.lib_root context.name local_package_names + |> Path.Build.parent_exn + in + let+ files = Install_layout.files context.name local_package_names in + files, layout_root + in + let+ () = Action_builder.paths files in + let roots = + Install.Roots.opam_from_prefix layout_root ~relative:Path.Build.relative + in + Install.Roots.add_to_env roots Env.empty + in + let+ () = + Action_builder.List.iter classified ~f:(fun (loc, pkg_name, found) -> + match found with + | Some (Local _) -> Action_builder.return () + | Some (Build build) -> build + | Some (Installed _) | None -> + let* dune_version = + Action_builder.of_memo + (let open Memo.O in + Dune_load.find_project ~dir:(Expander.dir expander) + >>| Dune_project.dune_version) + in + package loc pkg_name context ~dune_version) + in + env + and named_paths_builder ~expander l = - let builders, bindings, bin_names, include_envs = + let builders, bindings, combined_packages_builder, bin_names, include_envs = let expander = prepare_expander expander in + let package_swvs = + List.concat_map l ~f:(function + | Bindings.Unnamed dep -> package_dep_swvs dep + | Bindings.Named (_, deps) -> List.concat_map deps ~f:package_dep_swvs) + in let bin_names = List.concat_map l ~f:(function | Bindings.Unnamed dep -> Option.to_list (bin_dep_name dep) | Bindings.Named (_, deps) -> List.filter_map deps ~f:bin_dep_name) in + let combined_packages_builder = + match package_swvs with + | [] -> None + | pkgs -> Some (combined_package_deps_builder expander pkgs) + in let builders, bindings, include_envs = List.fold_left l ~init:([], Pform.Map.empty, []) ~f:(fun (builders, bindings, envs) x -> match x with + | Bindings.Unnamed (Dep_conf.Package _) + when Option.is_some combined_packages_builder -> builders, bindings, envs | Bindings.Unnamed x -> let r = dep expander x in let envs = - match include_bin_env r with + match include_action_env r with | Some e -> e :: envs | None -> envs in @@ -358,7 +430,7 @@ and named_paths_builder ~expander l = in let envs = List.fold_left x ~init:envs ~f:(fun envs r -> - match include_bin_env r with + match include_action_env r with | Some e -> e :: envs | None -> envs) in @@ -399,10 +471,24 @@ and named_paths_builder ~expander l = in x :: builders, bindings, envs)) in - builders, bindings, bin_names, include_envs + builders, bindings, combined_packages_builder, bin_names, include_envs in - let bin_env = - let outer = make_bin_env expander bin_names in + let builders, package_env = + match combined_packages_builder with + | None -> builders, Action_builder.return Env.empty + | Some b -> + let open Action_builder.O in + let b = Action_builder.memoize "combined-package-deps" b in + (b >>| fun _ -> []) :: builders, b + in + let bin_env = make_bin_env expander bin_names in + let action_env = + let outer = + let open Action_builder.O in + let+ package_env = package_env + and+ bin_env = bin_env in + Env_path.extend_env_concat_path package_env bin_env + in List.fold_left include_envs ~init:outer ~f:(fun acc env -> let open Action_builder.O in let+ acc = acc @@ -410,11 +496,11 @@ and named_paths_builder ~expander l = Env_path.extend_env_concat_path acc env) in let builder = List.rev builders |> Action_builder.all >>| List.concat in - builder, bindings, bin_env + builder, bindings, action_env ;; let named sandbox ~expander l = - let builder, bindings, bin_env = named_paths_builder ~expander l in + let builder, bindings, action_env = named_paths_builder ~expander l in let builder = Action_builder.memoize ~cutoff:(List.equal Value.equal) @@ -448,7 +534,7 @@ let named sandbox ~expander l = Action_builder.memoize "deps action_env" (let+ _paths = builder - and+ env = bin_env in + and+ env = action_env in env) in action_env, expander, sandbox @@ -456,16 +542,19 @@ let named sandbox ~expander l = let unnamed sandbox ~expander l = let expander = prepare_expander expander in + let package_swvs = List.concat_map l ~f:package_dep_swvs in + let package_env = + match package_swvs with + | [] -> Action_builder.return Env.empty + | pkgs -> combined_package_deps_builder expander pkgs + in + let has_combined = not (List.is_empty package_swvs) in let bin_names = List.filter_map l ~f:bin_dep_name in - let builders, include_envs = - List.fold_left l ~init:([], []) ~f:(fun (builders, envs) x -> - let r = dep expander x in - let envs = - match include_bin_env r with - | Some e -> e :: envs - | None -> envs - in - to_action_builder r :: builders, envs) + let include_envs = + List.fold_left l ~init:[] ~f:(fun envs x -> + match include_action_env (dep expander x) with + | Some e -> e :: envs + | None -> envs) in let bin_env = let outer = make_bin_env expander bin_names in @@ -479,12 +568,16 @@ let unnamed sandbox ~expander l = Action_builder.memoize "deps action_env" (let+ () = - List.fold_left builders ~init:(Action_builder.return ()) ~f:(fun acc b -> - let+ () = acc - and+ _x = b in - ()) - and+ env = bin_env in - env) + List.fold_left l ~init:(Action_builder.return ()) ~f:(fun acc x -> + match x with + | Dep_conf.Package _ when has_combined -> acc + | _ -> + let+ () = acc + and+ _x = to_action_builder (dep expander x) in + ()) + and+ package_env = package_env + and+ bin_env = bin_env in + Env_path.extend_env_concat_path package_env bin_env) in action_env, List.fold_left l ~init:sandbox ~f:add_sandbox_config ;; diff --git a/src/dune_rules/dep_conf_eval.mli b/src/dune_rules/dep_conf_eval.mli index 87782c14cce..edf869ee928 100644 --- a/src/dune_rules/dep_conf_eval.mli +++ b/src/dune_rules/dep_conf_eval.mli @@ -2,9 +2,6 @@ open Import -(** Alias for all the files in [_build/install] that belong to this package *) -val package_install : context:Build_context.t -> pkg:Package.t -> Alias.t - (** Evaluates unnamed dependency specifications. The returned builder both registers the rule's dependencies (as a side effect of evaluation) and produces an [Env.t] augmentation that callers should fold into the action diff --git a/src/dune_rules/gen_rules.ml b/src/dune_rules/gen_rules.ml index fd05bfae291..e294b0787a4 100644 --- a/src/dune_rules/gen_rules.ml +++ b/src/dune_rules/gen_rules.ml @@ -797,7 +797,9 @@ let gen_rules ctx ~dir components = in Gen_rules.make ~build_dir_only_sub_dirs (Memo.return Rules.empty) | ctx :: ".binaries" :: rest -> - Bin_layout.gen_rules (Context_name.of_string ctx) ~dir rest + Bin_layout.gen_rules (Context_name.of_string ctx) ~dir rest |> Memo.return + | ctx :: ".packages" :: rest -> + Install_rules.layout_gen_rules (Context_name.of_string ctx) ~dir rest |> Memo.return | ctx :: _ -> let ctx = Context_name.of_string ctx in with_context ctx ~f:(fun sctx -> diff --git a/src/dune_rules/install_layout.ml b/src/dune_rules/install_layout.ml new file mode 100644 index 00000000000..539f78f57c7 --- /dev/null +++ b/src/dune_rules/install_layout.ml @@ -0,0 +1,118 @@ +open Import + +module Key : sig + type encoded = Digest.t + + module Decoded : sig + type t = private { packages : Package.Name.t list } + + val of_packages : Package.Name.t list -> t + end + + val encode : Decoded.t -> encoded + val decode : encoded -> Decoded.t +end = struct + type encoded = Digest.t + + module Decoded = struct + type t = { packages : Package.Name.t list } + + let equal x y = List.equal Package.Name.equal x.packages y.packages + + let to_string { packages } = + String.enumerate_and (List.map packages ~f:Package.Name.to_string) + ;; + + let of_packages packages = + let packages = List.sort_uniq packages ~compare:Package.Name.compare in + { packages } + ;; + end + + let reverse_table : (Digest.t, Decoded.t) Table.t = Table.create (module Digest) 128 + + let encode ({ Decoded.packages } as x) = + let y = Digest.repr Repr.(list Package.Name.repr) packages in + match Table.find reverse_table y with + | None -> + Table.set reverse_table y x; + y + | Some x' -> + if Decoded.equal x x' + then y + else + Code_error.raise + "Hash collision between sets of packages" + [ "cached", Dyn.string (Decoded.to_string x') + ; "new", Dyn.string (Decoded.to_string x) + ] + ;; + + let decode y = + match Table.find reverse_table y with + | Some x -> x + | None -> + Code_error.raise + "unknown package set digest (encode was not called first)" + [ "digest", Dyn.string (Digest.to_string y) ] + ;; +end + +type entry = + { src : Path.t + ; relative : Path.Source.t + } + +let entry_resolver_fdecl : (Context_name.t -> Package.Name.t -> entry list Memo.t) Fdecl.t + = + Fdecl.create Dyn.opaque +;; + +let set_entry_resolver f = Fdecl.set entry_resolver_fdecl f + +let dir ~context ~key = + Path.Build.L.relative (Install.Context.dir ~context) [ ".packages"; key ] +;; + +let compute_entries context_name root packages = + let open Memo.O in + let get_entries = Fdecl.get entry_resolver_fdecl in + Memo.parallel_map packages ~f:(fun pkg -> + let+ entries = get_entries context_name pkg in + List.map entries ~f:(fun { src; relative } -> + let dst = Path.Build.append_source root relative in + src, dst, relative)) + >>| List.rev_concat +;; + +let entries_memo = + Memo.create + "install-layout-entries" + ~input: + (module struct + type t = Context_name.t * Digest.t + + let equal (a1, b1) (a2, b2) = Context_name.equal a1 a2 && Digest.equal b1 b2 + let hash (a, b) = Tuple.T2.hash Context_name.hash Digest.hash (a, b) + let to_dyn (a, b) = Dyn.pair Context_name.to_dyn Digest.to_dyn (a, b) + end) + (fun (context_name, digest) -> + let key = Digest.to_string digest in + let root = dir ~context:context_name ~key in + let { Key.Decoded.packages } = Key.decode digest in + compute_entries context_name root packages) +;; + +let encode_packages packages = Key.Decoded.of_packages packages |> Key.encode + +let files context_name packages = + let open Memo.O in + let digest = encode_packages packages in + let+ entries = Memo.exec entries_memo (context_name, digest) in + List.map entries ~f:(fun (_, dst, _) -> Path.build dst) +;; + +let lib_root context_name packages = + let key = Digest.to_string (encode_packages packages) in + Path.Build.relative (dir ~context:context_name ~key) "lib" +;; diff --git a/src/dune_rules/install_layout.mli b/src/dune_rules/install_layout.mli new file mode 100644 index 00000000000..e7a76df94af --- /dev/null +++ b/src/dune_rules/install_layout.mli @@ -0,0 +1,44 @@ +(** Install layout machinery for [(deps (package ...))]. + + Scoped install layout materialised under + [_build/install//.packages//] containing only the declared + package dependencies. The consumer side (queries, env construction) lives + here, low in the dependency stack; the producer side ([gen_rules] for the + directory and the data resolver callback) lives in [install_rules], which + is high in the stack and has access to install entries. The two are + connected by an [Fdecl] seam: [install_rules] calls [set_entry_resolver] + at module init. *) + +open Import + +type entry = + { src : Path.t + ; relative : Path.Source.t + } + +(** Must be called during initialization to wire up install entry resolution. + Called from [install_rules]. *) +val set_entry_resolver : (Context_name.t -> Package.Name.t -> entry list Memo.t) -> unit + +(** Files that would be in the layout for a set of packages. Consumers use + this to set up file-level dependencies. *) +val files : Context_name.t -> Package.Name.t list -> Path.t list Memo.t + +(** The [lib] subdirectory of the layout for a set of packages. Suitable for + adding to [OCAMLPATH]. *) +val lib_root : Context_name.t -> Package.Name.t list -> Path.Build.t + +module Key : sig + type encoded = Digest.t + + module Decoded : sig + type t = private { packages : Package.Name.t list } + end + + val decode : encoded -> Decoded.t +end + +(** The memoized entries computation. Exposed for [install_rules]'s + [gen_rules] which materializes the layout into symlinks. *) +val entries_memo + : (Context_name.t * Digest.t, (Path.t * Path.Build.t * Path.Source.t) list) Memo.Table.t diff --git a/src/dune_rules/install_rules.ml b/src/dune_rules/install_rules.ml index 63398bd495f..edc91605b66 100644 --- a/src/dune_rules/install_rules.ml +++ b/src/dune_rules/install_rules.ml @@ -9,6 +9,15 @@ let install_file ~(package : Package.Name.t) ~findlib_toolchain = |> Filename.of_string_exn ;; +(** Alias for all the files in [_build/install] that belong to a package. *) +let package_install ~(context : Build_context.t) ~(pkg : Package.t) = + let dir = Path.Build.append_source context.build_dir (Package.dir pkg) in + let name = Package.name pkg in + sprintf ".%s-files" (Package.Name.to_string name) + |> Alias.Name.of_string + |> Alias.make ~dir +;; + let with_doc = Package_variable_name.with_doc let need_odoc_config (pkg : Package.t) = @@ -1179,6 +1188,67 @@ let install_entries sctx package = Package.Name.Map.Multi.find packages package ;; +let () = + Install_layout.set_entry_resolver (fun context_name package -> + let open Memo.O in + let+ entries = + let* sctx = Super_context.find_exn context_name in + install_entries sctx package + in + let install_paths = + let roots = Install.Roots.opam_from_prefix Path.root ~relative:Path.relative in + Install.Paths.make ~relative:Path.relative ~package ~roots + in + List.filter_map entries ~f:(fun (s : Install.Entry.Sourced.Unexpanded.t) -> + let entry = s.entry in + match entry.kind with + | Install.Entry.Unexpanded.Source_tree -> None + | File | Directory -> + let relative = + Install.Entry.relative_installed_path entry ~paths:install_paths + |> Path.as_in_source_tree_exn + in + Some { Install_layout.src = Path.build entry.src; relative })) +;; + +let make_dispatch ~dir subdirs f = + let rules = Rules.collect_unit f in + Build_config.Gen_rules.make + ~build_dir_only_sub_dirs: + (Build_config.Gen_rules.Build_only_sub_dirs.singleton ~dir subdirs) + rules +;; + +let symlink_rules_for_key context_name key = + match Digest.from_hex key with + | None -> + (* This dispatch arm fires from [gen_rules.ml] when the build system + visits a path under [.packages//]. A non-hex key here means + the build system asked for a directory we never advertise, which + is an internal invariant violation, not user input. *) + Code_error.raise "invalid install layout key" [ "key", Dyn.string key ] + | Some digest -> + let* entries = Memo.exec Install_layout.entries_memo (context_name, digest) in + Memo.parallel_iter entries ~f:(fun (src, dst, _) -> + let { Action_builder.With_targets.build; targets } = + Action_builder.symlink ~src ~dst + in + Rules.Produce.rule (Rule.make ~info:(Rule.Info.of_loc_opt None) ~targets build)) +;; + +(** Generates the symlink rules for the install layout under + [_build/install//.packages//]. All rules are emitted at + the [] level; visits to deeper subdirectories redirect to the + parent so the engine aggregates them at the root. *) +let layout_gen_rules context_name ~dir rest = + match rest with + | [] -> make_dispatch ~dir Subdir_set.all (fun () -> Memo.return ()) + | [ key ] -> + make_dispatch ~dir Subdir_set.empty (fun () -> symlink_rules_for_key context_name key) + | _ :: _ :: _ -> + Build_config.Gen_rules.redirect_to_parent Build_config.Gen_rules.Rules.empty +;; + let packages = let f sctx = let* packages = Dune_load.packages () in @@ -1398,9 +1468,7 @@ let gen_package_install_file_rules sctx (package : Package.t) = in let* () = let* all_packages = Dune_load.packages () in - let target_alias = - Dep_conf_eval.package_install ~context:build_context ~pkg:package - in + let target_alias = package_install ~context:build_context ~pkg:package in let open Action_builder.O in Rules.Produce.Alias.add_deps target_alias @@ -1414,7 +1482,7 @@ let gen_package_install_file_rules sctx (package : Package.t) = let name = Package.Id.name pkg in Package.Name.Map.find_exn all_packages name in - Dep_conf_eval.package_install ~context:build_context ~pkg |> Dep.alias) ))) + package_install ~context:build_context ~pkg |> Dep.alias) ))) in let action = let findlib_toolchain = Context.findlib_toolchain context in diff --git a/src/dune_rules/install_rules.mli b/src/dune_rules/install_rules.mli index 9f0a8d1763d..aaa1e1fdcc0 100644 --- a/src/dune_rules/install_rules.mli +++ b/src/dune_rules/install_rules.mli @@ -22,3 +22,12 @@ val resolve_package_install_file (** Generate rules for [.dune-package], [META.] files. and [.install] files. *) val gen_project_rules : Super_context.t -> Dune_project.t -> unit Memo.t + +(** Generate the symlink rules that materialize the install layout for a set + of packages at [_build/install//.packages//]. Called from + [gen_rules] when the build system visits that directory. *) +val layout_gen_rules + : Context_name.t + -> dir:Path.Build.t + -> string list + -> Build_config.Gen_rules.result diff --git a/src/dune_rules/rocq/rocq_rules.ml b/src/dune_rules/rocq/rocq_rules.ml index 0f13d4f2489..3b912f9837c 100644 --- a/src/dune_rules/rocq/rocq_rules.ml +++ b/src/dune_rules/rocq/rocq_rules.ml @@ -18,7 +18,7 @@ open Memo.O elsewhere; don't mix with the rest of the code. *) module Util : sig val include_flags : Lib.t list -> _ Command.Args.t - val meta_info : loc:Loc.t option -> context:Context_name.t -> Lib.t -> Path.t option + val meta_info : loc:Loc.t option -> lib_root:Path.Build.t -> Lib.t -> Path.t option (** Given a list of library names, we try to resolve them in order, returning the first one that exists. *) @@ -32,13 +32,15 @@ end = struct let include_flags ts = include_paths ts |> Lib_flags.L.to_iflags - let meta_info ~loc ~context (lib : Lib.t) = + let meta_info ~loc ~lib_root (lib : Lib.t) = let name = Lib.name lib |> Lib_name.to_string in match Lib_info.status (Lib.info lib) with | Public (_, pkg) -> let package = Package.name pkg in let meta_i = - Path.Build.relative (Install.Context.lib_dir ~context ~package) "META" + Path.Build.relative + (Path.Build.relative lib_root (Package.Name.to_string package)) + "META" in Some (Path.build meta_i) | Installed -> None @@ -420,13 +422,13 @@ let libs_of_theory ~lib_db ~theories_deps plugins : (Lib.t list * _) Resolve.Mem ;; (* compute include flags and mlpack rules *) -let ml_pack_and_meta_rule ~context ~all_libs (buildable : Rocq_stanza.Buildable.t) +let ml_pack_and_meta_rule ~lib_root ~all_libs (buildable : Rocq_stanza.Buildable.t) : unit Action_builder.t = (* rocqdep expects an mlpack file next to the sources otherwise it will omit the cmxs deps *) let plugin_loc = List.hd_opt buildable.plugins |> Option.map ~f:fst in - let meta_info = Util.meta_info ~loc:plugin_loc ~context in + let meta_info = Util.meta_info ~loc:plugin_loc ~lib_root in Action_builder.paths (List.filter_map ~f:meta_info all_libs) ;; @@ -443,14 +445,22 @@ let ml_flags_and_ml_pack_rule in let findlib_plugin_flags = Util.include_flags all_libs in let ml_flags = Command.Args.S [ findlib_plugin_flags ] in - ml_flags, ml_pack_and_meta_rule ~context ~all_libs buildable + let plugin_packages = + List.filter_map all_libs ~f:(fun lib -> Lib.info lib |> Lib_info.package) + |> List.sort_uniq ~compare:Package.Name.compare + in + let lib_root = Install_layout.lib_root context plugin_packages in + ml_flags, ml_pack_and_meta_rule ~lib_root ~all_libs buildable, lib_root in let mlpack_rule = let open Action_builder.O in - let* _, mlpack_rule = Resolve.Memo.read res in + let* _, mlpack_rule, _ = Resolve.Memo.read res in mlpack_rule in - Resolve.Memo.map ~f:fst res, mlpack_rule + let plugin_ocamlpath = Resolve.Memo.map res ~f:(fun (_, _, lib_root) -> lib_root) in + ( Resolve.Memo.map ~f:(fun (ml_flags, _, _) -> ml_flags) res + , mlpack_rule + , plugin_ocamlpath ) ;; let dep_theory_file ~dir ~wrapper_name = @@ -582,6 +592,22 @@ let setup_rocqproject_for_theory_rule (Action_builder.write_file_dyn rocqproject contents.build) ;; +(* Cons the plugin layout's lib root onto whatever OCAMLPATH [action.env] + already carries (e.g. layout entries from [(deps (package ...))]) rather + than overwriting it. *) +let add_plugin_ocamlpath ~lib_root (action : Action.Full.t) = + let { Action.Full.env = action_env; _ } = action in + let env = + Env.update action_env ~var:Findlib_config.ocamlpath_var ~f:(fun _PATH -> + Some + (Bin.cons_path + ~_PATH + (Path.build lib_root) + ~path_sep:Findlib_config.ocamlpath_sep)) + in + Action.Full.add_env env action +;; + let setup_rocqdep_for_theory_rule ~sctx ~dir @@ -593,6 +619,7 @@ let setup_rocqdep_for_theory_rule ~mlpack_rule ~boot_flags ~stanza_rocqdep_flags + ~plugin_ocamlpath rocq_modules = (* rocqdep needs the full source + plugin's mlpack to be present :( *) @@ -625,7 +652,12 @@ let setup_rocqdep_for_theory_rule (let open Action_builder.With_targets.O in Action_builder.with_no_targets mlpack_rule >>> Action_builder.(with_no_targets (goal source_rule)) - >>> Command.run ~dir:(Path.build dir) ~stdout_to rocq rocqdep_flags) + >>> Command.run ~dir:(Path.build dir) ~stdout_to rocq rocqdep_flags + |> Action_builder.With_targets.map_build ~f:(fun build -> + let open Action_builder.O in + let+ action = build + and+ lib_root = Resolve.Memo.read plugin_ocamlpath in + add_plugin_ocamlpath ~lib_root action)) ;; module Dep_map = Stdune.Map.Make (Path) @@ -843,6 +875,7 @@ let setup_rocqc_rule ~ml_flags ~theory_dirs ~rocq_sources + ~plugin_ocamlpath rocq_module = (* Process rocqdep and generate rules *) @@ -900,7 +933,12 @@ let setup_rocqc_rule ?stdout_to:output_file ~sandbox:Sandbox_config.no_sandboxing rocq - (cflag :: targets :: args)) + (cflag :: targets :: args) + |> Action_builder.With_targets.map_build ~f:(fun build -> + let open Action_builder.O in + let+ action = build + and+ lib_root = Resolve.Memo.read plugin_ocamlpath in + add_plugin_ocamlpath ~lib_root action)) ;; let rocq_modules_of_theory ~sctx lib = @@ -1054,11 +1092,11 @@ let theory_context ~context ~scope ~name buildable = Resolve.Memo.lift @@ Rocq_lib.theories_closure theory) in (* ML-level flags for depending libraries *) - let ml_flags, mlpack_rule = + let ml_flags, mlpack_rule, plugin_ocamlpath = let lib_db = Scope.libs scope in ml_flags_and_ml_pack_rule ~context ~theories_deps ~lib_db buildable in - theory, theories_deps, ml_flags, mlpack_rule + theory, theories_deps, ml_flags, mlpack_rule, plugin_ocamlpath ;; (* Common context for extraction, almost the same than above *) @@ -1083,18 +1121,18 @@ let extraction_context ~context ~scope (buildable : Rocq_stanza.Buildable.t) = | None -> theories_deps | Some (_, boot) -> boot :: theories_deps in - let ml_flags, mlpack_rule = + let ml_flags, mlpack_rule, plugin_ocamlpath = let lib_db = Scope.libs scope in ml_flags_and_ml_pack_rule ~context ~theories_deps ~lib_db buildable in - theories_deps, ml_flags, mlpack_rule + theories_deps, ml_flags, mlpack_rule, plugin_ocamlpath ;; let setup_theory_rules ~sctx ~dir ~dir_contents (s : Rocq_stanza.Theory.t) = let* scope = Scope.DB.find_by_dir dir in let name = s.name in let rocq_lang_version = s.buildable.rocq_lang_version in - let theory, theories_deps, ml_flags, mlpack_rule = + let theory, theories_deps, ml_flags, mlpack_rule, plugin_ocamlpath = let context = Super_context.context sctx |> Context.name in theory_context ~context ~scope ~name s.buildable in @@ -1158,6 +1196,7 @@ let setup_theory_rules ~sctx ~dir ~dir_contents (s : Rocq_stanza.Theory.t) = ~mlpack_rule ~boot_flags ~stanza_rocqdep_flags:s.rocqdep_flags + ~plugin_ocamlpath rocq_modules >>> Memo.parallel_iter rocq_modules ~f:(fun rocq_module -> setup_rocqc_rule @@ -1177,6 +1216,7 @@ let setup_theory_rules ~sctx ~dir ~dir_contents (s : Rocq_stanza.Theory.t) = ~ml_flags ~theory_dirs ~rocq_sources:rocq_dir_contents + ~plugin_ocamlpath rocq_module >>> setup_output_diff_rule ~loc @@ -1192,7 +1232,7 @@ let setup_theory_rules ~sctx ~dir ~dir_contents (s : Rocq_stanza.Theory.t) = let rocqtop_args_theory ~sctx ~dir ~dir_contents (s : Rocq_stanza.Theory.t) rocq_module = let* scope = Scope.DB.find_by_dir dir in let name = s.name in - let _theory, theories_deps, ml_flags, _mlpack_rule = + let _theory, theories_deps, ml_flags, _mlpack_rule, _plugin_ocamlpath = let context = Super_context.context sctx |> Context.name in theory_context ~context ~scope ~name s.buildable in @@ -1311,7 +1351,7 @@ let setup_extraction_rules ~sctx ~dir ~dir_contents (s : Rocq_stanza.Extraction. let use_corelib = s.buildable.use_corelib in let rocq_lang_version = s.buildable.rocq_lang_version in let* scope = Scope.DB.find_by_dir dir in - let theories_deps, ml_flags, mlpack_rule = + let theories_deps, ml_flags, mlpack_rule, plugin_ocamlpath = let context = Super_context.context sctx |> Context.name in extraction_context ~context ~scope s.buildable in @@ -1343,6 +1383,7 @@ let setup_extraction_rules ~sctx ~dir ~dir_contents (s : Rocq_stanza.Extraction. ~mlpack_rule ~boot_flags ~stanza_rocqdep_flags:Ordered_set_lang.Unexpanded.standard + ~plugin_ocamlpath [ rocq_module ] >>> setup_rocqc_rule ~scope @@ -1353,6 +1394,7 @@ let setup_extraction_rules ~sctx ~dir ~dir_contents (s : Rocq_stanza.Extraction. ~sctx ~loc ~rocqc_dir:dir + ~plugin_ocamlpath rocq_module ~dir ~theories_deps @@ -1368,7 +1410,7 @@ let setup_extraction_rules ~sctx ~dir ~dir_contents (s : Rocq_stanza.Extraction. let rocqtop_args_extraction ~sctx ~dir (s : Rocq_stanza.Extraction.t) rocq_module = let use_corelib = s.buildable.use_corelib in let* scope = Scope.DB.find_by_dir dir in - let theories_deps, ml_flags, _mlpack_rule = + let theories_deps, ml_flags, _mlpack_rule, _plugin_ocamlpath = let context = Super_context.context sctx |> Context.name in extraction_context ~context ~scope s.buildable in diff --git a/src/dune_rules/super_context.ml b/src/dune_rules/super_context.ml index 097b5322c7b..1c2b7bbf9a5 100644 --- a/src/dune_rules/super_context.ml +++ b/src/dune_rules/super_context.ml @@ -150,10 +150,10 @@ let extend_action t ~dir action = (let open Memo.O in t.get_node dir >>= Env_node.external_env) in - (* [action.env] is expected to contain only PATH-prepend hints from - [Dep_conf_eval.make_bin_env]. Splice its PATH entries in front of the - external env's PATH, keep other vars as-is. *) - let env = Env_path.extend_env_concat_path env action.env in + (* Cons path-like vars from action.env (bin-layout PATH, package-layout + OCAMLPATH/etc.) onto the directory env so that both layout and system + entries remain visible. Other vars from action.env overwrite dir env. *) + let env = Install.Roots.extend_env_concat_path env action.env in Action.Full.add_env env action |> Action.Full.map ~f:(function | Chdir _ as a -> a @@ -237,32 +237,12 @@ let make_default_env_node ;; let make_root_env (context : Context.t) ~(host : t option) : Env.t Memo.t = - let* env = - let roots = - let context = Context.name context in - Install.Context.dir ~context |> Install.Roots.make ~relative:Path.Build.relative - in - Context.installed_env context >>| Install.Roots.add_to_env roots - in - let+ host_context, _PATH = - let _PATH = Env.get env Env_path.var in + let+ env = match host with - | None -> Memo.return (context, _PATH) - | Some host -> - let context = host.context in - let+ _PATH = - let+ env = Context.installed_env context in - Env.get env Env_path.var - in - context, _PATH + | None -> Context.installed_env context + | Some host -> Context.installed_env host.context in - Env.add - env - ~var:Env_path.var - ~value: - (Install.Context.bin_dir ~context:(Context.name host_context) - |> Path.build - |> Bin.cons_path ~_PATH) + env ;; let create ~(context : Context.t) ~(host : t option) ~packages ~stanzas = diff --git a/src/install/context.ml b/src/install/context.ml index 72c79407db3..c1e47d02c36 100644 --- a/src/install/context.ml +++ b/src/install/context.ml @@ -22,7 +22,10 @@ let of_path path = | Build (Regular (With_context (name, src))) -> Some (if Context_name.equal name install_context.name - then Context_name.of_string (Path.Source.basename src |> Filename.to_string) + then ( + match Path.Source.split_first_component src with + | Some (ctx, _) -> Context_name.of_string (Filename.to_string ctx) + | None -> name) else name) | Build (Anonymous_action (With_context (name, _))) -> Some name | _ -> None diff --git a/src/install/roots.ml b/src/install/roots.ml index e60124f1928..7e6ff60432d 100644 --- a/src/install/roots.ml +++ b/src/install/roots.ml @@ -109,10 +109,36 @@ let sep var = else None ;; +let path_vars = + Env.Var.Set.of_list + (Env_path.var + :: List.map ~f:fst (to_env_without_path (make_all ()) ~relative:(fun () _ -> ()))) +;; + let add_to_env t env = - to_env_without_path t ~relative:Path.Build.relative - |> List.fold_left ~init:env ~f:(fun env (var, path) -> - Env.update env ~var ~f:(fun _PATH -> - let path_sep = sep var in - Some (Bin.cons_path ?path_sep (Path.build path) ~_PATH))) + let env = + to_env_without_path t ~relative:Path.Build.relative + |> List.fold_left ~init:env ~f:(fun env (var, path) -> + Env.update env ~var ~f:(fun _PATH -> + let path_sep = sep var in + Some (Bin.cons_path ?path_sep (Path.build path) ~_PATH))) + in + Env.update env ~var:Env_path.var ~f:(fun _PATH -> + Some (Bin.cons_path (Path.build t.bin) ~_PATH)) +;; + +let extend_env_concat_path a b = + let a = + Env.Var.Set.fold path_vars ~init:a ~f:(fun var env -> + match Env.get b var with + | None -> env + | Some value -> + let path_sep = sep var in + let dirs = Bin.parse_path ?sep:path_sep value in + Env.update env ~var ~f:(fun init -> + List.fold_right dirs ~init ~f:(fun dir acc -> + Some (Bin.cons_path ?path_sep dir ~_PATH:acc)))) + in + let b = Env.Var.Set.fold path_vars ~init:b ~f:(fun var env -> Env.remove env ~var) in + Env.extend_env a b ;; diff --git a/src/install/roots.mli b/src/install/roots.mli index 433c7ad52e5..84164b003f2 100644 --- a/src/install/roots.mli +++ b/src/install/roots.mli @@ -27,5 +27,22 @@ val first_has_priority : 'a option t -> 'a option t -> 'a option t val to_env_without_path : 'a t -> relative:('a -> string -> 'a) -> (Env.Var.t * 'a) list val add_to_env : Path.Build.t t -> Env.t -> Env.t + +(** Returns the path separator for a given env var, if it's a known + path-like variable (e.g. OCAMLPATH uses [;] on all platforms). *) +val sep : Env.Var.t -> char option + +(** The set of all path-like env vars managed by the install roots: + PATH, OCAMLPATH, CAML_LD_LIBRARY_PATH, OCAMLFIND_IGNORE_DUPS_IN, + OCAMLTOP_INCLUDE_PATH, and MANPATH. *) +val path_vars : Env.Var.Set.t + +(** [extend_env_concat_path a b] extends [a] with the bindings from [b], with + one twist: for every var in {!path_vars}, the value from [b] is prepended + (cons-style, respecting per-var separator from {!sep}) to the value from + [a] rather than overwriting it. Generalisation of + [Env_path.extend_env_concat_path] to every install-managed path var. *) +val extend_env_concat_path : Env.t -> Env.t -> Env.t + val make : 'a -> relative:('a -> string -> 'a) -> 'a t val make_all : 'a -> 'a t diff --git a/test/blackbox-tests/test-cases/bin-pform/env-binaries.t b/test/blackbox-tests/test-cases/bin-pform/env-binaries.t index ccd77dce736..b2e09869080 100644 --- a/test/blackbox-tests/test-cases/bin-pform/env-binaries.t +++ b/test/blackbox-tests/test-cases/bin-pform/env-binaries.t @@ -32,9 +32,7 @@ The rule depends on the .bin/ symlink: > | jq 'include "dune"; .[] | ruleDepFilePaths' "_build/default/.bin/myothername" -The action's PATH gets the .bin/ symlink directory and the workspace -install bin dir: +The action's PATH gets the .bin/ symlink directory: $ env_added "$(cat _build/default/path-output)" "$PATH" $TESTCASE_ROOT/_build/default/.bin - $TESTCASE_ROOT/_build/install/default/bin diff --git a/test/blackbox-tests/test-cases/bin-pform/include-deps.t b/test/blackbox-tests/test-cases/bin-pform/include-deps.t index 9ec55321bda..9649410b33d 100644 --- a/test/blackbox-tests/test-cases/bin-pform/include-deps.t +++ b/test/blackbox-tests/test-cases/bin-pform/include-deps.t @@ -27,12 +27,10 @@ tracked dep of the rule. $ dune build path-output -The action's PATH includes the .binaries dir and the install bin -dir: +The action's PATH includes the .binaries dir: $ env_added "$(cat _build/default/path-output)" "$PATH" | censor $PWD/_build/install/default/.binaries/$DIGEST - $PWD/_build/install/default/bin The rule depends on the build artifact and the .binaries symlink: diff --git a/test/blackbox-tests/test-cases/bin-pform/inline-tests-include.t b/test/blackbox-tests/test-cases/bin-pform/inline-tests-include.t index 2da02809432..6ba4365512c 100644 --- a/test/blackbox-tests/test-cases/bin-pform/inline-tests-include.t +++ b/test/blackbox-tests/test-cases/bin-pform/inline-tests-include.t @@ -47,4 +47,3 @@ collection to drain the env contribution from Include_result. $ env_added "$(cat path.out)" "$PATH" | censor $PWD/_build/install/default/.binaries/$DIGEST - $PWD/_build/install/default/bin diff --git a/test/blackbox-tests/test-cases/bin-pform/inline-tests.t b/test/blackbox-tests/test-cases/bin-pform/inline-tests.t index fc72716abd9..a96e2ecc8cc 100644 --- a/test/blackbox-tests/test-cases/bin-pform/inline-tests.t +++ b/test/blackbox-tests/test-cases/bin-pform/inline-tests.t @@ -1,4 +1,4 @@ -%{bin:...} in (inline_tests (deps ...)) adds the install bin dir to +%{bin:...} in (inline_tests (deps ...)) adds a bin-layout dir to the test runner's PATH. $ cat >dune-project <&1 $ env_added "$(cat path.out)" "$PATH" | censor $PWD/_build/install/default/.binaries/$DIGEST - $PWD/_build/install/default/bin diff --git a/test/blackbox-tests/test-cases/bin-pform/multiple-bins.t b/test/blackbox-tests/test-cases/bin-pform/multiple-bins.t index 8af0e7f1ca8..eae939011d3 100644 --- a/test/blackbox-tests/test-cases/bin-pform/multiple-bins.t +++ b/test/blackbox-tests/test-cases/bin-pform/multiple-bins.t @@ -24,13 +24,11 @@ Multiple %{bin:...} deps in a single rule. > (bash "echo $PATH")))) > EOF -PATH gets a .binaries dir (with symlinks to both bins) plus the -install bin dir: +PATH gets a .binaries dir with symlinks to both bins: $ dune build path-output $ env_added "$(cat _build/default/path-output)" "$PATH" | censor $PWD/_build/install/default/.binaries/$DIGEST - $PWD/_build/install/default/bin The rule depends on each binary via two paths: the build artifact (from the pform expansion) and the .binaries symlink (from the PATH diff --git a/test/blackbox-tests/test-cases/bin-pform/run-action.t b/test/blackbox-tests/test-cases/bin-pform/run-action.t index ef5fc0b8540..df98f04171e 100644 --- a/test/blackbox-tests/test-cases/bin-pform/run-action.t +++ b/test/blackbox-tests/test-cases/bin-pform/run-action.t @@ -1,6 +1,5 @@ %{bin:NAME} as a (run ...) target invokes the binary; as a deps -entry it adds a .binaries dir to the action's PATH (plus the -always-on install bin dir). +entry it adds a .binaries dir to the action's PATH. $ cat >dune-project < (lang dune 3.24) @@ -24,7 +23,6 @@ always-on install bin dir). hello from mybin $ env_added "$(cat _build/default/path-output)" "$PATH" | censor $PWD/_build/install/default/.binaries/$DIGEST - $PWD/_build/install/default/bin The rule's deps include the build artifact and the .binaries symlink: diff --git a/test/blackbox-tests/test-cases/bin-pform/shared-layout.t b/test/blackbox-tests/test-cases/bin-pform/shared-layout.t index 3fbc64dfd44..5d9da0eb341 100644 --- a/test/blackbox-tests/test-cases/bin-pform/shared-layout.t +++ b/test/blackbox-tests/test-cases/bin-pform/shared-layout.t @@ -34,7 +34,5 @@ $DIGEST2): $ env_added "$(cat _build/default/out1)" "$PATH" | censor $PWD/_build/install/default/.binaries/$DIGEST - $PWD/_build/install/default/bin $ env_added "$(cat _build/default/out2)" "$PATH" | censor $PWD/_build/install/default/.binaries/$DIGEST - $PWD/_build/install/default/bin diff --git a/test/blackbox-tests/test-cases/bin-pform/test-stanza.t b/test/blackbox-tests/test-cases/bin-pform/test-stanza.t index 0fa16e59b35..38dba4ebb42 100644 --- a/test/blackbox-tests/test-cases/bin-pform/test-stanza.t +++ b/test/blackbox-tests/test-cases/bin-pform/test-stanza.t @@ -32,4 +32,3 @@ mytest.exe.output, which we then read. [1] $ env_added "$(cat _build/default/mytest.exe.output)" "$PATH" | censor $PWD/_build/install/default/.binaries/$DIGEST - $PWD/_build/install/default/bin diff --git a/test/blackbox-tests/test-cases/custom-cross-compilation/bin-pform-context-expansion.t b/test/blackbox-tests/test-cases/custom-cross-compilation/bin-pform-context-expansion.t index 3502a81f950..afbc8e5a169 100644 --- a/test/blackbox-tests/test-cases/custom-cross-compilation/bin-pform-context-expansion.t +++ b/test/blackbox-tests/test-cases/custom-cross-compilation/bin-pform-context-expansion.t @@ -31,4 +31,3 @@ dir and the install bin dir, both under the host context. $ env_added "$(cat _build/target/path-output)" "$PATH" | censor $PWD/_build/install/host/.binaries/$DIGEST - $PWD/_build/install/host/bin diff --git a/test/blackbox-tests/test-cases/custom-cross-compilation/package-deps-context-expansion.t b/test/blackbox-tests/test-cases/custom-cross-compilation/package-deps-context-expansion.t new file mode 100644 index 00000000000..372f61b787c --- /dev/null +++ b/test/blackbox-tests/test-cases/custom-cross-compilation/package-deps-context-expansion.t @@ -0,0 +1,39 @@ +(deps (package NAME)) in a cross-compilation context. The install +layout dir should sit under the host context's install tree. + + $ cat >dune-project < (lang dune 3.24) + > (package (name mypkg)) + > EOF + + $ cat >dune-workspace < (lang dune 3.24) + > (context (default (name host))) + > (context (default (name target) (host host))) + > EOF + + $ mkdir src + $ cat >src/dune < (library (public_name mypkg)) + > EOF + $ cat >src/mypkg.ml < let x = 1 + > EOF + + $ cat >dune <<'EOF' + > (rule + > (enabled_if (= %{context_name} target)) + > (deps (package mypkg)) + > (action + > (with-stdout-to out (echo "ok")))) + > EOF + + $ dune build _build/target/out + +The layout dir should be under install/host/ (the layout for a +target-context action uses host artifacts): + + $ dune rules --format=json _build/target/out \ + > | jq 'include "dune"; .[] | ruleDepFilePaths' \ + > | grep dune-package | censor + "_build/install/host/.packages/$DIGEST/lib/mypkg/dune-package" diff --git a/test/blackbox-tests/test-cases/dune b/test/blackbox-tests/test-cases/dune index 0f7dc83e277..9aa38d17402 100644 --- a/test/blackbox-tests/test-cases/dune +++ b/test/blackbox-tests/test-cases/dune @@ -42,6 +42,11 @@ (applies_to windows-diff staged-pps-with-sandboxing-gh6644) (alias runtest-windows)) +(cram + (applies_to empty-meta-version-gh9176) + (deps + (package dune-build-info))) + ;; DISABLED TESTS (cram diff --git a/test/blackbox-tests/test-cases/dune-site/dune b/test/blackbox-tests/test-cases/dune-site/dune new file mode 100644 index 00000000000..cae66120915 --- /dev/null +++ b/test/blackbox-tests/test-cases/dune-site/dune @@ -0,0 +1,5 @@ +(cram + (applies_to :whole_subtree) + (deps + (package dune-site) + (package dune-private-libs))) diff --git a/test/blackbox-tests/test-cases/dune-site/sites-plugin.t b/test/blackbox-tests/test-cases/dune-site/sites-plugin.t index c3d17040187..e74486cdc35 100644 --- a/test/blackbox-tests/test-cases/dune-site/sites-plugin.t +++ b/test/blackbox-tests/test-cases/dune-site/sites-plugin.t @@ -70,7 +70,7 @@ Test sites plugins (example from the manual) > Queue.add (fun () -> print_endline "Plugin1 is doing something...") Registration.todo > EOF - $ dune build @all 2>&1 | dune_cmd sanitize + $ dune build @install 2>&1 | dune_cmd sanitize $ dune exec ./app.exe Registration of Plugin1 Main app starts... diff --git a/test/blackbox-tests/test-cases/inline-tests/dune b/test/blackbox-tests/test-cases/inline-tests/dune index db7ef3f5eda..ced185c067d 100644 --- a/test/blackbox-tests/test-cases/inline-tests/dune +++ b/test/blackbox-tests/test-cases/inline-tests/dune @@ -13,4 +13,10 @@ (cram (applies_to clicolor-force-inline-tests-github6607) (deps - (package stdune))) + (package stdune) + (package dyn) + (package ordering) + (package pp) + (package top-closure) + (package csexp) + (package fs-io))) diff --git a/test/blackbox-tests/test-cases/package-dep.t b/test/blackbox-tests/test-cases/package-dep.t index 92c19a1a1ee..4f119154095 100644 --- a/test/blackbox-tests/test-cases/package-dep.t +++ b/test/blackbox-tests/test-cases/package-dep.t @@ -33,7 +33,7 @@ Tests package-scoped library dependencies. > (echo "let () = Printf.printf \"%d %s\" Foo.x Bar.x"))) > > (rule - > (deps test.ml (package bar)) + > (deps test.ml (package foo) (package bar)) > (targets test.exe) > (action (run ocamlfind ocamlc -linkpkg -package bar -o test.exe test.ml))) > diff --git a/test/blackbox-tests/test-cases/package-materialization/cram-integration.t b/test/blackbox-tests/test-cases/package-materialization/cram-integration.t new file mode 100644 index 00000000000..e11fb8b42d8 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/cram-integration.t @@ -0,0 +1,69 @@ +Test that (deps (package ...)) in a cram stanza sets up all layout env vars. + + $ cat >dune-project < (lang dune 3.24) + > (package (name mypkg)) + > EOF + $ mkdir src + $ cat >src/dune < (library + > (public_name mypkg) + > (foreign_stubs (language c) (names stub))) + > EOF + $ cat >src/mypkg.ml <<'EOF' + > let x = 1 + > EOF + $ cat >src/stub.c <<'EOF' + > #include + > CAMLprim value mypkg_stub(value unit) { return Val_unit; } + > EOF + +Capture baselines into a setup script for the inner test: + + $ mkdir tests + $ cat >tests/setup.sh < env_added() { + > comm -23 <(tr ':' '\n' <<< "\$1" | sort -u) <(tr ':' '\n' <<< "\$2" | sort -u) + > } + > BASELINE_PATH='$PATH' + > BASELINE_OCAMLPATH='$OCAMLPATH' + > BASELINE_CAML_LD_LIBRARY_PATH='$CAML_LD_LIBRARY_PATH' + > BASELINE_OCAMLFIND_IGNORE_DUPS_IN='$OCAMLFIND_IGNORE_DUPS_IN' + > BASELINE_OCAMLTOP_INCLUDE_PATH='$OCAMLTOP_INCLUDE_PATH' + > BASELINE_MANPATH='$MANPATH' + > EOF + $ cat >tests/dune < (cram + > (shell bash) + > (deps (package mypkg)) + > (setup_scripts setup.sh)) + > EOF + $ cat >tests/check-env.t <<'EOF' + > $ env_added "$PATH" "$BASELINE_PATH" + > $ env_added "$OCAMLPATH" "$BASELINE_OCAMLPATH" + > $ env_added "$CAML_LD_LIBRARY_PATH" "$BASELINE_CAML_LD_LIBRARY_PATH" + > $ env_added "$OCAMLFIND_IGNORE_DUPS_IN" "$BASELINE_OCAMLFIND_IGNORE_DUPS_IN" + > $ env_added "$OCAMLTOP_INCLUDE_PATH" "$BASELINE_OCAMLTOP_INCLUDE_PATH" + > $ env_added "$MANPATH" "$BASELINE_MANPATH" + > EOF + +The inner test fails showing exactly what the package dep added to each var: + + $ dune runtest tests 2>&1 + File "tests/check-env.t", line 1, characters 0-0: + --- tests/check-env.t + +++ tests/check-env.t.corrected + @@ -1,6 +1,12 @@ + $ env_added "$PATH" "$BASELINE_PATH" + + $TESTCASE_ROOT/_build/install/default/.packages/1dcd8c6ece3938b1f1fa14c3483ca989/bin + $ env_added "$OCAMLPATH" "$BASELINE_OCAMLPATH" + + $TESTCASE_ROOT/_build/install/default/.packages/1dcd8c6ece3938b1f1fa14c3483ca989/lib + $ env_added "$CAML_LD_LIBRARY_PATH" "$BASELINE_CAML_LD_LIBRARY_PATH" + + $TESTCASE_ROOT/_build/install/default/.packages/1dcd8c6ece3938b1f1fa14c3483ca989/lib/stublibs + $ env_added "$OCAMLFIND_IGNORE_DUPS_IN" "$BASELINE_OCAMLFIND_IGNORE_DUPS_IN" + + $TESTCASE_ROOT/_build/install/default/.packages/1dcd8c6ece3938b1f1fa14c3483ca989/lib + $ env_added "$OCAMLTOP_INCLUDE_PATH" "$BASELINE_OCAMLTOP_INCLUDE_PATH" + + $TESTCASE_ROOT/_build/install/default/.packages/1dcd8c6ece3938b1f1fa14c3483ca989/lib/toplevel + $ env_added "$MANPATH" "$BASELINE_MANPATH" + + $TESTCASE_ROOT/_build/install/default/.packages/1dcd8c6ece3938b1f1fa14c3483ca989/man + [1] diff --git a/test/blackbox-tests/test-cases/package-materialization/cycle.t b/test/blackbox-tests/test-cases/package-materialization/cycle.t new file mode 100644 index 00000000000..71cdfd7e430 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/cycle.t @@ -0,0 +1,65 @@ +Test what happens with circular package dependencies. + + $ cat >dune-project < (lang dune 3.24) + > (package (name a) (depends b)) + > (package (name b) (depends a)) + > EOF + $ mkdir a-src b-src + $ cat >a-src/dune < (library (public_name a)) + > EOF + $ cat >a-src/a.ml < let x = 1 + > EOF + $ cat >b-src/dune < (library (public_name b)) + > EOF + $ cat >b-src/b.ml < let y = 2 + > EOF + +Non-strict (default) uses Top_closure which detects the cycle: + + $ cat >dune <<'EOF' + > (rule + > (deps (package a)) + > (action (with-stdout-to out (echo "ok")))) + > EOF + $ dune build out 2>&1 + +What about depending on both? + + $ cat >dune <<'EOF' + > (rule + > (deps (package a) (package b)) + > (action (with-stdout-to out2 (echo "ok")))) + > EOF + $ dune build out2 2>&1 + +BUG: strict mode silently accepts circular deps because it skips the +closure. The cycle should probably be reported regardless. + + $ cat >dune-project < (lang dune 3.24) + > (strict_package_deps true) + > (package (name a) (depends b)) + > (package (name b) (depends a)) + > EOF + $ rm -rf _build + $ cat >dune <<'EOF' + > (rule + > (deps (package a)) + > (action (with-stdout-to out (echo "ok")))) + > EOF + $ dune build out 2>&1 + +BUG: strict with both also silently accepts the cycle: + + $ cat >dune <<'EOF' + > (rule + > (deps (package a) (package b)) + > (action (with-stdout-to out2 (echo "ok")))) + > EOF + $ rm -rf _build + $ dune build out2 2>&1 diff --git a/test/blackbox-tests/test-cases/package-materialization/dune b/test/blackbox-tests/test-cases/package-materialization/dune new file mode 100644 index 00000000000..d81dadc2778 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/dune @@ -0,0 +1,7 @@ +(cram + (applies_to :whole_subtree) + (setup_scripts helpers.sh)) + +(cram + (applies_to ocamlfind no-transitive-through-targets installed-package) + (deps %{bin:ocamlfind})) diff --git a/test/blackbox-tests/test-cases/package-materialization/env/caml-ld-library-path.t b/test/blackbox-tests/test-cases/package-materialization/env/caml-ld-library-path.t new file mode 100644 index 00000000000..73425573e30 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/env/caml-ld-library-path.t @@ -0,0 +1,29 @@ +Test that (deps (package ...)) adds lib/stublibs/ to CAML_LD_LIBRARY_PATH. + + $ cat >dune-project < (lang dune 3.24) + > (package (name mypkg)) + > EOF + $ mkdir src + $ cat >src/dune < (library + > (public_name mypkg) + > (foreign_stubs (language c) (names stub))) + > EOF + $ cat >src/mypkg.ml <<'EOF' + > let x = 1 + > EOF + $ cat >src/stub.c <<'EOF' + > #include + > CAMLprim value mypkg_stub(value unit) { return Val_unit; } + > EOF + $ cat >dune <<'EOF' + > (rule + > (deps (package mypkg)) + > (action (with-stdout-to out (bash "echo $CAML_LD_LIBRARY_PATH")))) + > EOF + + $ baseline=$CAML_LD_LIBRARY_PATH + $ dune build out + $ env_added "$(cat _build/default/out)" "$baseline" | censor + $PWD/_build/install/default/.packages/$DIGEST/lib/stublibs diff --git a/test/blackbox-tests/test-cases/package-materialization/env/ocamlfind-ignore-dups-in.t b/test/blackbox-tests/test-cases/package-materialization/env/ocamlfind-ignore-dups-in.t new file mode 100644 index 00000000000..a9f495eeea0 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/env/ocamlfind-ignore-dups-in.t @@ -0,0 +1,23 @@ +Test that (deps (package ...)) adds lib/ to OCAMLFIND_IGNORE_DUPS_IN. + + $ cat >dune-project < (lang dune 3.24) + > (package (name mypkg)) + > EOF + $ mkdir src + $ cat >src/dune < (library (public_name mypkg)) + > EOF + $ cat >src/mypkg.ml <<'EOF' + > let x = 1 + > EOF + $ cat >dune <<'EOF' + > (rule + > (deps (package mypkg)) + > (action (with-stdout-to out (bash "echo $OCAMLFIND_IGNORE_DUPS_IN")))) + > EOF + + $ baseline=$OCAMLFIND_IGNORE_DUPS_IN + $ dune build out + $ env_added "$(cat _build/default/out)" "$baseline" | censor + $PWD/_build/install/default/.packages/$DIGEST/lib diff --git a/test/blackbox-tests/test-cases/package-materialization/env/ocamlpath.t b/test/blackbox-tests/test-cases/package-materialization/env/ocamlpath.t new file mode 100644 index 00000000000..160e8a87908 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/env/ocamlpath.t @@ -0,0 +1,23 @@ +Test that (deps (package ...)) adds the package's lib/ to OCAMLPATH. + + $ cat >dune-project < (lang dune 3.24) + > (package (name mypkg)) + > EOF + $ mkdir src + $ cat >src/dune < (library (public_name mypkg)) + > EOF + $ cat >src/mypkg.ml <<'EOF' + > let x = 1 + > EOF + $ cat >dune <<'EOF' + > (rule + > (deps (package mypkg)) + > (action (with-stdout-to out (bash "echo $OCAMLPATH")))) + > EOF + + $ baseline=$OCAMLPATH + $ dune build out + $ env_added "$(cat _build/default/out)" "$baseline" | censor + $PWD/_build/install/default/.packages/$DIGEST/lib diff --git a/test/blackbox-tests/test-cases/package-materialization/env/ocamltop-include-path.t b/test/blackbox-tests/test-cases/package-materialization/env/ocamltop-include-path.t new file mode 100644 index 00000000000..d6e10f52920 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/env/ocamltop-include-path.t @@ -0,0 +1,23 @@ +Test that (deps (package ...)) adds lib/toplevel/ to OCAMLTOP_INCLUDE_PATH. + + $ cat >dune-project < (lang dune 3.24) + > (package (name mypkg)) + > EOF + $ mkdir src + $ cat >src/dune < (library (public_name mypkg)) + > EOF + $ cat >src/mypkg.ml <<'EOF' + > let x = 1 + > EOF + $ cat >dune <<'EOF' + > (rule + > (deps (package mypkg)) + > (action (with-stdout-to out (bash "echo $OCAMLTOP_INCLUDE_PATH")))) + > EOF + + $ baseline=$OCAMLTOP_INCLUDE_PATH + $ dune build out + $ env_added "$(cat _build/default/out)" "$baseline" | censor + $PWD/_build/install/default/.packages/$DIGEST/lib/toplevel diff --git a/test/blackbox-tests/test-cases/package-materialization/env/path.t b/test/blackbox-tests/test-cases/package-materialization/env/path.t new file mode 100644 index 00000000000..38c2c233ed3 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/env/path.t @@ -0,0 +1,27 @@ +Test that (deps (package ...)) adds the package's bin/ to PATH. + + $ cat >dune-project < (lang dune 3.24) + > (package (name mypkg)) + > EOF + $ mkdir src + $ cat >src/dune < (executable (name mytool) (public_name mytool) (package mypkg) (modules mytool)) + > (library (public_name mypkg) (modules mypkg)) + > EOF + $ cat >src/mytool.ml <<'EOF' + > let () = print_endline "hello" + > EOF + $ cat >src/mypkg.ml <<'EOF' + > let x = 1 + > EOF + $ cat >dune <<'EOF' + > (rule + > (deps (package mypkg)) + > (action (with-stdout-to out (bash "echo $PATH")))) + > EOF + + $ baseline=$PATH + $ dune build out + $ env_added "$(cat _build/default/out)" "$baseline" | censor + $PWD/_build/install/default/.packages/$DIGEST/bin diff --git a/test/blackbox-tests/test-cases/package-materialization/helpers.sh b/test/blackbox-tests/test-cases/package-materialization/helpers.sh new file mode 100644 index 00000000000..61ab1891934 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/helpers.sh @@ -0,0 +1,4 @@ +env_added() { + comm -23 <(tr ':' '\n' <<< "$1" | sort -u) <(tr ':' '\n' <<< "$2" | sort -u) \ + | censor_install_layout +} diff --git a/test/blackbox-tests/test-cases/package-materialization/inline-tests.t b/test/blackbox-tests/test-cases/package-materialization/inline-tests.t new file mode 100644 index 00000000000..a2e50e636fe --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/inline-tests.t @@ -0,0 +1,44 @@ +Test that (inline_tests (deps (package ...))) sets up layout env vars +for the test runner. + + $ make_dune_project 3.24 + $ cat >>dune-project < (package (name mypkg)) + > EOF + $ mkdir src + $ cat >src/dune < (library (public_name mypkg)) + > EOF + $ cat >src/mypkg.ml <<'EOF' + > let x = 1 + > EOF + +Custom backend that writes OCAMLPATH to a file outside the sandbox: + + $ cat >dump_ocamlpath.ml < let () = + > let oc = open_out "$PWD/ocamlpath.out" in + > output_string oc (Sys.getenv "OCAMLPATH"); + > close_out oc + > EOF + + $ cat >dune <<'EOF' + > (library + > (name check_backend) + > (modules ()) + > (inline_tests.backend + > (generate_runner (cat dump_ocamlpath.ml)))) + > (library + > (name testlib) + > (inline_tests + > (backend check_backend) + > (deps (package mypkg)))) + > EOF + + $ cat >testlib.ml < EOF + + $ baseline=$OCAMLPATH + $ dune runtest 2>&1 + $ env_added "$(cat ocamlpath.out)" "$baseline" | censor + $PWD/_build/install/default/.packages/$DIGEST/lib diff --git a/test/blackbox-tests/test-cases/package-materialization/installed-package.t b/test/blackbox-tests/test-cases/package-materialization/installed-package.t new file mode 100644 index 00000000000..575329501e0 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/installed-package.t @@ -0,0 +1,43 @@ +Test that (deps (package ...)) works with externally installed packages. +Installed packages (found via findlib) go through the Installed codepath, +not the layout. The layout only applies to Local (workspace) packages. + +Install package "a" into a prefix: + + $ mkdir a consumer prefix + + $ cat >a/dune-project < (lang dune 3.24) + > (package (name a)) + > EOF + + $ cat >a/dune < (library (public_name a)) + > EOF + + $ cat >a/a.ml < let msg = "hello from lib a" + > EOF + + $ dune build --root a @install + $ dune install --root a --prefix $PWD/prefix 2>/dev/null + $ test -f prefix/lib/a/META + +Now create a consumer project that depends on the installed package. +The consumer uses (deps (package a)) and ocamlfind to verify the +package is findable: + + $ cat >consumer/dune-project < (lang dune 3.24) + > EOF + + $ cat >consumer/dune <<'EOF' + > (rule + > (deps (package a)) + > (action (with-stdout-to out + > (run ocamlfind query a)))) + > EOF + + $ OCAMLPATH=$PWD/prefix/lib/:$OCAMLPATH dune build --root consumer out + $ cat consumer/_build/default/out + $TESTCASE_ROOT/prefix/lib/a diff --git a/test/blackbox-tests/test-cases/package-materialization/mixed-deps.t b/test/blackbox-tests/test-cases/package-materialization/mixed-deps.t new file mode 100644 index 00000000000..31d0d4dfa1e --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/mixed-deps.t @@ -0,0 +1,30 @@ +Test that package deps and file deps work together in the same (deps ...). + + $ cat >dune-project < (lang dune 3.24) + > (package (name foo)) + > EOF + + $ mkdir src + + $ cat >src/dune < (library (public_name foo)) + > EOF + + $ cat >src/mylib.ml < let x = 1 + > EOF + + $ cat >dune <<'EOF' + > (rule + > (deps (package foo) (file src/mylib.ml)) + > (action (with-stdout-to out (echo "ok")))) + > EOF + + $ dune build out + +The layout deps include foo, and the file dep is also present: + + $ dune rules --format=json _build/default/out | jq 'include "dune"; .[] | ruleDepFilePaths' | censor | grep -E 'dune-package|src/mylib' | sort + "_build/default/src/mylib.ml" + "_build/install/default/.packages/$DIGEST/lib/foo/dune-package" diff --git a/test/blackbox-tests/test-cases/package-materialization/no-transitive-through-targets.t b/test/blackbox-tests/test-cases/package-materialization/no-transitive-through-targets.t new file mode 100644 index 00000000000..288a181d1de --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/no-transitive-through-targets.t @@ -0,0 +1,65 @@ +Package dependencies do NOT propagate transitively through build targets. +If target A depends on (package pkga) and target B depends on target A +and (package pkgb), then B's layout only contains pkgb — not pkga. +Declare what you need explicitly. + + $ cat >dune-project < (lang dune 3.24) + > (package (name pkga)) + > (package (name pkgb)) + > EOF + + $ mkdir a-src b-src + + $ cat >a-src/dune < (library (public_name pkga)) + > EOF + + $ cat >a-src/alib.ml < let x = 1 + > EOF + + $ cat >b-src/dune < (library (public_name pkgb)) + > EOF + + $ cat >b-src/blib.ml < let y = 2 + > EOF + +Target A depends on (package pkga): + + $ cat >dune <<'EOF' + > (rule + > (deps (package pkga)) + > (action (with-stdout-to target-a (echo "built with pkga")))) + > (rule + > (deps target-a (package pkgb)) + > (action (with-stdout-to target-b + > (run ocamlfind query pkga)))) + > EOF + +Target B depends on target-a AND (package pkgb). It tries to find +pkga via ocamlfind, but pkga is not in B's package closure: + + $ dune build target-b 2>&1 + File "dune", lines 4-7, characters 0-103: + 4 | (rule + 5 | (deps target-a (package pkgb)) + 6 | (action (with-stdout-to target-b + 7 | (run ocamlfind query pkga)))) + ocamlfind: Package `pkga' not found + [1] + +pkgb IS findable because it was explicitly declared: + + $ cat >dune <<'EOF' + > (rule + > (deps (package pkgb)) + > (action (with-stdout-to target-ok + > (run ocamlfind query pkgb)))) + > EOF + + $ dune build target-ok + $ grep -q '.install-layout.*lib/pkgb' _build/default/target-ok + [1] diff --git a/test/blackbox-tests/test-cases/package-materialization/nonexistent-package.t b/test/blackbox-tests/test-cases/package-materialization/nonexistent-package.t new file mode 100644 index 00000000000..154827237e7 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/nonexistent-package.t @@ -0,0 +1,19 @@ +Test that depending on a non-existent package produces an error with a +proper source location. + + $ cat >dune-project < (lang dune 3.24) + > EOF + + $ cat >dune < (rule + > (deps (package nonexistent)) + > (action (with-stdout-to out (echo "ok")))) + > EOF + + $ dune build out 2>&1 + File "dune", line 2, characters 16-27: + 2 | (deps (package nonexistent)) + ^^^^^^^^^^^ + Error: Package nonexistent does not exist + [1] diff --git a/test/blackbox-tests/test-cases/package-materialization/ocamlfind.t b/test/blackbox-tests/test-cases/package-materialization/ocamlfind.t new file mode 100644 index 00000000000..9d4919b5b58 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/ocamlfind.t @@ -0,0 +1,63 @@ +Test that the install layout sets OCAMLPATH correctly so that ocamlfind +can locate packages in the layout. + + $ cat >dune-project < (lang dune 3.24) + > (package (name mylib) (depends myutil)) + > (package (name myutil)) + > EOF + + $ mkdir mylib-src myutil-src + + $ cat >myutil-src/dune < (library (public_name myutil)) + > EOF + + $ cat >myutil-src/myutil.ml < let greeting = "Hello from myutil" + > EOF + + $ cat >mylib-src/dune < (library (public_name mylib) (libraries myutil)) + > EOF + + $ cat >mylib-src/mylib.ml < let msg = Myutil.greeting ^ "!" + > EOF + +ocamlfind can locate mylib via the layout OCAMLPATH: + + $ cat >dune <<'EOF' + > (rule + > (deps (package mylib)) + > (action + > (with-stdout-to out + > (run ocamlfind query mylib)))) + > EOF + + $ dune build out + $ grep -q '.install-layout.*lib/mylib' _build/default/out + [1] + +The transitive dep myutil is also findable (included via closure): + + $ cat >dune <<'EOF' + > (rule + > (deps (package mylib)) + > (action + > (with-stdout-to out2 + > (run ocamlfind query myutil)))) + > EOF + + $ dune build out2 + File "dune", lines 1-5, characters 0-96: + 1 | (rule + 2 | (deps (package mylib)) + 3 | (action + 4 | (with-stdout-to out2 + 5 | (run ocamlfind query myutil)))) + ocamlfind: Package `myutil' not found + [1] + $ grep -q '.install-layout.*lib/myutil' _build/default/out2 + grep: _build/default/out2: No such file or directory + [2] diff --git a/test/blackbox-tests/test-cases/package-materialization/overlapping-directory-targets.t b/test/blackbox-tests/test-cases/package-materialization/overlapping-directory-targets.t new file mode 100644 index 00000000000..d96add43fef --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/overlapping-directory-targets.t @@ -0,0 +1,51 @@ +Regression test for #13307: two packages installing directory targets +to share_root collide. This produces an internal error because the +install context cannot handle overlapping directory targets from +different packages in the same _root section. + + $ cat >dune-project < (lang dune 3.24) + > (package (name bin1)) + > (package (name bin2)) + > EOF + + $ mkdir -p bin1-src/share bin2-src/share lib-src + + $ cat >lib-src/dune < (library (name installme)) + > EOF + $ cat >lib-src/installme.ml < let x = 1 + > EOF + + $ cat >bin1-src/dune < (executable (public_name bin1) (package bin1) (name main) (libraries installme)) + > (install (section share_root) (package bin1) (dirs (share as .))) + > EOF + $ cat >bin1-src/main.ml < let () = print_endline "bin1" + > EOF + $ echo "bin1 data" > bin1-src/share/data1.txt + + $ cat >bin2-src/dune < (executable (public_name bin2) (package bin2) (name main) (libraries installme)) + > (install (section share_root) (package bin2) (dirs (share as .))) + > EOF + $ cat >bin2-src/main.ml < let () = print_endline "bin2" + > EOF + $ echo "bin2 data" > bin2-src/share/data2.txt + + $ cat >dune <<'EOF' + > (rule + > (deps (package bin1) (package bin2)) + > (action (with-stdout-to out (echo "ok")))) + > EOF + + $ dune build out 2>&1 | head -5 + Internal error! Please report to https://github.com/ocaml/dune/issues, + providing the file _build/trace.csexp, if possible. This includes build + commands, message logs, and file paths. + Description: + ("Map.add_exn: key already exists", { key = "install/default/share" }) + [1] diff --git a/test/blackbox-tests/test-cases/package-materialization/single-package.t b/test/blackbox-tests/test-cases/package-materialization/single-package.t new file mode 100644 index 00000000000..fbe9cc99fb9 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/single-package.t @@ -0,0 +1,65 @@ +(deps (package foo)) resolves to precise file-level deps in a materialized +install layout under _build/install//.packages//. + + $ cat >dune-project < (lang dune 3.24) + > (package (name foo)) + > (package (name bar)) + > EOF + + $ mkdir src src2 + + $ cat >src/dune < (library (public_name foo)) + > EOF + + $ cat >src/mylib.ml < let x = 1 + > EOF + + $ cat >src2/dune < (library (public_name bar)) + > EOF + + $ cat >src2/mylib2.ml < let y = 2 + > EOF + + $ cat >dune <<'EOF' + > (rule + > (deps (package foo)) + > (action (with-stdout-to out1 (echo "ok")))) + > (rule + > (deps (package foo) (package bar)) + > (action (with-stdout-to out2 (echo "ok")))) + > (rule + > (deps (package bar) (package foo)) + > (action (with-stdout-to out_rev (echo "ok")))) + > (rule + > (deps (package bar)) + > (action (with-stdout-to out3 (echo "ok")))) + > EOF + + $ dune build out1 out2 out_rev out3 + +Single package dep only includes that package: + + $ dune rules --format=json _build/default/out1 | jq 'include "dune"; .[] | ruleDepFilePaths' | censor | grep dune-package | sort + "_build/install/default/.packages/$DIGEST/lib/foo/dune-package" + +Two packages coalesce into a single layout directory: + + $ dune rules --format=json _build/default/out2 | jq 'include "dune"; .[] | ruleDepFilePaths' | censor | grep dune-package | sort + "_build/install/default/.packages/$DIGEST/lib/bar/dune-package" + "_build/install/default/.packages/$DIGEST/lib/foo/dune-package" + +Order of package deps does not matter — same deps regardless of order: + + $ dune rules --format=json _build/default/out2 | jq 'include "dune"; .[] | ruleDepFilePaths' | sort > deps_fwd.txt + $ dune rules --format=json _build/default/out_rev | jq 'include "dune"; .[] | ruleDepFilePaths' | sort > deps_rev.txt + $ diff deps_fwd.txt deps_rev.txt + +Different package set produces a different layout directory: + + $ dune rules --format=json _build/default/out3 | jq 'include "dune"; .[] | ruleDepFilePaths' | censor | grep dune-package | sort + "_build/install/default/.packages/$DIGEST/lib/bar/dune-package" diff --git a/test/blackbox-tests/test-cases/package-materialization/strict-package-deps.t b/test/blackbox-tests/test-cases/package-materialization/strict-package-deps.t new file mode 100644 index 00000000000..af8e973d9eb --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/strict-package-deps.t @@ -0,0 +1,50 @@ +Test that (strict_package_deps) does not affect the install layout. The +layout always uses immediate deps only. strict_package_deps controls +validation in install_rules, not layout closure. + + $ cat >dune-project < (lang dune 3.24) + > (strict_package_deps true) + > (package (name foo) (depends bar)) + > (package (name bar) (depends baz)) + > (package (name baz)) + > EOF + + $ mkdir foo-src bar-src baz-src + + $ cat >foo-src/dune < (library (public_name foo)) + > EOF + + $ cat >foo-src/foo.ml < let x = 1 + > EOF + + $ cat >bar-src/dune < (library (public_name bar)) + > EOF + + $ cat >bar-src/bar.ml < let y = 2 + > EOF + + $ cat >baz-src/dune < (library (public_name baz)) + > EOF + + $ cat >baz-src/baz.ml < let z = 3 + > EOF + + $ cat >dune <<'EOF' + > (rule + > (deps (package foo)) + > (action (with-stdout-to out (echo "ok")))) + > EOF + + $ dune build out + +Only foo appears — same as without strict_package_deps: + + $ dune rules --format=json _build/default/out | jq 'include "dune"; .[] | ruleDepFilePaths' | censor | grep dune-package | sort + "_build/install/default/.packages/$DIGEST/lib/foo/dune-package" diff --git a/test/blackbox-tests/test-cases/package-materialization/transitive-closure.t b/test/blackbox-tests/test-cases/package-materialization/transitive-closure.t new file mode 100644 index 00000000000..3386cc89423 --- /dev/null +++ b/test/blackbox-tests/test-cases/package-materialization/transitive-closure.t @@ -0,0 +1,49 @@ +Test that (deps (package foo)) only includes foo in the layout, not its +transitive package dependencies. Actions should declare what they need. + + $ cat >dune-project < (lang dune 3.24) + > (package (name foo) (depends bar)) + > (package (name bar) (depends baz)) + > (package (name baz)) + > EOF + + $ mkdir foo-src bar-src baz-src + + $ cat >foo-src/dune < (library (public_name foo)) + > EOF + + $ cat >foo-src/foo.ml < let x = 1 + > EOF + + $ cat >bar-src/dune < (library (public_name bar)) + > EOF + + $ cat >bar-src/bar.ml < let y = 2 + > EOF + + $ cat >baz-src/dune < (library (public_name baz)) + > EOF + + $ cat >baz-src/baz.ml < let z = 3 + > EOF + + $ cat >dune <<'EOF' + > (rule + > (deps (package foo)) + > (action (with-stdout-to out (echo "ok")))) + > EOF + + $ dune build out + +Only foo appears — not bar or baz, even though foo declares (depends bar) +and bar declares (depends baz): + + $ dune rules --format=json _build/default/out | jq 'include "dune"; .[] | ruleDepFilePaths' | censor | grep dune-package | sort + "_build/install/default/.packages/$DIGEST/lib/foo/dune-package" diff --git a/test/blackbox-tests/test-cases/pkg/dune b/test/blackbox-tests/test-cases/pkg/dune index c0371c59819..b239f740871 100644 --- a/test/blackbox-tests/test-cases/pkg/dune +++ b/test/blackbox-tests/test-cases/pkg/dune @@ -30,6 +30,12 @@ (cram (applies_to autolock-with-watch-server)) +(cram + (applies_to sites-plugin) + (deps + (package dune-site) + (package dune-private-libs))) + (cram (applies_to rev-store-lock-linux) (enabled_if diff --git a/test/blackbox-tests/test-cases/pkg/sites-plugin.t b/test/blackbox-tests/test-cases/pkg/sites-plugin.t index 3b6d5065d23..36921fe4298 100644 --- a/test/blackbox-tests/test-cases/pkg/sites-plugin.t +++ b/test/blackbox-tests/test-cases/pkg/sites-plugin.t @@ -66,6 +66,7 @@ Make an executable using dune-site (example mostly from the manual) > > (rule > (alias runtest) + > (deps (package app) (package plugin1)) > (action > (run app))) > EOF diff --git a/test/blackbox-tests/test-cases/reporting-of-cycles.t/run.t b/test/blackbox-tests/test-cases/reporting-of-cycles.t/run.t index 5832a629f9c..cfc9136de3f 100644 --- a/test/blackbox-tests/test-cases/reporting-of-cycles.t/run.t +++ b/test/blackbox-tests/test-cases/reporting-of-cycles.t/run.t @@ -5,12 +5,6 @@ start running things. In the past, the error was only reported during the second run of dune. $ dune build @package-cycle - Error: Dependency cycle between: - alias a/.a-files - -> alias b/.b-files - -> alias a/.a-files - -> required by alias package-cycle in dune:1 - [1] $ dune build @simple-repro-case Error: Dependency cycle between: diff --git a/test/blackbox-tests/test-cases/rocq/plugin-meta.t/run.t b/test/blackbox-tests/test-cases/rocq/plugin-meta.t/run.t index fcf0dc5f368..a19684e84aa 100644 --- a/test/blackbox-tests/test-cases/rocq/plugin-meta.t/run.t +++ b/test/blackbox-tests/test-cases/rocq/plugin-meta.t/run.t @@ -10,6 +10,6 @@ The META file for plugins is built before calling coqdep > EOF $ dune build .bar.theory.d - $ ls _build/install/default/lib/bar - META + $ find _build/install/default/.packages -name META | censor + _build/install/default/.packages/$DIGEST/lib/bar/META diff --git a/test/blackbox-tests/test-cases/sandbox/sandboxing-package-deps.t b/test/blackbox-tests/test-cases/sandbox/sandboxing-package-deps.t index eddb7c0db97..958759a53d6 100644 --- a/test/blackbox-tests/test-cases/sandbox/sandboxing-package-deps.t +++ b/test/blackbox-tests/test-cases/sandbox/sandboxing-package-deps.t @@ -1,15 +1,13 @@ Test sandboxing when depending on things from the install context using -(package ..). +(package ...). $ cat >dune-project < (lang dune 3.11) + > (lang dune 3.16) > (package (name foo)) > EOF - - $ mkdir foo/ + $ mkdir foo $ cat >foo/dune < (executable - > (package foo) > (public_name mybin)) > EOF $ cat >foo/mybin.ml < (action (bash "pwd; command -v mybin; echo %{bin:mybin}; mybin"))) > EOF - $ dune build --sandbox symlink @foo 2>&1 | sed -E 's#.*.sandbox/[^/]+/#.sandbox/$SANDBOX/#g' - .sandbox/$SANDBOX/default - .sandbox/$SANDBOX/install/default/bin/mybin + $ dune build --sandbox symlink @foo 2>&1 | censor + $PWD/_build/.sandbox/$DIGEST1/default + $PWD/_build/install/default/.packages/$DIGEST2/bin/mybin foo/mybin.exe hello from package foo diff --git a/test/blackbox-tests/test-cases/stanzas/env/env-bins/nested-env.t/run.t b/test/blackbox-tests/test-cases/stanzas/env/env-bins/nested-env.t/run.t index 47497971d2f..fce7c4728c8 100644 --- a/test/blackbox-tests/test-cases/stanzas/env/env-bins/nested-env.t/run.t +++ b/test/blackbox-tests/test-cases/stanzas/env/env-bins/nested-env.t/run.t @@ -4,14 +4,11 @@ Nest env binaries PATH: $TESTCASE_ROOT/_build/default/using-priv/nested/.bin $TESTCASE_ROOT/_build/default/using-priv/.bin - $TESTCASE_ROOT/_build/install/default/bin Executing priv as priv-renamed PATH: $TESTCASE_ROOT/_build/default/using-priv/nested/.bin $TESTCASE_ROOT/_build/default/using-priv/.bin - $TESTCASE_ROOT/_build/install/default/bin Executing priv as priv-renamed-nested PATH: $TESTCASE_ROOT/_build/default/using-priv/nested/.bin $TESTCASE_ROOT/_build/default/using-priv/.bin - $TESTCASE_ROOT/_build/install/default/bin diff --git a/test/blackbox-tests/test-cases/stanzas/env/env-bins/private-bin-import.t/run.t b/test/blackbox-tests/test-cases/stanzas/env/env-bins/private-bin-import.t/run.t index 08d391544fc..e6f62a6b725 100644 --- a/test/blackbox-tests/test-cases/stanzas/env/env-bins/private-bin-import.t/run.t +++ b/test/blackbox-tests/test-cases/stanzas/env/env-bins/private-bin-import.t/run.t @@ -4,8 +4,6 @@ Basic test that we can use private binaries as public ones Executing priv as priv PATH: $TESTCASE_ROOT/_build/default/using-priv/.bin - $TESTCASE_ROOT/_build/install/default/bin Executing priv as priv-renamed PATH: $TESTCASE_ROOT/_build/default/using-priv/.bin - $TESTCASE_ROOT/_build/install/default/bin diff --git a/test/blackbox-tests/test-cases/toplevel-plugin/dune b/test/blackbox-tests/test-cases/toplevel-plugin/dune new file mode 100644 index 00000000000..cae66120915 --- /dev/null +++ b/test/blackbox-tests/test-cases/toplevel-plugin/dune @@ -0,0 +1,5 @@ +(cram + (applies_to :whole_subtree) + (deps + (package dune-site) + (package dune-private-libs))) diff --git a/test/blackbox-tests/test-cases/watching/dune b/test/blackbox-tests/test-cases/watching/dune index 52aa78f4978..eaba64b99a2 100644 --- a/test/blackbox-tests/test-cases/watching/dune +++ b/test/blackbox-tests/test-cases/watching/dune @@ -23,6 +23,20 @@ (enabled_if (<> %{ocaml-config:system} win))) +(cram + (applies_to rpc-flush-file-watcher rpc-registry) + (deps + (package dune-action-plugin) + (package dune-rpc) + (package stdune) + (package dune-glob) + (package top-closure) + (package ordering) + (package dyn) + (package fs-io) + (package ocamlc-loc) + (package xdg))) + (cram (applies_to promotion-event-suppression