feat: per-module library dependency filter via ocamldep BFS by robinbb · Pull Request #14516 · ocaml/dune

robinbb · 2026-05-13T18:22:59Z

Layer 4 of 9 of #14492. The core BFS. NOTE: this is the only layer that is expected to fail tests. This cannot be merged without layer 5.

Module_compilation.lib_deps_for_module activates the tight branch: a per-module BFS over the cross-library dep graph (cross_lib_tight_set) computes the dep-lib modules the consumer actually references. The compile rule sees only those cmi/cmx files. can_filter falls back to glob for Melange, dummy dep graphs, synthesised modules, non-filterable kinds, and consumer-side virtual / parameter cctxs.

Dep_rules.rules gates its singleton short-circuit on ~has_library_deps || for_ = Melange. Compilation_context.create computes and passes the flag. Ocaml_flags.extract_open_module_names surfaces -open Foo references for the BFS frontier. Virtual_rules.is_virtual_or_parameter powers the consumer-side can_filter check. Parameterised_rules passes ~has_library_deps:true to its Dep_rules.rules call.

Includes are still the cctx-wide -I / -H; filtered include flags ship in layer 6. Soundness recovery for wrapped libs / ppx-runtime / virtual-impl deps ships in layer 5.

Cram outputs across per-module-lib-deps/ reflect the tighter dep / rebuild behaviour L4 produces (e.g. singleton-with-requires.t flips from wide-glob to specific-cmi dep tracking, single-module-lib.t / transitive.t / lib-to-lib-*.t etc. drop unreferenced-lib cmi globs). melange-conditional-modules.t (outside per-module-lib-deps/) is also promoted: the Dep_rules.rules change introduces one extra ocamldep (internal) invocation and reorders melc/ocamlc steps in the build trace; the assertion accepts the new shape. A few unrelated test directories (reporting-of-cycles.t, strict-package-deps.t, watching/multiple-errors-output.t, inline-tests/alias-cycle.t, custom-cross-compilation/cross-compilation-ocamlfind.t) get promotions for the same reason — the new dep graph changes their traces.

Five existing cram tests fail on this PR; layer 5's soundness recovery restores them without test file changes:

test/blackbox-tests/test-cases/per-module-lib-deps/auto-wrapped-child-reexport.t
test/blackbox-tests/test-cases/per-module-lib-deps/ppx-runtime-libraries.t
test/blackbox-tests/test-cases/per-module-lib-deps/virtual-library.t
test/blackbox-tests/test-cases/per-module-lib-deps/wrapped-closure-precision.t
test/blackbox-tests/test-cases/per-module-lib-deps/wrapped-reexport-via-open-flag.t

Stack: rebases on #14515. Next: #14517.

Part of #14492. Related to #4572.

Layer 3 of 9 of #14492. Behaviour-equivalent refactor. `Compilation_context.Includes.make` drops `~opaque` and stops emitting `Hidden_deps`; `Includes.t` carries only `-I`/`-H` flags. `Module_compilation.lib_deps_for_module` (scaffold form: always-glob) and `lib_cm_deps` wire the dep set back in via `deps_of_entries`. `build_cm` and `ocamlc_i` are switched to consume `lib_cm_deps`; `Alias` / `Wrapped_compat` short-circuit to the cctx's now-flag-only `Includes`. `Lib_file_deps.deps_with_exts` removed — only used by the old `Cmx` arm of `Includes.make`. Combined effect: same `-I` / `-H`, same lib file deps. Stack: rebases on #14514. Next: #14516. Part of #14492. Related to #4572. Signed-off-by: Robin Bate Boerop <me@robinbb.com>

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

Activates the tight branch in [lib_deps_for_module]: a per-module BFS over the cross-library dependency graph (built from each lib's [ocamldep -modules] output, normalised through [build_lib_index]'s post-pp module map) produces the set of dep-lib modules actually referenced by the consumer module. The compile rule sees only those [.cmi]/[.cmx] files; sibling-module recompilations on unreferenced dep-lib cmi changes drop out. Include flags are still the cctx-wide [-I]/[-H] in this layer; the filtered include flags ship separately. Wrapped-lib soundness recovery, virtual-impl gating on the deps side, ppx-runtime force-glob, and the new soundness test fixtures ship in a follow-up — this commit leaves five existing cram tests broken ([auto-wrapped-child-reexport.t], [ppx-runtime-libraries.t], [virtual-library.t], [wrapped-closure-precision.t], [wrapped-reexport-via-open-flag.t]) that the soundness recovery restores. [Module_compilation]: - [union_module_name_sets_mapped]: parallel fold over a list of [Module_name.Set.t] producers. - [module_kind_is_filterable]: predicate excluding kinds whose dep story is handled outside the BFS ([Root], [Wrapped_compat], [Impl_vmodule], [Virtual], [Parameter]). - [cross_lib_tight_set]: BFS expanding through the lib_index's [(lib, entry)] pairs, reading each entry's impl + intf [ocamldep] output. Non-tight-eligible libs terminate chains. - [lib_deps_for_module]: replaces the scaffold body. A [can_filter] guard (consumer-side virtual / parameter, dummy dep graph, module kind, [Module.has m ~ml_kind]) falls back to glob; otherwise runs the BFS, classifies libs via [Lib_file_deps.Lib_index.filter_libs_with_modules], and emits specific-file deps for tight libs + glob deps for non-tight / unreached-non-eligible libs. Returns the cctx-wide [Includes]; filtered include flags follow in a later layer. [Compilation_context.create]: peek [direct_requires] / [hidden_requires] and pass [has_library_deps] to [Dep_rules.rules]. Single-module stanzas with library deps now produce real dep graphs (the filter needs them). [Dep_rules.rules]: gate the singleton short-circuit on [(not has_library_deps) || for_ = Melange]. Other singletons fall through to the full dep-graph build. [Ocaml_flags]: [extract_open_module_names] surfaces [-open Foo] references that ocamldep doesn't see; they join [BFS]'s initial frontier. [Virtual_rules]: [is_virtual_or_parameter] — true for virtual impls and parameter cctxs; used by [can_filter] to suppress per-module filtering on consumer cctxs whose dep story [Dep_rules] handles specially. [Parameterised_rules]: pass [~has_library_deps:true] to the [Dep_rules.rules] call; conservative — the dep-rules path here serves external parameterised libs whose dep set is built from generated sources. Tests: rebuild-precision promotions for the existing modified-test set in #14492 — cram outputs reflect the tighter dep / rebuild behavior that L4 already produces. New soundness test fixtures (and the two tests gated on filtered include flags, [per-module-include-flags.t] / [add-unreferenced-sibling-lib.t]) are deferred to their respective follow-ups. Signed-off-by: Robin Bate Boerop <me@robinbb.com>

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

+(* BFS over tight-eligible entries: each (lib, entry) pair's impl+intf ocamldep
+   names extend the frontier. Non-tight-eligible libs (wrapped locals,
+   externals, virtual-impls, staged-Pps libs) are skipped by
+   [lookup_tight_entries], terminating chains through them. The [Module.t]
+   supplied here is the post-pp form (constructed in [build_lib_index]), so
+   ocamldep runs on the dep lib's [.pp.ml] (action / non-staged Pps) or directly
+   on the source (no preprocessing / future-syntax). *)


+    (* Classify each lib in [all_libs]: - lib has [None]-entry referenced
+       (mixed-entry or wrapped) → glob (covers the None entries' [.cmi]s); -
+       lib has only [Some] entries referenced → per-module deps; - lib
+       unreached but tight-eligible → drop (link rule pulls it in via
+       [requires_link]); - lib unreached and not tight-eligible → glob.
+       [kept_libs] gets every lib that contributes a tight or glob dep — used
+       by [Compilation_context.filtered_include_flags] to scope the consumer's
+       [-I]/[-H] flags. *)


robinbb self-assigned this May 13, 2026

This was referenced May 13, 2026

refactor: route lib file deps through [lib_deps_for_module] #14515

Merged

fix: soundness recovery for wrapped libs, ppx-runtime, virtual-impl deps #14517

Open

Filter inter-library dependencies per module #14492

Draft

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch 3 times, most recently from d3418da to 246a914 Compare May 14, 2026 00:35

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from da056f0 to a0893d7 Compare May 14, 2026 00:35

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch from 246a914 to d88da28 Compare May 14, 2026 19:07

robinbb force-pushed the robinbb-14492-l4-tight-branch branch 3 times, most recently from 4e63363 to 4c95b95 Compare May 15, 2026 01:31

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch 2 times, most recently from 8dcb05b to 60b92c8 Compare May 20, 2026 01:01

robinbb force-pushed the robinbb-14492-l4-tight-branch branch 2 times, most recently from 48451fc to e65efd4 Compare May 20, 2026 01:16

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch from 60b92c8 to dde100d Compare May 20, 2026 20:13

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from e65efd4 to 6fc605c Compare May 20, 2026 20:16

robinbb requested a review from Alizter May 20, 2026 20:18

robinbb assigned Alizter May 20, 2026

robinbb marked this pull request as ready for review May 20, 2026 20:41

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch from dde100d to 63c3651 Compare May 21, 2026 01:34

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from 6fc605c to b95dd0f Compare May 21, 2026 01:34

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch from 63c3651 to 416e701 Compare May 21, 2026 20:46

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from b95dd0f to d5a41a6 Compare May 21, 2026 20:47

robinbb force-pushed the robinbb-14492-l3-includes-refactor branch from 416e701 to 1c1953d Compare May 22, 2026 21:16

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from d5a41a6 to 73246cc Compare May 22, 2026 21:44

Base automatically changed from robinbb-14492-l3-includes-refactor to robinbb-14492-test-stack-integration May 23, 2026 04:51

robinbb force-pushed the robinbb-14492-test-stack-integration branch from c8ab405 to 57f1c08 Compare May 23, 2026 05:17

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from 73246cc to f156ea2 Compare May 23, 2026 05:17

robinbb requested a review from Copilot May 23, 2026 05:18

Copilot AI reviewed May 23, 2026

View reviewed changes

Copilot started reviewing on behalf of robinbb May 23, 2026 05:19 View session

robinbb requested a review from Copilot May 23, 2026 06:25

Copilot started reviewing on behalf of robinbb May 23, 2026 06:25 View session

Copilot AI reviewed May 23, 2026

View reviewed changes

Comment thread src/dune_rules/module_compilation.ml Outdated

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from f156ea2 to ec8180d Compare May 23, 2026 20:35

robinbb requested a review from Copilot May 23, 2026 20:36

Copilot started reviewing on behalf of robinbb May 23, 2026 20:36 View session

Copilot AI reviewed May 23, 2026

View reviewed changes

Comment thread src/dune_rules/module_compilation.ml Outdated

robinbb force-pushed the robinbb-14492-l4-tight-branch branch from ec8180d to 662e008 Compare May 23, 2026 21:24

robinbb requested a review from Copilot May 23, 2026 21:24

Copilot started reviewing on behalf of robinbb May 23, 2026 21:24 View session

Copilot AI reviewed May 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: per-module library dependency filter via ocamldep BFS#14516

feat: per-module library dependency filter via ocamldep BFS#14516
robinbb wants to merge 1 commit into
robinbb-14492-test-stack-integrationfrom
robinbb-14492-l4-tight-branch

robinbb commented May 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

robinbb commented May 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

robinbb commented May 13, 2026 •

edited

Loading