Skip to content

perf: memoise [Lib.closure] keyed on (linking, for_, libs)#14521

Draft
robinbb wants to merge 3 commits into
robinbb-14492-l8-raw-refs-cachefrom
robinbb-14492-l9-lib-closure-memo
Draft

perf: memoise [Lib.closure] keyed on (linking, for_, libs)#14521
robinbb wants to merge 3 commits into
robinbb-14492-l8-raw-refs-cachefrom
robinbb-14492-l9-lib-closure-memo

Conversation

@robinbb
Copy link
Copy Markdown
Collaborator

@robinbb robinbb commented May 13, 2026 •
edited
Loading

Layer 9 of 9 of #14492. Performance + algorithm reference doc.

Lib.closure is now defined as Memo.exec over a Memo.create keyed on (linking, for_, libs). The per-module filter calls Lib.closure twice per consumer module (once for direct_libs, once for must_glob_libs); without memoisation every call re-traverses the dependency graph.

The list-of-libs key is order- and multiplicity-sensitive — callers that share inputs (e.g. lib_deps_for_module at both call sites) need to canonicalise via List.sort_uniq ~compare:Lib.compare; the existing call sites already do.

Adds doc/dev/per-module-narrowing.md (626 lines): a reference for the narrowing algorithm — frontier construction, BFS expansion through dep-lib references, classification into tight vs glob, soundness recovery (wrapped-closure, ppx-runtime, virtual-impl, stanza-(flags -open)), and the filtered-include-flag emit path. Intended as the implementer's first stop when revisiting lib_deps_for_module or filtered_include_flags.

Stack: rebases on #14520. Final layer of the stack.

Part of #14492. Related to #4572.

@robinbb robinbb self-assigned this May 13, 2026
This was referenced May 13, 2026
Draft
Draft
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from ee1f8ce to 58ed81a Compare May 14, 2026 00:36
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch 2 times, most recently from 215bfef to bf10fd7 Compare May 14, 2026 18:38
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from 58ed81a to ec1ccdb Compare May 14, 2026 23:58
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from bf10fd7 to 6470974 Compare May 14, 2026 23:58
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from ec1ccdb to 8054767 Compare May 15, 2026 02:57
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch 2 times, most recently from 4f3d261 to 8ed3374 Compare May 16, 2026 03:59
robinbb added a commit that referenced this pull request May 16, 2026
@robinbb
doc: per-module narrowing algorithm reference
8ed3374 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request May 16, 2026
@robinbb
doc: per-module narrowing algorithm reference
698f3e4 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in this PR (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request May 18, 2026
@robinbb
doc: per-module narrowing algorithm reference
f543574 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from 8ed3374 to 356728c Compare May 18, 2026 00:07
robinbb added a commit that referenced this pull request May 20, 2026
@robinbb
doc: per-module narrowing algorithm reference
0a75c04 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request May 20, 2026
@robinbb
doc: per-module narrowing algorithm reference
15b7cca 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request May 21, 2026
@robinbb
doc: per-module narrowing algorithm reference
9f29bf4 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from 8054767 to a4b31bc Compare May 21, 2026 01:34
robinbb added a commit that referenced this pull request May 21, 2026
@robinbb
doc: per-module narrowing algorithm reference
0d73789 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from 356728c to e7aa8f5 Compare May 21, 2026 01:34
robinbb added a commit that referenced this pull request May 22, 2026
@robinbb
doc: per-module narrowing algorithm reference
79bae76 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from a4b31bc to bccbb29 Compare May 22, 2026 03:29
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from e7aa8f5 to a38b747 Compare May 22, 2026 03:29
robinbb added a commit that referenced this pull request May 22, 2026
@robinbb
doc: per-module narrowing algorithm reference
5adf365 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added a commit that referenced this pull request May 22, 2026
@robinbb
doc: per-module narrowing algorithm reference
736abe3 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from bccbb29 to 2095cf4 Compare May 22, 2026 23:57
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from a38b747 to 1ca1078 Compare May 22, 2026 23:57
robinbb added a commit that referenced this pull request May 22, 2026
@robinbb
doc: per-module narrowing algorithm reference
f832f21 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from 2095cf4 to 5d6842c Compare May 23, 2026 05:18
robinbb added a commit that referenced this pull request May 23, 2026
@robinbb
doc: per-module narrowing algorithm reference
b2ecadc 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from 1ca1078 to 0c88302 Compare May 23, 2026 05:18
robinbb added a commit that referenced this pull request May 23, 2026
@robinbb
doc: per-module narrowing algorithm reference
952e75f 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb added 3 commits May 23, 2026 14:11
@robinbb
perf: memoise [Lib.closure] keyed on (linking, for_, libs)
aab14c2 
The per-module filter calls [Lib.closure] twice per consumer module
(once for [direct_libs], once for [must_glob_libs]) on each compile
rule. Across a cctx, many modules pass overlapping inputs to these
closures; without memoisation every call re-traverses the dependency
graph.

[Lib.closure] is now defined as [Memo.exec] over a [Memo.create]
keyed on [(bool * Compilation_mode.t * t list)]. The list-of-libs key
is order- and multiplicity-sensitive, so callers that share inputs
need to canonicalise (sort by [Lib.compare]) for maximum cache reuse —
[lib_deps_for_module] already does this at both call sites. A
docstring on [val closure] notes the requirement.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb
doc: per-module narrowing algorithm reference
09a4bc2 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb
doc: extend narrowing reference with effective -open BFS edge
dbf9713 
Documents the fourth class of soundness recovery added in L5: the
BFS's per-iteration step now extends the frontier with the modules
named by each visited library's effective [-open] flags, in
addition to the entry's impl + intf ocamldep raw refs. "Effective"
covers both the library stanza's own [(flags ...)] and any
[(env ...)] stanza that contributes default flags to the dep lib's
directory; both injection paths are captured even when the library
stanza is [:standard]. The reachability rule the BFS computes
becomes a three-way disjunction (consumer references it; reached
module's ocamldep names it; reached module's owning lib's effective
flags open it).

Updates the [cross_lib_tight_set] code snippet's signature (now
threads [~mode]), the per-iteration description, the "Soundness
recovery and known edge cases" list (adds a fourth class), the L5
layer-summary line, and the cost-characteristics list (adds the
per-visited-lib effective-flags expansion).

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
@robinbb robinbb force-pushed the robinbb-14492-l8-raw-refs-cache branch from 5d6842c to 8a4a3d3 Compare May 23, 2026 21:24
@robinbb robinbb force-pushed the robinbb-14492-l9-lib-closure-memo branch from 0c88302 to dbf9713 Compare May 23, 2026 21:24
robinbb added a commit that referenced this pull request May 23, 2026
@robinbb
doc: per-module narrowing algorithm reference
3948c6d 
Add [doc/dev/per-module-narrowing.md] describing the per-module
library file dependency narrowing introduced in #14492 (split into
PRs #14513..#14521 as layers L1..L9):

- The motivation and soundness model.
- The [can_filter] precondition and [has_virtual_impl] early-out.
- The narrowing pipeline: read ocamldep raw refs → [referenced] →
  [Lib.closure] → cross-library BFS → classification → emit per-lib
  deps and filtered include flags.
- The data structures used ([Lib_index], the per-cctx
  [cached_raw_refs] / [Filtered_includes] / [Lib.closure] memos).
- Soundness fallbacks (wrapped libs, virtual impls, ppx runtime).
- A source map locating each concern in [src/dune_rules/].
- A layer-by-layer summary of #14513..#14521.

Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@robinbb robinbb

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@robinbb