From 263beff4b6d715c1c3ed86fbb60a62b6d0f7700e Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 13 Aug 2026 08:52:20 +0800 Subject: [PATCH 01/19] docs: plan the module interface/implementation split, with the mechanism measured first 110 .cppm files carry both interface and implementation, so every body edit changes the BMI and every importer recompiles. Average transitive downstream is 10.7 modules; platform.cppm's is 63. Before planning the edit, a throwaway probe established that mcpp orders .cpp implementation units via P1689 dyndep and emits no BMI for them, and that the three shapes this codebase actually needs all hold: a non-exported helper declared in the interface and defined in the implementation unit is callable from an exported template instantiated downstream; an extern module-linkage global works the same way; export/default-args/constexpr have to stay put. Records the classification rules, the globals-ordering invariant, the 84.9%/15.1% free-function vs class-body split, the inline/LTO trade-off to measure rather than assume, and main's cold-build baseline (68.85/55.14/64.12s). --- ...-08-13-module-interface-impl-separation.md | 136 ++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 .agents/plans/2026-08-13-module-interface-impl-separation.md diff --git a/.agents/plans/2026-08-13-module-interface-impl-separation.md b/.agents/plans/2026-08-13-module-interface-impl-separation.md new file mode 100644 index 00000000..1bc12acc --- /dev/null +++ b/.agents/plans/2026-08-13-module-interface-impl-separation.md @@ -0,0 +1,136 @@ +# Module interface / implementation separation + +**Branch:** `refactor/module-impl-separation` +**Date:** 2026-08-13 + +## Goal + +Split every C++23 module in `src/` into a standard-conforming pair: + +- `X.cppm` — **module interface unit** (`export module M;`): types, declarations, + templates, `constexpr`. Produces the BMI. +- `X.cpp` — **module implementation unit** (`module M;`): the function bodies. + Produces **no BMI**. + +No directory moves, no behaviour changes, no new product code. + +## Why: the BMI is the recompile trigger + +Today every function body lives in the interface unit, so **every body edit +changes the BMI**, and everything that imports the module recompiles. + +Measured fan-out over the 99 local modules (`build/bench/analyze.py` + +import-graph scan): + +| module | direct importers | transitive downstream | +|---|---|---| +| `xlings.platform` | 45 | **63** | +| `xlings.core.palette` | 8 | 53 | +| `xlings.libs.json` | 31 | 52 | +| `xlings.core.log` | 39 | 47 | +| `xlings.core.config` | 34 | 35 | + +Average transitive downstream: **10.7 modules**. Editing one line in +`platform.cppm`'s implementation rebuilds 64 translation units. After the +split it rebuilds one. + +## Mechanism verified before writing any code + +`build/bench/probe2` (a throwaway 3-file mcpp project) established that mcpp +and gcc@16.1.0 support the standard shape, and that the four rules the +migration depends on actually hold: + +1. mcpp scans `.cpp` implementation units with P1689 dyndep and orders them + after their interface's BMI. `thing.cppm` → `thing.m.o` + `p2.thing.gcm`; + `thing.cpp` → `thing.o` and **no `.gcm`**. +2. A **non-exported** helper declared in the interface and defined in the + implementation unit is callable from an **exported template** that gets + instantiated in a downstream TU. It links and runs. +3. An `extern` module-linkage global declared in the interface and defined in + the implementation unit is readable from such a template. +4. The `export` keyword must be omitted in the implementation unit; a default + argument must appear only in the interface; `constexpr` + `static_assert` + stay in the interface. + +Incremental behaviour on that probe: + +| edit | recompiled | time | +|---|---|---| +| implementation body (`thing.cpp`) | `thing.cpp` only | 0.29s | +| interface (`thing.cppm`) | `.cppm` + `.cpp` + every importer | 0.79s | + +mcpp additionally preserves a BMI's mtime when a recompile produces +byte-identical content, so an interface edit that does not change the BMI +already avoids downstream work. That is why the bodies are the thing to move. + +## Classification rules + +Applied per namespace-scope (or class-scope) entity. + +**STAY in the interface — whole:** +- type definitions (`struct` / `class` / `union` / `enum`), `using`, `typedef`, + namespace aliases, `concept`, `static_assert` +- anything `template<...>` (11 sites, all variadic log/format wrappers) +- `constexpr` / `consteval` functions and variables, `inline` variables +- preprocessor conditionals that select declarations + +**SPLIT — declaration in the interface, definition in the implementation unit:** +- non-template, non-`constexpr` function definitions at namespace scope, + exported or not (a non-exported helper still needs its declaration in the + interface when a template or another staying entity calls it) +- out-of-line-able member functions of non-template classes, including + `static` member functions (defined as `T C::f(...)`, no `static` keyword) +- namespace-scope variable definitions with dynamic initialisation + +**MOVE WHOLE to the implementation unit:** +- anonymous-namespace blocks (4 files, all under `src/core/mirror/`) +- namespace-scope `static` free functions **not** referenced by a staying + entity (internal linkage cannot span two units) + +## Ordering invariant + +All namespace-scope variable *definitions* of a module move to the same unit +(the implementation unit), so their relative dynamic-initialisation order is +preserved. Never split a module's globals across the two units. + +## Scope sizing + +| | lines | share | +|---|---|---| +| outside class bodies (free functions) | 39,258 | 84.9% | +| inside class/struct bodies (need out-of-line members) | 6,995 | 15.1% | +| total across 110 `.cppm` | 46,253 | | + +71 `static ... (...) {` definitions; indent 0 = namespace-scope internal +linkage, indent 4 = static member functions (ordinary out-of-line definitions). + +Both groups are in scope. The class-heavy files are the high-fan-out ones +(`config.cppm` is 93% class body **and** has 34 direct importers), so +skipping them would forfeit much of the benefit. + +## Known trade-off, to be measured not assumed + +Moving a body out of the interface drops its implicit `inline`. Without LTO +the release build loses those cross-TU inlining opportunities. The dev build +is `-O0`; release is `-O2`. The report must carry binary size and, where +cheap, a runtime check — not a claim that this is free. + +Cold-build direction is genuinely uncertain: TU count roughly doubles +(110 → ~220), which costs, while every BMI gets smaller, which pays. Measure +both cold and incremental. + +## Verification + +1. `mcpp build` succeeds. +2. `mcpp test` — full unit suite green (must run `mcpp build` first; + `test_interface_protocol` drives the real binary). +3. e2e suite via `tests/e2e/run_all.sh`. +4. Benchmark main vs branch in this one worktree by switching branches, so + path, filesystem and toolchain fingerprint are identical and only source + content differs. + +## Baseline captured (main, this worktree) + +Cold `mcpp build` after `rm -rf target`, warm global dependency cache, 32 cores: +**68.85s / 55.14s / 64.12s**. Variance is ~25%, so the comparison needs +repeats and a median, not a single pair of numbers. From 826cc2165cc7685adfbf1eceb8daa3189c82d8f1 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 13 Aug 2026 08:57:20 +0800 Subject: [PATCH 02/19] tools: a splitter for the interface/implementation migration, and the sizing analysis behind it split.py transforms one .cppm that carries its own implementation into a standard (.cppm interface, .cpp implementation) pair. It scrubs comments and literals to a length-preserving mask, scans one brace level into items, and classifies each: types/templates/constexpr/inline/const stay whole in the interface; non-template function definitions and dynamically-initialised namespace-scope variables split into a declaration plus a definition; #if conditionals are emitted to both units so a definition never loses its guard. Doing it with one tool rather than 110 hand edits is what makes the diff reviewable -- every file is transformed by the same rules, and gcc verifies the result. Three facts established by probe before the rules were written: - an implementation unit implicitly sees everything its interface declares, including non-exported namespace aliases, type aliases and types, so nothing has to be duplicated - `export` must not appear in the implementation unit, and a default argument must appear only in the declaration - namespace-scope `static` has to lose the keyword: internal linkage cannot span two units, and module linkage already keeps the name module-private analyze.py sizes the job: 84.9% of the 46,253 lines sit outside class bodies. --- .agents/tools/module-split/analyze.py | 93 ++++++ .agents/tools/module-split/cold.sh | 25 ++ .agents/tools/module-split/split.py | 391 ++++++++++++++++++++++++++ 3 files changed, 509 insertions(+) create mode 100644 .agents/tools/module-split/analyze.py create mode 100644 .agents/tools/module-split/cold.sh create mode 100644 .agents/tools/module-split/split.py diff --git a/.agents/tools/module-split/analyze.py b/.agents/tools/module-split/analyze.py new file mode 100644 index 00000000..6cc4fab7 --- /dev/null +++ b/.agents/tools/module-split/analyze.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +"""Size the interface/implementation split: how much of each .cppm is a +namespace-scope function body (movable) vs inside a class body (needs +out-of-line member syntax) vs must-stay (template/constexpr/type).""" +import re, glob, sys, collections + +def strip_for_scan(s): + """Blank out string/char literals and comments so brace matching is sane. + Keeps byte offsets identical.""" + out = list(s) + i, n = 0, len(s) + while i < n: + c = s[i] + if c == '/' and i + 1 < n and s[i+1] == '/': + j = s.find('\n', i) + j = n if j < 0 else j + for k in range(i, j): out[k] = ' ' + i = j + elif c == '/' and i + 1 < n and s[i+1] == '*': + j = s.find('*/', i + 2) + j = n if j < 0 else j + 2 + for k in range(i, j): + if s[k] != '\n': out[k] = ' ' + i = j + elif c in '"\'': + q = c; j = i + 1 + # raw strings R"(...)" + if q == '"' and i > 0 and s[i-1] == 'R': + m = re.match(r'"([^(]*)\(', s[i:]) + if m: + delim = m.group(1) + end = s.find(')' + delim + '"', i) + j = n if end < 0 else end + len(delim) + 2 + for k in range(i, j): + if s[k] != '\n': out[k] = ' ' + i = j; continue + while j < n: + if s[j] == '\\': j += 2; continue + if s[j] == q: j += 1; break + if s[j] == '\n': break + j += 1 + for k in range(i, j): + if s[k] != '\n': out[k] = ' ' + i = j + else: + i += 1 + return ''.join(out) + +tot = collections.Counter() +per_file = [] +for f in sorted(glob.glob('src/**/*.cppm', recursive=True)): + src = open(f).read() + scan = strip_for_scan(src) + lines = src.count('\n') + # find class/struct/union bodies at any depth: `struct X ... {` ... matching `}` + cls_lines = 0 + for m in re.finditer(r'\b(?:struct|class|union)\s+(\w+)[^;{]*\{', scan): + # skip forward declarations (handled by the `{` requirement) + start = scan.index('{', m.start()) + depth, i = 0, start + while i < len(scan): + if scan[i] == '{': depth += 1 + elif scan[i] == '}': + depth -= 1 + if depth == 0: break + i += 1 + cls_lines += src.count('\n', start, i) + tot['lines'] += lines + tot['class_body_lines'] += cls_lines + per_file.append((lines, cls_lines, f)) + +print(f"total .cppm lines : {tot['lines']}") +print(f"inside class/struct : {tot['class_body_lines']} " + f"({100*tot['class_body_lines']/tot['lines']:.1f}%)") +print(f"outside class/struct : {tot['lines']-tot['class_body_lines']} " + f"({100*(tot['lines']-tot['class_body_lines'])/tot['lines']:.1f}%)") +print() +print("files with the most class-body code:") +per_file.sort(key=lambda r: -r[1]) +for lines, cls, f in per_file[:12]: + print(f" {cls:5}/{lines:5} ({100*cls/max(lines,1):4.0f}%) {f}") + +# namespace-scope `static` free functions (internal linkage -> cannot be +# declared in one unit and defined in another) +print() +stat_fns = [] +for f in sorted(glob.glob('src/**/*.cppm', recursive=True)): + scan = strip_for_scan(open(f).read()) + for m in re.finditer(r'^([ \t]*)static\s+(?!.*\b(?:constexpr|inline)\b)([\w:<>,& *]+?)\s+(\w+)\s*\([^;]*?\)\s*(?:const\s*)?\{', scan, re.M): + stat_fns.append((f, m.group(3), len(m.group(1)))) +print(f"namespace/class-scope `static ... (...) {{` definitions: {len(stat_fns)}") +for f, name, ind in stat_fns[:15]: + print(f" indent={ind:2} {name:28} {f}") diff --git a/.agents/tools/module-split/cold.sh b/.agents/tools/module-split/cold.sh new file mode 100644 index 00000000..d4f9a11d --- /dev/null +++ b/.agents/tools/module-split/cold.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# Cold full-build timing. Usage: cold.sh