meta(perf): externalize mimalloc-pprof and optimize the standalone LLVM COFF driver

[zackees]

Context

reld currently has unfinished local work for #94 that vendors zackees/mimalloc-pprof below _vendor/, but that direction is obsolete. The mixed local state also demonstrates why provenance must be explicit: the submodule index and checkout can point at different commits, and a manifest version is not proof that the published crate contents are present.

This issue supersedes #92 and the conflicting vendoring direction in #94. The new program has two independently maintained boundaries:

• Rust consumes an exact, reviewed crates.io release of mimalloc-pprof; _vendor/mimalloc-pprof is deleted.
• The existing public zackees/llvm-ld repository is the standalone C/C++ isolation of LLVM LLD's COFF driver. It is a repository, not a reld branch and not a Rust crate.

The performance premise is a hypothesis, not an expected result. Research in #91 records historical Windows LTO evidence where allocator replacement alone changed a large LLVM/Clang link from roughly 38m47s to roughly 2m22s with mimalloc. That establishes plausible allocator contention on that workload; it does not establish a reld, ordinary COFF, PDB, or current-toolchain speedup.

Version and profiler facts

The actual mimalloc-pprof v0.9.4 tag (5962c145) contains its internal exact DHAT implementation. However, crates.io 0.9.4 has no Cargo feature that removes sampled-profiler hooks, and its build script forces MI_PPROF=1. Therefore default-features = false on published 0.9.4 does not produce a hook-free allocator.

A newer crates.io release carrying the existing pprof compile-time feature gate is a prerequisite for authoritative hook-free benchmarks. Runtime-disabled 0.9.4 measurements may be retained as exploratory evidence only and must be labeled as such. Exact DHAT remains part of mimalloc-pprof and runtime-gated; reld must not switch to the unrelated Rust dhat::Alloc in order to collect it.

Proposal

1. Re-externalize the Rust allocator

• Publish/verify a newer mimalloc-pprof crates.io release whose Cargo feature gate demonstrably compiles sampled pprof hooks out.
• Depend on that release using an exact Cargo version (=x.y.z) and committed checksum/lockfile resolution.
• Delete _vendor/mimalloc-pprof, its .gitmodules entry, and vendoring/update documentation. Update allocator documentation and the Linux platform guide to describe registry provenance and upgrade procedure.
• Keep one mimalloc implementation in each process. Remove the separate global dhat::Alloc branch; use mimalloc-pprof's internal exact DHAT mode.
• This issue records explicit developer approval, as required by AGENTS.md and ci: reject unapproved Rust dependency additions #88, for the direct exact-version crates.io mimalloc-pprof dependency and its reviewed transitive graph. Update the dependency baseline/enforcement in the same reviewed change; this approval extends to no other crate.

2. Isolate the COFF driver in zackees/llvm-ld

Use the existing standalone public repository, C/C++ only. It owns:

• a machine-readable import manifest and reproducible import script;
• the current pinned LLVM commit and the minimal LLD COFF/Common, LLVM support, and PDB source closure;
• a small, auditable patch queue and documented manual upstream-update procedure;
• a narrow stable C ABI around LLD's in-process library entry point;
• native allocator integration for MSVC and MinGW builds.

The Rust safe wrapper, routing, and process-level policy remain in reld. _vendor/llvm-ld may remain a pinned git-backed import because it is a source integration boundary, not a crates.io Rust dependency.

Allocator ownership must be demonstrated rather than inferred. There must be exactly one mimalloc implementation per linker process, and the C/C++ library must prove that LLD/LLVM allocations actually reach it. On Windows, document and validate CRT allocation redirection/interposition. Use allocator counters and attributed stacks/call sites to detect bypasses or duplicate allocator states.

3. Correctness and benchmark matrix

For both the stock/system baseline and every candidate allocator configuration, cover:

• MSVC and GNU/MinGW personas;
• non-LTO, ThinLTO, and full LTO;
• debug/PDB and non-debug links;
• deterministic EXE and PDB production.

Each deterministic case must run twice for self-determinism, then compare raw EXE and PDB bytes against the pinned stock LLD/reference configuration using identical effective arguments, inputs, executable path, cwd, environment, and deterministic switches. Native Windows execution must match exact stdout, stderr, and exit status for both personas. Any unavoidable nondeterministic PDB field must be named and normalized narrowly under DESIGN.md §3.1 while all other bytes remain compared.

Measure this allocator matrix:

1. pinned stock/system allocator baseline;
2. mimalloc hook-free timing build, with sampled pprof and exact DHAT proven compiled/runtime off as applicable;
3. pprof-compiled but runtime-off build, to quantify hook tax;
4. sampled pprof collection build/run;
5. internal exact DHAT collection run.

Sampled-pprof and exact-DHAT collection runs are diagnostic evidence only and must never be used as timing evidence. Runtime-off crates.io 0.9.4 is likewise exploratory, not a substitute for item 2.

For authoritative timing, randomize and interleave configurations, collect at least 10 valid samples per workload/configuration after declared warm-up, and report wall time, CPU time, peak RSS, dispersion, and bootstrap 95% confidence intervals. Keep the allocator change only when the aggregate evidence is better; otherwise revert it rather than merging a speculative default.

4. Profile-driven optimization loop

After the allocator decision, repeatedly collect complementary profiles on the pinned representative workload suite:

• CPU on-CPU profiles;
• off-CPU/wait/I/O/lock profiles;
• sampled allocation pprof;
• internal exact DHAT allocation/lifetime evidence.

Each performance PR contains exactly one causal performance change, RED -> GREEN correctness/regression evidence, before/after profiles, raw benchmark data, artifact-equivalence results, and a detailed explanation of the causal mechanism. Do not batch unrelated optimizations.

Accept an optimization only when the aggregate improvement's lower bootstrap 95% confidence bound is greater than 3%, with no material regression in an individual required workload, correctness case, CPU time, or peak RSS unless the tradeoff was explicitly approved before merge. Merge every accepted improvement as its own validated PR, then reset the unsuccessful-attempt counter and re-profile the new baseline.

Stop only after all of the following are true:

• at least three consecutive, materially distinct deep attempts, each aimed at an evidenced hotspot representing at least 5% of wall time, fail to clear the greater-than-3% acceptance gate;
• any compatible, non-regressing sub-3% changes considered in combination still have an aggregate upper bootstrap 95% confidence bound of at most 3%;
• the current profiles attribute at least 90% of end-to-end wall time to named on-CPU or off-CPU categories;
• a final dossier records attempted hypotheses, patches/results, raw data, residual hotspots, why further work is below threshold, and a concrete revisit trigger such as a new LLVM revision, workload, allocator release, or profiler finding.

Acceptance criteria

• Add a focused RED test/configuration check that fails while reld uses _vendor/mimalloc-pprof, accepts a non-exact registry dependency, retains external dhat::Alloc, or cannot prove profiler-off timing configuration; turn it GREEN with the externalization change.
• Publish or select a post-0.9.4 crates.io release with the verified pprof compile-time gate, pin it exactly, review its resolved graph/checksum, record ci: reject unapproved Rust dependency additions #88 developer approval, and delete all _vendor/mimalloc-pprof state without leftover files or duplicate .gitmodules entries.
• Re-run Linux allocator correctness, native execution, artifact equivalence, and the full randomized benchmark matrix; explicitly prove pprof and exact DHAT are off in authoritative timing binaries/runs.
• Keep and merge the allocator PR only if its controlled evidence is better; otherwise restore the stock allocator and record the negative result.
• Bootstrap zackees/llvm-ld as the C/C++ repository described above, with reproducible source isolation, current LLVM pin, patch queue, C ABI, and allocator attribution proof.
• Add reld-owned Rust bindings/routing while retaining the direct pinned stock-LLD oracle and escape path; _vendor/llvm-ld is pinned and reproducible if used.
• Produce twice-repeated artifact identity and exact native execution evidence across MSVC/GNU, debug/non-debug, and non-LTO/ThinLTO/full-LTO.
• Run the profile-driven loop under the stated statistical merge and stop gates; one detailed validated PR per accepted causal improvement.
• Merge every accepted child PR, close its child issue, leave both repositories clean on their default branches and synchronized with origin, publish the final optimization dossier, and close this meta issue only after every required child is closed.

Decisions

• Supersede Adopt mimalloc-pprof in the Wild fork; stage a minimal LLVM COFF library #92 instead of editing its implementation plan in place: its central _vendor/mimalloc-pprof constraint directly conflicts with the approved registry direction.
• Use zackees/llvm-ld, not a branch: a public repository already exists and provides the clean C/C++ ownership and upstream-sync boundary.
• Keep Rust ownership in reld: it prevents llvm-ld from becoming a Rust packaging project and keeps routing/product policy beside reld.
• Allow _vendor/llvm-ld but prohibit _vendor/mimalloc-pprof: the former pins a C/C++ source boundary; the latter duplicates a published Rust package and caused provenance drift.
• Treat speed as falsifiable: allocator and subsequent optimizations merge only after correctness and statistically defensible improvement.
• Use a greater-than-3% lower-confidence-bound gate: it keeps noise and marginal wins from creating permanent fork maintenance cost.

Dependency graph and related work

• Strong duplicate/superseded umbrella: Adopt mimalloc-pprof in the Wild fork; stage a minimal LLVM COFF library #92
• Research and selected strategy: Research: Windows PE/COFF fast-linking strategy - optimize LLD-COFF fork vs retrofit COFF into Wild #91
• Linux evidence phase to amend: Phase 1b: prove allocator artifact equivalence and benchmark/profile the Linux matrix #93
• Conflicting allocator-vendoring implementation to supersede: Phase 1a: vendor mimalloc-pprof and make it reld's Linux allocator #94
• Standalone C/C++ extraction to amend and execute in the existing repository: Phase 2: bootstrap zackees/llvm-ld as a minimal COFF C++ library with mimalloc-pprof #95
• reld Rust integration to retain/amend: Phase 3: add reld-owned Rust bindings and route Windows through vendored llvm-ld #96
• Dependency approval enforcement: ci: reject unapproved Rust dependency additions #88
• Windows GNU/MSRV acceptance prerequisite: chore: raise MSRV to Rust 1.95 and define Windows GNU acceptance #90

Create or rewrite child issues from this dependency graph so their criteria agree with this issue. A child is complete only after its validated PR is merged and its evidence is attached; this meta issue is complete only when all children are closed.

Open questions

None currently. Measurement results decide whether allocator replacement and later optimizations survive; they are not design questions to resolve by assumption.

[zackees]

Phase 2 handoff: llvm-ld issue #1 / PR #2 head 5ef95b5fd6856e9120616888fab215ccb3213410; staging _staging/llvm-ld local commit 454df45d, clean. LLVM ea7d852a70e8bdfaf601d6626a760f9771b2c4b4, 6,488 SHA-locked files; mimalloc-pprof 0.9.5 checksum 9143eb24178e618a9671d22074e2e64badd61454a1ae5c042325daf8238f5407. C ABI, WinLink/MinGW, UTF-8 lengths, byte callbacks, busy/poison/reentry, one allocator state, pprof/DHAT-off timing implemented. Prior correctness green: https://github.com/zackees/llvm-ld/actions/runs/33107250199 ; CI green: https://github.com/zackees/llvm-ld/actions/runs/33107250248 . Benchmark valid rejection: wall CI95 [0.4533%,3.1742%] vs lower>3%; CPU [0.9760,1.0369], RSS [1.0281,1.0688] vs upper<=1.03; do not reroll/claim gain. Latest runs active: correctness https://github.com/zackees/llvm-ld/actions/runs/33111389886 CI https://github.com/zackees/llvm-ld/actions/runs/33111389899 benchmark https://github.com/zackees/llvm-ld/actions/runs/33111389889 ; provenance green. Latest fixes direct pinned lldMain oracle, Ninja/CMake trace closed audit plus four negative controls, direct-vs-C-ABI non-gating timing and separate time trace. Remaining: terminal artifact inspection, no-HIGH/MEDIUM review, accurate PR/#1 update, main-agent merge decision; then reld _vendor integration/artifact/perf validation. Constraints: remote GHA only; no Docker/Bosn/local build-test-lint; shared checkout/no worktrees; preserve staging; no merge/closure yet; final reld clean main synced and no files left.