Skip to content

Research: Windows PE/COFF fast-linking strategy - optimize LLD-COFF fork vs retrofit COFF into Wild #91

Description

@zackees

Research: Windows (PE/COFF) fast-linking strategy — fork/optimize LLD-COFF vs retrofit COFF into Wild

Purpose

Decide, with evidence, which of two strategies reld pursues to dramatically improve Windows link performance. This issue collects the research; the decision will be recorded here once the research phases (appended as comments) are complete. macOS is explicitly out of scope for this cycle. Windows only.

The two candidate strategies

Option 1 — Fork LLD (lld-link / rust-lld) and optimize its COFF driver

Optimize for non-LTO, thin-LTO, and full-LTO regimes.

  • Pro: lld-link already links Windows correctly, covering the full flag/feature surface. Every optimization is testable with a brutal, cheap oracle: does the optimized path produce a byte-identical deterministic binary vs the unoptimized path? Low bring-up risk.
  • Con: presumably a lower performance ceiling — we inherit lld's architecture (memory model, serialization points, PDB pipeline).

Option 2 — Retrofit PE/COFF into the Wild linker

Wild is dramatically faster on ELF, which makes it the tempting choice.

  • Pro: highest performance ceiling; modern parallel architecture; written in Rust.
  • Con: Wild is ELF-only today. Windows linking has a long tail of machinery with no ELF analog (PDB/CodeView, import libraries, dllimport thunks, SEH/.pdata, /GUARD:CF, resources, manifests, delay-load). Bring-up could burn days-to-weeks of effort and heavy token spend before we even learn whether a blocker (most likely PDB) is fatal. Byte-identity testing vs lld is impossible by construction — a different linker produces different bytes — so the test oracle is weaker (behavioral equivalence).

Fixed constraints (decided up front)

  1. Both toolchain flavors are in scope: MSVC (link.exe-style, *-pc-windows-msvc) and GNU/MinGW (*-pc-windows-gnu). (soldr preferred MSVC early on; we've since evolved — both now.)
  2. This is a general-purpose C/C++/Rust linker. Linkers are language-agnostic: rustc emits ordinary COFF objects and .rlibs (plain ar archives whose .rmeta member a linker must skip — standard archive handling). No Rust-specific format support exists or is needed; "Rust support" = correct COFF/archive handling + accepting the flags rustc passes. (Research must verify this claim, not assume it.)
  3. All link modes matter: non-LTO (dev inner loop), thin-LTO, full-LTO. Once the design is right, an agent grinds each mode until it is as fast as possible.
  4. Full link.exe flag/feature compatibility is mandatory — the whole surface (/WHOLEARCHIVE, /ALTERNATENAME, /EXPORT, /DEF, /DELAYLOAD, /GUARD:CF, /DEBUG + PDB, def files, manifests, resources, …). Sole MVP exception: /INCREMENTAL may be deferred. Everything else is required.
  5. CRT selection /MT vs /MD is super important and must be handled correctly (defaultlib resolution, /NODEFAULTLIB, mixed-CRT diagnostics).
  6. Zig's self-hosted COFF linker is potentially super important prior art — their bring-up experience (esp. PDB) is direct evidence for the Wild-retrofit cost estimate.

Research questions

A. Scoping / baseline

  • A1. Where does Windows link time actually go today (lld-link vs link.exe; symbols vs no symbols; LTO vs not)? Profile evidence, not vibes. If PDB emission dominates, that reshapes everything.
  • A2. Verify constraint Meta: road to a working cross-platform link (Windows / Linux / macOS) #2: confirm rustc's linker contract on windows-msvc and windows-gnu (flags passed, rlib handling, self-contained mode, -fuse-ld/linker-flavor mechanics). Is there truly zero Rust-specific linker work?
  • A3. In thin/full LTO, is link time dominated by LLVM codegen inside the LTO plugin rather than link mechanics? If yes, does linker choice even matter for LTO builds, meaning the decision should be driven by the non-LTO dev loop?

B. LLD-fork path

  • B1. How fast is lld-link today vs GNU-ld/mold/wild on comparable inputs — how much headroom exists?
  • B2. Where does lld-link spend time (symbol resolution, /ICF, PDB emission, LTO)? What has upstream already parallelized (e.g. parallel PDB type merging, ghash)? What serialization points remain?
  • B3. Is lld-link deterministic/reproducible by default (/Brepro, timestamps, PDB GUIDs)? The byte-identity oracle depends on it.
  • B4. Fork maintenance: COFF-driver churn rate upstream; can optimizations be upstreamed instead of forked; how hard is shipping a patched lld to Rust/C++ users?
  • B5. What did Rust's "lld by default on Windows" efforts learn? Known blockers?

C. Wild-retrofit path

  • C1. What fraction of Wild is format-generic (symbol resolution, layout, parallelism, string merging) vs hard-wired ELF (relocations, section model, dynamic linking, TLS)? Is there any abstraction boundary today, or is ELF baked into core types? This is the load-bearing feasibility question.
  • C2. Has Wild upstream stated any position on COFF/PE (or Mach-O) support? Would they accept a port, or is this a hard fork of a fast-moving project?
  • C3. Enumerate the COFF machinery with no ELF analog that must be written from scratch: import/export tables + import libraries, dllimport thunks, SEH/.pdata unwind, safeseh, /GUARD:CF metadata, resources (.res), manifest embedding, delay-load, TLS callbacks, base relocations, and — the monster — PDB/CodeView writing. Estimated cost of each; which can be borrowed from existing Rust crates or LLVM?
  • C4. Where does Wild's speed actually come from — and does the advantage survive once COFF's costs (esp. PDB) are added? Risk: weeks of porting to land at lld-speed anyway.
  • C5. What is Wild's correctness/test story, and what would our COFF test oracle be given byte-identity vs lld is impossible?
  • C6. Licensing of Wild vs LLVM (Apache-2.0 WITH LLVM-exception) — any shipping constraint?

D. Prior art & third options

  • D1. Zig's self-hosted COFF linker: current status, what works, what stalled, PDB story, lessons on effort. Direct evidence for Option 2's cost.
  • D2. mold/sold history: mold's author split macOS support into a separate product (sold) and later abandoned it. What does that say about cross-format code sharing in a single-format fast linker?
  • D3. Any other fast-COFF efforts (e.g. Microsoft's link.exe improvements, radlink, others)?
  • D4. Is there a hybrid/staged option: optimize lld now (ships this cycle) while a strictly time-boxed Wild-COFF spike answers C1–C4? GNU-flavor-first as a stepping stone (avoids some MSVC machinery)?

E. Decision criteria & kill criteria (to be finalized before any build starts)

  • E1. Estimated bring-up cost to a minimum viable milestone per path: links a real Rust hello-world + one mid-size real C++/Rust project on windows-msvc and windows-gnu; binary runs; tests pass.
  • E2. Written kill criteria per path (e.g. Wild: "if PDB or import-library handling has no feasible design within N effort, stop"). Purpose: never discover infeasibility at the bottom of a rabbit hole.
  • E3. Is byte-for-byte reproducibility a product requirement for reld (content-addressed caching) or only a testing convenience? If product: heavy thumb on the scale for lld.

Known corner-case watchlist (adversarial pass will expand this)

PDB writing (the classic underestimated monster) · /INCREMENTAL expectations (deferred for MVP but users assume it) · full link.exe flag pass-through surface · dllimport-from-static-lib fixups · CRT selection /MT vs /MD and defaultlib resolution · import library flavors (short vs long format, MinGW .dll.a) · ARM64/ARM64EC · authenticode signing space · antivirus/mmap I/O behavior on Windows · string/COMDAT merging semantics differences · /Brepro + PDB GUID determinism.

Process

  1. Opus research agents gather evidence per section (LLD-COFF, Wild internals, Rust linker contract, Zig/mold prior art, link.exe compatibility surface). Each phase's findings are appended to this issue as a comment with links.
  2. Fable reads all findings and runs an adversarial/follow-up pass (corner cases, feasibility gaps); follow-up research appended likewise.
  3. Decision recorded as a final comment on this issue: chosen option, rationale, staged plan, kill criteria.

🤖 Generated with Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions