Would you accept moving inline tests to per-module test roots to enable per-module test steps?

[espetro]

Question

Would core maintainers be open to a refactor that moves the inline test blocks out of the runtime source files into per-module test root files, so that build.zig can define per-module test steps (test-core, test-gateway, test-ui, ...) with real incremental-compile wins?

We prototyped this and hit a fundamental blocker with the current layout, backed by measurements below. Before investing in a large refactor we'd like a steer on whether the direction is acceptable and aligned with Zig best practices.

The blocker: transitive test discovery

Zig includes the tests of every file reachable through transitive imports. This repo keeps tests inline in the runtime files (e.g. src/core/app/app_input_runtime.zig has ~379 test blocks, src/ui/transcript/runtime_tests.zig ~227), and the module directories cross-import heavily (src/ui/* and src/gateway/* import src/core/*; src/core/tooling/tool_runtime.zig alone imports ~90 files across core, builtins, and tools).

Consequence: a per-module test root that imports only that module's files still discovers almost the entire suite:

Test unit	Direct imports	Tests actually run
test-core shim (369 core files)	core only	8,638
test-ui shim (72 ui files)	ui only	8,634
test-gateway shim (13 gateway files)	gateway only	2,227
aggregate test (via src/main.zig)	whole graph	8,633

The ui shim runs 8,634 tests because its files transitively pull in core's tests. Making the aggregate test step depend on these per-module units would run the suite roughly three times (we measured 28,135 tests in one zig build test).

Spike measurements

Machine: Apple silicon MacBook Pro (arm64), macOS, Zig 0.16.0, dedicated ZIG_GLOBAL_CACHE_DIR, nothing else building concurrently. Timings via time -p:

Scenario	Time
Baseline zig build test cold	292s
Baseline zig build test warm	172s
Baseline zig build test after touching src/gateway/client.zig	172s (full recompile, as expected)
Per-module test-gateway warm (shim approach)	246s
Per-module test-ui warm (shim approach)	581s
Per-module test-core warm rerun (shim approach)	580s

So the shim approach is strictly worse than the status quo for core and ui: the compile unit isn't actually smaller once transitive imports are counted, and each module unit re-runs nearly the whole suite. The only genuine wins we found were the already-existing standalone benchmark test units (test-ui-activity-benchmark, test-approval-review-benchmark), which compile in milliseconds.

The proposed refactor (if the direction is acceptable)

1. Move inline test blocks out of runtime files into per-module test root files (e.g. src/core/tests.zig, src/gateway/tests.zig, src/ui/tests.zig) that reference the tests explicitly, keeping runtime files import-only.
2. Add per-module test-<module> steps in build.zig rooted at those files, with the aggregate test step as the superset (CI unchanged).
3. Expected ROI (extrapolated from the baseline numbers above): touching one file in src/gateway currently costs ~172s (full-graph recompile through src/main.zig); with a true per-module compile boundary the same edit should cost a test-gateway run in the tens of seconds, and test-core/test-ui iterations would no longer invalidate each other's caches.

This is a large mechanical change (thousands of test blocks across ~470 files), so we don't want to start it without an explicit go-ahead. Alternatives we considered and rejected: shim-based per-module steps without moving tests (measured above - does not work), and restructuring imports so zig test <file> works per-file (cross-module imports are intentional).

Happy to adjust scope (e.g. pilot with just src/gateway, the smallest module at 13 test files / 218 test blocks) if that's easier to review.

Context

This came out of a spike into faster incremental iteration, motivated by the measurement that any test-only edit anywhere in src/core currently costs a full ~172s recompile. The one concrete defect we found along the way (approval-review benchmark tests not wired into the test step) is being submitted as a separate small PR.