perf(loader): stop parsing dependencies from source (2.5–3.6x faster, 626MB → 42MB) - #30
perf(loader): stop parsing dependencies from source (2.5–3.6x faster, 626MB → 42MB)#30nccapo wants to merge 2 commits into
Conversation
Package loading was most of the wall clock and nearly all of the memory: 680ms and 905MB for a 27-route service, of which 280 packages were gin, gorm and the standard library, parsed and type-checked from source to build syntax trees this analyzer never reads. Dependencies are only ever consulted through types.Type, and export data provides that completely. Dropping packages.NeedDeps leaves the four packages of the application itself parsed and everything else type-checked from export data: 297ms and 16MB for the same load. The type index changed to match. It used to expand every name of every package into a map, walking packages.Package, which needs dependency source. It now holds package pointers and resolves names through types.Package.Imports, which export data populates — and a scope lookup was already O(1), so expanding names up front only ever materialized names nothing asked for. Measured end to end, against the pinned upstream RealWorld corpus and synthetic services of 192 and 800 routes: 2.5x to 3.6x faster, and peak RSS from 626MB to 42MB. Analyzer output is byte-identical across all 23 corpora, with one exception, below. Two tests lock the property in: dependencies must not be parsed, and their types must still resolve. Getting only the first is easy and useless. Dropping dependency source also removed a false positive. The credential scan follows one call level below a middleware, and with the standard library parsed it would descend into net/http's own (*Request).BasicAuth, which reads the Authorization header — reporting bearer for a route that uses HTTP Basic. The load mode hid it by accident, so the descent is now explicitly confined to the analyzed packages, and its test loads with NeedDeps on purpose to check the boundary rather than the accident. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Loading dependencies from export data instead of source made the analyzer depend on the toolchain's export data format, and x/tools v0.42.0 cannot read Go 1.27's. Under a Go 1.27 go list every corpus dies before producing anything: internal error: package "github.com/gin-gonic/gin" without types was imported from ".../testdata/gin-realworld/common" x/tools calls log.Fatal there, so it is not a degraded result — the process exits. Go 1.27 is the current release, so this would have hit anyone whose go was newer than this repository's. x/tools v0.49.0 reads it. Adding NeedCompiledGoFiles or NeedExportFile does not help; only the dependency version does. Verified by probing each mode against both drivers: with v0.49.0 the lean mode loads on 1.25 and 1.27 alike, still parsing no dependency from source. CI now runs the tests on 1.27 as well as the 1.25 that go.mod requires. The load path depends on a format the toolchain owns, so testing only the pinned version cannot catch this class of break — and did not. Analyzer output is unchanged across all 23 corpora and the upstream RealWorld repository, under both drivers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Go 1.27 compatibility: this PR needed a fix, pushed as
|
| mode | go1.25 driver | go1.27 driver |
|---|---|---|
| lean (this PR) | 4 roots, 0 deps parsed | ❌ fatal |
lean + NeedCompiledGoFiles |
0 deps parsed | ❌ fatal |
lean + NeedExportFile |
0 deps parsed | ❌ fatal |
| lean + both | 0 deps parsed | ❌ fatal |
NeedDeps (master) |
252 deps parsed | 268 deps parsed |
So no mode flag fixes it — only the dependency version. x/tools v0.42.0 → v0.49.0 reads it, and the lean mode then loads correctly on both drivers with 0 dependencies parsed.
Verified with x/tools v0.49.0: full suite green under both drivers, the pinned upstream corpus gate passes under both, and analyze --json output is identical across all 23 corpora and the upstream repository — and identical between the two drivers.
CI
The test job now runs on 1.25 and 1.27. The load path depends on a format the toolchain owns, so testing only the pinned version cannot catch this class of break, and didn't.
And the answer on 1.27 performance
Not what either of us expected — it is slower here, consistently, with warm build caches for both toolchains and 100 samples each:
| phase | go1.25 | go1.27 | delta |
|---|---|---|---|
| load (min) | 160.0ms | 176.4ms | −10% |
| load + analyze (min) | 276.3ms | 303.9ms | −10% |
| load + analyze (median) | 285.6ms | 309.7ms | −8.5% |
Allocations went up too: 487k → 508k per run, 66.4MB → 70.4MB. The ~1% the release notes claim for size-specialized malloc is swamped by a slower package-loading path, which is where this workload spends its time.
Caveats: one machine (darwin/arm64), one workload dominated by go/packages. This is not a claim about Go 1.27 generally — for allocation-bound pure-Go code the improvement may well show up. It is a reason not to bump the toolchain for performance here.
1 participant
Where the time actually went
I measured before changing anything, splitting the wall clock into the part
go/packagesowns and the part this analyzer owns:Loading was 80–93% of the run, at 905 MB and 9.4M allocations. The reason:
NeedDeps)NeedDepsNeedDepsmakes go/packages parse and type-check every transitive dependency from source — gin, gorm, the standard library — to build syntax trees this analyzer never reads. Dependencies are only ever consulted throughtypes.Type, and export data provides that completely.Change
internal/loader— droppedpackages.NeedDeps.NeedImportsstays: router detection reads each analyzed package's import set.internal/pipeline— the type index had to change to match. It expanded every name of every package into a map by walkingpackages.Package, which requires dependency source. It now holds package pointers and resolves names throughtypes.Package.Imports, which export data populates. A scope lookup was already O(1), so pre-expanding names only materialized names nothing asked for.Results
Measured with
analyzeend to end, best of 3, against the pinned upstream corpus and synthetic services of 192 and 800 routes:Peak RSS (
/usr/bin/time -l):The test suite also drops from ~160s to ~88s, since every test loads packages.
Correctness
analyze --jsonoutput compared againstmasteracross all 23 testdata corpora plus the upstream repository. Byte-identical everywhere except one endpoint, which this PR fixes:bearerwas a false positive. The credential scan follows one call level below a middleware body, and with the standard library parsed it descended into net/http's own(*Request).BasicAuth:The route uses HTTP Basic. Dropping dependency source hides the bug by accident, so I confined the descent to the analyzed packages explicitly — verified by rebuilding with
NeedDepsrestored plus the guard, which also reports[basic].Tests
TestLoadPackages_DoesNotParseDependencies— no dependency may be parsed from sourceTestLoadPackages_DependencyTypesStillResolve— export data must still give complete types (gin.Contextresolves to a struct). Getting only the first of these is easy and uselessTestDetectAuth_DoesNotTraceIntoDependencies— loads withNeedDepson purpose so dependency source is available, asserts the premise holds, then checks the boundary. It fails withschemes = [basic bearer]without the guardBenchmarkPerf_LoadVsAnalyze— keeps the load/analyze split measurable; point it at a real repo withGODOCLIVE_CORPUS_DIRgo build,go vet,go test ./...,golangci-lintand the pinned upstream corpus gate all pass.Measured next step (not in this PR)
Profiling the 800-route service after this change, our own analysis is 39% of what remains, concentrated in two places that both scan every package and every declaration once per route:
pipeline.findInfoForFuncDeclresolver.findFuncDecl→resolveIdentpackages.Visitsorting map keys on each callThe same "walk every package, then match
fd.Name.Pos()" pattern is duplicated in five places acrossresolver,contract,authandpipeline. Onemap[token.Pos]index built once per run would collapse all of them to O(1) and remove most of thepackages.Visitcost too. It needs a shared index type threaded through those four packages, so it belongs in its own PR.🤖 Generated with Claude Code