feat(aot): generate native-entry Go frame from main/-main (#425 Item 1) - #628
Conversation
|
@ingydotnet A note on gloat's let-go backend, prompted by a recent let-go change: #628 in let-go adds a runtime-only AOT entry frame: boot core, load the program's namespaces, then enter decode This PR pulls that loop into three public rt helpers:
Both templates could drop to a couple of lines and shed the compiler/resolver imports, which should shrink those binaries. One gotcha if you adopt it: No urgency (the two don't share a code path today and nothing's broken).
Let us know your thoughts. Thanks! |
nnunley
left a comment
nnunley
left a comment
There was a problem hiding this comment.
Reviewed (multi-agent + verification). Implements #425 Item 1; sits at the Go executable shell, no new IR ops, parity unaffected. One item to fix pre-merge:
Undeduplicated blank imports can emit invalid Go. In the entry-frame generation the blanks vector is built by keeping :go-pkg per lowered spec with no distinct. If two inputs resolve to the same Go package (multi-file same-namespace, or a duplicate input), the generated import block gets _ "pkg" twice → duplicate-import compile failure. Wrapping the collected packages in distinct before emitting fixes it.
Robustness nits (safe today — both fall back conservatively): go-func-needs-error? (entry_frame.lg ~L260) defaults to true when a signature spans multiple lines or has non-gofmt spacing (it parses only the first line); go-src-has-func? (~L256) uses exact func <name>( matching, so non-standard formatting forces a VM fallback instead of the direct native call. Both are fine for gofmt'd output.
Callers collect one :go-pkg per lowered spec, so a namespace spanning several files (or an input passed twice) hands the same package back more than once and the frame emitted a repeated '_ "pkg"' — which Go rejects, so the generated main.go would not compile. Dedupe at the emission site rather than in lg-compile: :blank-imports is a documented public opt, and a repeated import is wrong whoever passed it, so the invariant belongs with the code that writes the import block. Reported by nnunley in review of #628. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Fixed the import bug, rebased, and added the lane that would have caught it. Now at Duplicate blank imports ( Nothing compiled a generated frame ( The example now runs in the same job as the expensive lowering e2e: The two parser nits — I left both. Your read of the failure modes matches mine, and they're the tolerable ones: an unmatched The coupling bothers me more than the parsing. gogen knows the name and result list structurally, renders them to text, and entry-frame re-derives them with |
nnunley
left a comment
nnunley
left a comment
There was a problem hiding this comment.
So, it looks like this should lean on the code I wrote for gogen - there's a lot of direct string generation that could be handled by it, rather than hand-rolling another copy. The gogen code already has name munging support, too, so that might remove some of the duplication.
|
Thanks for all the great feedback, I will take a look. |
Callers collect one :go-pkg per lowered spec, so a namespace spanning several files (or an input passed twice) hands the same package back more than once and the frame emitted a repeated '_ "pkg"' — which Go rejects, so the generated main.go would not compile. Dedupe at the emission site rather than in lg-compile: :blank-imports is a documented public opt, and a repeated import is wrong whoever passed it, so the invariant belongs with the code that writes the import block. Reported by nnunley in review of #628. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Phase 0 checkpoint — rebased onto current
|
|
Phase 1 checkpoint — lowering metadata is now the source of truth (#657). Head:
Next: Phase 2 (gogen AST frame) or Phase 3 (arity rejection / args order / ns-qualified fallback). |
|
Phase 3 checkpoint — entry/runtime semantics corrected. Head:
Verified: entry_frame + InvokeProgramEntry tests, fib.native Next: Phase 2 (gogen AST frame) or Phase 5 (generic E2E matrix). Phase 4 still needs the Gloat ownership decision. |
|
Follow-up on Phase 1 test hardening + return-hinted entry gap. Head:
Ready for Phase 5 (E2E matrix should include the return-hint cases). |
|
Phase 4 (partial) — frame emission is opt-in. Head:
Correction to the earlier Gloat note (comment): gloat does not need a new gate to avoid a duplicate Draft note for Ingy is staged locally for a quick prose pass before posting. |
|
Status checkpoint on Since the last round:
Next on this branch:
|
|
Phase 6/7 complete — ready for re-review. Phase 6 (local): Phase 7: replied + resolved the five review threads; closed #657 as done here. Lint CI failure on @nnunley — requesting re-review. |
|
CI update on |
|
@ingydotnet follow-up on the earlier note about #628 and gloat's templates. Short version of the boundary I think we want: Gloat owns Gloat's frames; Let-Go owns explicitly requested standalone frames.
Without So:
Earlier I suggested gating one of the two frames if gloat pointed Does that match how you want the boundary drawn for now? Longer term I'm curious whether gloat would ever want to consume a structured let-go manifest/frame, or prefer to keep its specialized drivers. |
installHttpNS is registered from an init(), so net/http — and crypto/tls and crypto/x509 behind it — is reachable in every binary linking pkg/rt, whether or not the program ever opens a socket. The linker has no way to see it is unused, which is part of why an AOT hello-world and an AOT fib(35) differ by 16 bytes. Build with -tags lg_no_http to leave it out. On darwin/arm64, cmd/lg-runtime goes 17,594,402 -> 12,175,106 bytes, and 12,131,874 -> 8,382,754 with -s -w: about 30% either way, and `go tool nm` finds zero net/http or crypto symbols in the tagged binary. That is more than the ~1.9 MB the issue estimated from symbol sizes, because the symbol table undercounts what a package drags in. TinyGo's net/http does not compile, so that lane has always built without the namespace and gains nothing new here — but it already had exactly the stub this tag needs, so http_tinygo.go becomes http_stub.go and serves both lanes off `tinygo || lg_no_http`. One stub, two reasons. This is the http half of #652 only. The other tagged tree named there, go/parser and go/printer via pkg/rt/gogen.go, is left alone because #628 and #657 are both in flight on gogen. The no-http-build CI job exists because nothing in the default build compiles http_stub.go; without it a signature change in http.go would break the tagged lane silently. It also asserts net/http stayed out, so the size win cannot regress unnoticed. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
nooga
left a comment
nooga
left a comment
There was a problem hiding this comment.
LGTM. This is a well-executed, well-tested feature — checked out the branch and ran it for real: built and ran examples/aot/native-entry end to end (fib.native → 55, fib.native x → 5702887), inspected the generated main.go (clean, correct — boot → args → decode → load namespaces → argv-box → prog.Main → error-checked exit). test/e2e/native_entry_matrix_test.go is genuine E2E across 17 cases (public/private -main/main, argv/variadic adaptation, arity rejection, main+-main collision, error propagation and exit codes, both return-hint spellings) — actually builds and executes each binary, not just structural checks on generated source.
Verified the structural concern from nnunley's review is properly addressed: frame emission now goes through gogen AST constructors + go/format, not string templates, and entry selection is driven by structured :fns metadata rather than text-scanning; the old text-scan predicates are an explicit off-critical-path fallback.
On the tinygo-wasi-build CI failure: pulled the actual job log rather than taking the "unrelated infra" claim on faith — confirmed it's a wasmtime-installer flake (curl choking on an unresolved {} version placeholder in the download URL), and confirmed the identical step succeeded cleanly on the next main CI run the same day. Genuinely unrelated to this PR.
Two minor, non-blocking notes:
- Generated
main.go's import block isn't goimports-grouped (stdlib/third-party/local all together) — compiles fine, cosmetic. - A multi-arity
(defn -main ([] …) ([x] …))silently isn't recognized as an entry at all (falls through with no diagnostic) rather than erroring — probably fine given multi-arity-mainis unusual, but worth a doc note if anyone hits it.
One mechanical thing before merge: mergeable: CONFLICTING is, same as the other PRs in this batch, just the generated.sums digest — main has moved ~9 commits since the last rebase. Zero real conflicts locally via the merge=sums driver; needs a git rebase origin/main && make generate push to clear it on GitHub's side.
Approving.
Callers collect one :go-pkg per lowered spec, so a namespace spanning several files (or an input passed twice) hands the same package back more than once and the frame emitted a repeated '_ "pkg"' — which Go rejects, so the generated main.go would not compile. Dedupe at the emission site rather than in lg-compile: :blank-imports is a documented public opt, and a repeated import is wrong whoever passed it, so the invariant belongs with the code that writes the import block. Reported by nnunley in review of #628. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lg-compile now emits a BootCore-based main.go when a recognized entry lowers, so typed AOT binaries reach native code instead of silently running at VM speed. Includes LoadProgramNamespaces drain, fib demo, and the Item 3 summary line. Co-authored-by: Cursor <cursoragent@cursor.com>
…e, RunExecUnit guard Review response on the #425 native-entry frame: - entry-frame no longer reimplements the Go-name munge: entry-go-name now calls the (newly public) lower-go/go-name for private entries, so the munge — including reserved-word handling — stays single-sourced with the lowered emitter. Drops the duplicate private-go-name. - emit-vm-fallback shapes argv to the entry's arity: [] invokes with no args, [argv] passes a single string vector, [& args] spreads os.Args. Previously it always spread, mis-invoking non-variadic entries. - emit-entry-frame-go keeps the entry package blank-imported when the entry itself did not lower, so sibling lowered fns still register init() overrides. Also drops the now-dead `_ = ec` (fallback uses ec). - RunProgramMainChunk simplified; InvokeProgramEntry's registry fallback now iterates namespaces in sorted order for deterministic resolution. - count-lowered-funcs actually excludes __gogen* wrappers and init() now (the summary count was inflated); fixed the docstring and a stray tab. - New tests: TestRunExecUnitReplayOrderAndMainOnce pins the replay-order and main-runs-exactly-once contract (incl. the MainChunk-aliases-a-namespace case) that the refactor onto LoadProgramNamespaces relies on; .lg tests cover the arity-shaped fallback and retained blank import. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Callers collect one :go-pkg per lowered spec, so a namespace spanning several files (or an input passed twice) hands the same package back more than once and the frame emitted a repeated '_ "pkg"' — which Go rejects, so the generated main.go would not compile. Dedupe at the emission site rather than in lg-compile: :blank-imports is a documented public opt, and a repeated import is wrong whoever passed it, so the invariant belongs with the code that writes the import block. Reported by nnunley in review of #628. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing compiled a generated entry frame. test/entry_frame_test.lg asserts on the emitted string, and examples/aot/native-entry/build.sh was referenced by no workflow and no make target — so a frame that emitted invalid Go passed every gate and shipped green. The duplicate blank import fixed in the previous commit is exactly that class: a go build failure that no lane ran. Runs lg-compile, embeds the .lgb, links the binary, and asserts both documented outputs (fib 10 = 55, fib 34 = 5702887). About a second of wall clock from a clean out/, next to the ~60-120s determinism harness in the same job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sums merge driver wrote a stale digest during rebase; make build / lgbgen recomputed the correct source digest for the rebased tree. Co-authored-by: Cursor <cursoragent@cursor.com>
Entry-frame was recovering go-name / needs-error? by scanning rendered Go, which misses collision-resolved names (main + -main → Main vs Main__main) and couples the frame to gogen formatting. lower-ns-to-go- result now hands back per-fn records; lg-compile selects the entry by source name and passes that record to analyze-entry. Text scanning remains only as a compat fallback (#657). Co-authored-by: Cursor <cursoragent@cursor.com>
Publish *command-line-args* immediately after BootCore so top-level forms see the real argv; reject [fixed & rest] with a source diagnostic instead of emitting uncompilable Go; resolve VM fallback via the selected namespace only; and own defer ShutdownAllPods in the standalone frame (same contract as lg / lg-runtime). Co-authored-by: Cursor <cursoragent@cursor.com>
find-program-entry missed `(defn -main ^long [] …)` and `(defn ^long -main [] …)` because it required a bare symbol name and vector? binding — the reader reifies those as with-meta forms. Unwrap with the same ir.build return-hint helpers the lowering pipeline uses (#598), and add a direct lower-ns-to-go-result contract test for shape, Main__main collision metadata, and the string-wrapper equality. Co-authored-by: Cursor <cursoragent@cursor.com>
lg-compile historically emitted only lowered packages. Emitting main.go by default would collide with orchestrators such as Gloat that own their own executable and shared-library frames (and that skip MainChunk while draining overrides). Frame emission is now explicit via --entry-frame; package-only mode and the summary line remain the default. Example/CI pass the flag; e2e covers no-flag / with-flag / package parity. Co-authored-by: Cursor <cursoragent@cursor.com>
Pin private/public bridge emission and package parity under --entry-frame, omit unused pkg/vm imports for typed no-error [] frames, and replace the fib-only CI lane with a full go-build/run matrix (return hints, arity rejection, argv, collision, fallback). Co-authored-by: Cursor <cursoragent@cursor.com>
Replace hand-built Go string templates in entry_frame with gogen constructors (plus defer/slice/variadic-call/range/go-directive helpers) so the frame and private bridge render through go/format. Matrix and constructor unit tests fence the rewrite. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two non-blocking notes from the #628 review. A multi-arity `(defn -main ([] …) ([x] …))` had no single binding vector to classify, so parse-entry-form returned nil — which reads as "no entry in this namespace" and quietly emitted the VM fallback frame. It now parses as an entry with an :unsupported shape, so the existing entry-arity-error path reports it and lg-compile exits non-zero, same as [first & rest]. The frame's import block was one undifferentiated group. gogen renders through a FileSet whose positions are densed so output stays a pure function of the AST (renderFset), and that pass also drops any blank line the printer would emit — so the goimports group break is restored on the rendered text instead, local to this frame rather than by loosening the renderer's determinism invariant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two non-blocking notes from the #628 review. A multi-arity `(defn -main ([] …) ([x] …))` had no single binding vector to classify, so parse-entry-form returned nil — which reads as "no entry in this namespace" and quietly emitted the VM fallback frame. It now parses as an entry with an :unsupported shape, so the existing entry-arity-error path reports it and lg-compile exits non-zero, same as [first & rest]. The frame's import block was one undifferentiated group. gogen renders through a FileSet whose positions are densed so output stays a pure function of the AST (renderFset), and that pass also drops any blank line the printer would emit — so the goimports group break is restored on the rendered text instead, local to this frame rather than by loosening the renderer's determinism invariant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two non-blocking notes from the #628 review. A multi-arity `(defn -main ([] …) ([x] …))` had no single binding vector to classify, so parse-entry-form returned nil — which reads as "no entry in this namespace" and quietly emitted the VM fallback frame. It now parses as an entry with an :unsupported shape, so the existing entry-arity-error path reports it and lg-compile exits non-zero, same as [first & rest]. The frame's import block was one undifferentiated group. gogen renders through a FileSet whose positions are densed so output stays a pure function of the AST (renderFset), and that pass also drops any blank line the printer would emit — so the goimports group break is restored on the rendered text instead, local to this frame rather than by loosening the renderer's determinism invariant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two non-blocking notes from the #628 review. A multi-arity `(defn -main ([] …) ([x] …))` had no single binding vector to classify, so parse-entry-form returned nil — which reads as "no entry in this namespace" and quietly emitted the VM fallback frame. It now parses as an entry with an :unsupported shape, so the existing entry-arity-error path reports it and lg-compile exits non-zero, same as [first & rest]. The frame's import block was one undifferentiated group. gogen renders through a FileSet whose positions are densed so output stays a pure function of the AST (renderFset), and that pass also drops any blank line the printer would emit — so the goimports group break is restored on the rendered text instead, local to this frame rather than by loosening the renderer's determinism invariant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two non-blocking notes from the #628 review. A multi-arity `(defn -main ([] …) ([x] …))` had no single binding vector to classify, so parse-entry-form returned nil — which reads as "no entry in this namespace" and quietly emitted the VM fallback frame. It now parses as an entry with an :unsupported shape, so the existing entry-arity-error path reports it and lg-compile exits non-zero, same as [first & rest]. The frame's import block was one undifferentiated group. gogen renders through a FileSet whose positions are densed so output stays a pure function of the AST (renderFset), and that pass also drops any blank line the printer would emit — so the goimports group break is restored on the rendered text instead, local to this frame rather than by loosening the renderer's determinism invariant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two non-blocking notes from the #628 review. A multi-arity `(defn -main ([] …) ([x] …))` had no single binding vector to classify, so parse-entry-form returned nil — which reads as "no entry in this namespace" and quietly emitted the VM fallback frame. It now parses as an entry with an :unsupported shape, so the existing entry-arity-error path reports it and lg-compile exits non-zero, same as [first & rest]. The frame's import block was one undifferentiated group. gogen renders through a FileSet whose positions are densed so output stays a pure function of the AST (renderFset), and that pass also drops any blank line the printer would emit — so the goimports group break is restored on the rendered text instead, local to this frame rather than by loosening the renderer's determinism invariant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3 participants
Summary
gogen, and stop the silent perf-cliff #425 Item 1: with--entry-frame,lg-compileemits amain.gonative-entry frame when it recognizesmain/-main(includingdefn-), choosing a direct native call when the entry lowered or a namespace-qualified VMInvokeProgramEntryfallback when it did not.--entry-frame). Without the flag,lg-compilewrites only lowered packages — historical behavior for orchestrators such as Gloat that own their own executable / shared-library frames.#520BootCore+rt.LoadProgramNamespaces(NSOrder load withApplyGoOverridesdrain), publishes*command-line-args*before namespace replay, and ownsdefer rt.ShutdownAllPods().:go-name), not rendered-Go text scanning (AOT entry frame: recover the lowered signature from gogen as data, not by string-matching rendered Go #657).N fns lowered; native entry: Main ✓ / none) with or without the frame. Fib demo inexamples/aot/native-entry/reproduces the spike (~16–17× on fib(34)).Ownership (Gloat)
--entry-frame).template/lg-main.go/ shared-lib templates (different MainChunk / override-drain semantics).Test plan
go test ./pkg/rt/ -run 'TestLoadProgramNamespaces|TestRunProgramMainChunk|TestRunExecUnit|TestInvokeProgramEntry'go test ./test/ -run TestRunner/entry_framego test ./test/e2e/ -run TestLgCompileEntryFrameOptIn(no flag / flag / package parity)./examples/aot/native-entry/build.sh→fib.nativeprints55/5702887for fib(10)/fib(34)e21014a7; re-run pending)