Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .claude/rules/c-runtime-memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,11 @@ iteration, or a collector that quietly stops working).
`make asan-http` (ext_http + model under ASan/UBSan; CI runs the suite that
way, and the HTTP sections are probe-gated so they pull in automatically).
Same for `ext_gfx.c` — in **no** default build; compile-check with
`make gfx`. All variants land on `src/eigenscript`, so never rebuild one
while a suite run against another is in flight.
`make gfx`. Variants coexist in per-variant `build/<variant>/` objdirs
(#740); `src/eigenscript` is a hard link to the last-built one, so a
rebuild no longer destroys another variant's binary — but running any
`make` variant target mid-suite still re-points the alias under the
suite (the #681 guard catches it).
- **A per-request leak in `ext_http.c` will not be caught by any sanitizer
gate.** LeakSanitizer runs atexit, and the server is torn down with `kill`
against no SIGTERM handler, so LSan never runs in the server process —
Expand Down
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,36 @@ All notable changes to EigenScript are documented here.

## [Unreleased]

### Changed

- **Build variants now coexist: per-variant objdirs with dependency
tracking (#740).** Every runtime variant (`build`/`full`/`http`/`zlib`/
`net`/`gfx`/`asan`/`asan-http`/`tsan`/`valgrind`/`poison`) was a single
whole-program `gcc` invocation writing `src/eigenscript`, so an ASan
build destroyed the release binary, any edit cost a full 22-TU rebuild,
and the project paid for the collision twice over — the test runner's
binary-fingerprint guard (#681) and a standing CLAUDE.md prohibition.
Each variant now compiles into its own `build/<variant>/` objdir with
`-MMD`/`-MP` header-dependency tracking and links
`build/<variant>/eigenscript`; `src/eigenscript` becomes a **hard
link** the phony target re-points, so every existing consumer of that
path works unchanged — hard rather than symbolic because the runtime
resolves the stdlib relative to `/proc/self/exe`, which dereferences a
symlink into `build/` and loses `lib/` (a symlink prototype failed 58
suite checks exactly there; the hard link keeps the exec'd path in
`src/`). Measured: switching `make asan` → `make` went from a ~90 s
full rebuild to a 0.2 s relink; touching `vm.c` recompiles one TU;
touching `vm.h` recompiles exactly its 12 includers. Target names,
flags per variant, and output messages are unchanged. `pgo`/`coverage`
(whole-program by nature) and `build.sh` now `rm -f` the alias first
so they write a fresh regular file rather than truncating the shared
inode under a variant. The fingerprint guard stays: re-pointing the
alias or relinking the same variant mid-suite is still a mid-run swap,
and the guard is what catches it (the guard's stat now uses `-L` so
its metadata half describes the same file its cksum half reads) — what
is retired is the cross-variant half of the prohibition, not the
guard.

### Fixed

- **The `ValType` switches are now exhaustive too, closing out the #738
Expand Down
13 changes: 10 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@ ours onto the new main. Rebasing our own costs nothing; asking them does. (Only
## Build & test

```
make # release build -> src/eigenscript (HTTP/MODEL/DB off)
make # release build -> build/release/, src/eigenscript hard-links to it (HTTP/MODEL/DB off)
make test # build + full suite (tests/run_all_tests.sh)
make asan # ASan+UBSan build (same binary path!) — extensions OFF
make asan # ASan+UBSan build — extensions OFF
make asan-http # ASan+UBSan *with* ext_http+model (CI gate; leaks still need RSS, see #731)
make http # http+model variant — run tests/test_http_server.sh
make zlib # DEFLATE codecs (inflate/deflate builtins) via system zlib (-lz)
Expand Down Expand Up @@ -69,7 +69,14 @@ bash tools/embed_stack_soak.sh # embed REPL soak inside a 64 KiB stack rlimit (
transition, and the JIT counters / OSR / inline-cache writes / trace-line are
gated off under MT, name hashes precomputed at compile time. ThreadSanitizer
here needs `setarch -R` to disable ASLR.)
- `make asan` overwrites `src/eigenscript` — rebuild with `make` before timing.
- Variants build into per-variant `build/<variant>/` objdirs (#740) and
coexist; `src/eigenscript` is a hard link to the last `make` target
(hard, not symbolic — `/proc/self/exe`-relative stdlib resolution must
keep seeing `src/`), so switching variants is an instant relink (`make`
after `make asan` costs ~0.2s, not a rebuild). Don't run any `make`
variant target while a suite is in flight — it re-points the alias
under the suite (the #681 fingerprint guard catches it at the next
section seam).
- Benchmarks: `tests/bench_perf.eigs` (micro), `tests/bench_dmg_shape.eigs`
(dispatch-table interpreter shape, the DMG/cpu_instrs stand-in),
`tests/bench_idxset.eigs` (fn-local buffer/list write loop — one JIT thunk,
Expand Down
212 changes: 119 additions & 93 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,95 @@ LSP_BINARY := $(SRC_DIR)/eigenlsp
DAP_SOURCES := $(SRC_DIR)/eigsdap.c $(SRC_DIR)/tape_read.c $(filter-out $(CLI_ONLY),$(SOURCES))
DAP_BINARY := $(SRC_DIR)/eigsdap

.PHONY: all build full http net gfx zlib lib amalgamation tsan test install install-gfx clean coverage coverage-clean fuzz fuzz-run lsp dap jit-smoke embed-smoke asan valgrind pgo freestanding-check freestanding-libc-diff asan-http print-%
.PHONY: all build full http net gfx zlib lib amalgamation tsan test install install-gfx clean coverage coverage-clean fuzz fuzz-run lsp dap jit-smoke embed-smoke asan valgrind pgo poison freestanding-check freestanding-libc-diff asan-http print-%

# ---- Per-variant objdir engine (#740) -------------------------------------
# The engine's rules are defined before `all`, so pin the default goal.
.DEFAULT_GOAL := all
# Every runtime variant compiles into its own build/<variant>/ objdir with
# -MMD/-MP header-dependency tracking, links build/<variant>/eigenscript,
# and the phony target re-points src/eigenscript at it (hard link — see
# RELINK below for why not a symlink). So: variants COEXIST (make asan no
# longer destroys the release binary — and switching back is an instant
# relink, not a 22-TU rebuild), and rebuilds within a variant are
# incremental. The alias keeps every existing consumer of src/eigenscript
# working unchanged. The suite's fingerprint guard (#681) still applies:
# re-pointing the alias or relinking the same variant mid-suite is caught
# at the next section seam.
VERDEF := -DEIGENSCRIPT_VERSION='"$(VERSION)"'
DEFS_OFF := -DEIGENSCRIPT_EXT_HTTP=0 -DEIGENSCRIPT_EXT_MODEL=0 -DEIGENSCRIPT_EXT_DB=0
MODEL_SRC := $(SRC_DIR)/model_io.c $(SRC_DIR)/model_infer.c $(SRC_DIR)/model_train.c
ASAN_FLAGS := -fsanitize=address,undefined -Werror=switch -g -O1

SRC_V_release := $(SOURCES)
FLAGS_release := $(CFLAGS) $(DEFS_OFF) $(VERDEF)
LIBS_release := $(LDFLAGS)

SRC_V_full := $(FULL_SOURCES)
FLAGS_full := $(CFLAGS) -I/usr/include/postgresql -DEIGENSCRIPT_EXT_NET=1 $(VERDEF)
LIBS_full := $(LDFLAGS) -lpq

SRC_V_http := $(SOURCES) $(SRC_DIR)/ext_http.c $(MODEL_SRC)
FLAGS_http := $(CFLAGS) -DEIGENSCRIPT_EXT_HTTP=1 -DEIGENSCRIPT_EXT_MODEL=1 -DEIGENSCRIPT_EXT_DB=0 $(VERDEF)
LIBS_http := $(LDFLAGS)

SRC_V_zlib := $(SOURCES)
FLAGS_zlib := $(CFLAGS) $(DEFS_OFF) -DEIGENSCRIPT_EXT_ZLIB=1 $(VERDEF)
LIBS_zlib := $(LDFLAGS) -lz

SRC_V_net := $(SOURCES) $(SRC_DIR)/ext_net.c
FLAGS_net := $(CFLAGS) $(DEFS_OFF) -DEIGENSCRIPT_EXT_NET=1 $(VERDEF)
LIBS_net := $(LDFLAGS)

SRC_V_gfx := $(SOURCES) $(SRC_DIR)/ext_gfx.c
FLAGS_gfx := $(CFLAGS) $(DEFS_OFF) -DEIGENSCRIPT_EXT_GFX=1 $(VERDEF)
LIBS_gfx := $(LDFLAGS) -ldl

SRC_V_asan := $(SOURCES)
FLAGS_asan := $(ASAN_FLAGS) $(DEFS_OFF) $(VERDEF)
LIBS_asan := -lm -lpthread

SRC_V_asan-http := $(SOURCES) $(SRC_DIR)/ext_http.c $(SRC_DIR)/ext_net.c $(MODEL_SRC)
FLAGS_asan-http := $(ASAN_FLAGS) -DEIGENSCRIPT_EXT_HTTP=1 -DEIGENSCRIPT_EXT_MODEL=1 -DEIGENSCRIPT_EXT_DB=0 -DEIGENSCRIPT_EXT_NET=1 $(VERDEF)
LIBS_asan-http := -lm -lpthread

SRC_V_tsan := $(SOURCES)
FLAGS_tsan := -fsanitize=thread -Werror=switch -g -O1 $(DEFS_OFF) $(VERDEF)
LIBS_tsan := -lm -lpthread

SRC_V_valgrind := $(SOURCES)
FLAGS_valgrind := -Werror=switch -g -O1 -DEIGS_VALGRIND $(DEFS_OFF) $(VERDEF)
LIBS_valgrind := -lm -lpthread

SRC_V_poison := $(SOURCES)
FLAGS_poison := -g -O1 -DEIGS_POISON $(DEFS_OFF) $(VERDEF)
LIBS_poison := -lm -lpthread

VARIANTS := release full http zlib net gfx asan asan-http tsan valgrind poison

# Objects depend on Makefile+VERSION so a flag or version-string change
# rebuilds; header edits are covered by the generated .d files.
define VARIANT_RULES
OBJ_$(1) := $$(patsubst $(SRC_DIR)/%.c,build/$(1)/%.o,$$(SRC_V_$(1)))
build/$(1)/%.o: $(SRC_DIR)/%.c Makefile VERSION | build/$(1)
$$(CC) $$(FLAGS_$(1)) -MMD -MP -c $$< -o $$@
build/$(1)/eigenscript: $$(OBJ_$(1))
$$(CC) $$(FLAGS_$(1)) -o $$@ $$(OBJ_$(1)) $$(LIBS_$(1))
build/$(1):
@mkdir -p $$@
-include $$(OBJ_$(1):.o=.d)
endef
$(foreach V,$(VARIANTS),$(eval $(call VARIANT_RULES,$(V))))

# The retarget lives in the phony targets below (not the link recipe) so
# `make <variant>` always points src/eigenscript at that variant, even
# when its binary was already up to date. HARD link, not symlink: the
# runtime resolves lib/ relative to /proc/self/exe, which dereferences a
# symlink to build/<variant>/ and would lose the executable-relative
# stdlib; a hard link keeps the exec'd path at src/eigenscript.
define RELINK
@ln -f build/$(1)/eigenscript $(BINARY)
endef

# Introspection helper: `make print-SOURCES` echoes a variable's value.
# tests/test_leak_guard.sh derives its ASan build source list from the
Expand All @@ -57,71 +145,37 @@ print-%:

all: build

build:
$(CC) $(CFLAGS) -o $(BINARY) $(SOURCES) \
-DEIGENSCRIPT_EXT_HTTP=0 \
-DEIGENSCRIPT_EXT_MODEL=0 \
-DEIGENSCRIPT_EXT_DB=0 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' \
$(LDFLAGS)
@echo "EigenScript $(VERSION) built. Binary: $$(du -sh $(BINARY) | cut -f1)"
build: build/release/eigenscript
$(call RELINK,release)
@echo "EigenScript $(VERSION) built. Binary: $$(du -sh build/release/eigenscript | cut -f1)"

full:
$(CC) $(CFLAGS) -o $(BINARY) $(FULL_SOURCES) \
-I/usr/include/postgresql \
-DEIGENSCRIPT_EXT_NET=1 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' \
$(LDFLAGS) -lpq
@echo "EigenScript $(VERSION) (full) built. Binary: $$(du -sh $(BINARY) | cut -f1)"
full: build/full/eigenscript
$(call RELINK,full)
@echo "EigenScript $(VERSION) (full) built. Binary: $$(du -sh build/full/eigenscript | cut -f1)"

# Build with HTTP + model extensions but without DB (no libpq-dev required).
# Useful for running HTTP test suites on systems without PostgreSQL headers.
http:
$(CC) $(CFLAGS) -o $(BINARY) $(SOURCES) \
$(SRC_DIR)/ext_http.c \
$(SRC_DIR)/model_io.c $(SRC_DIR)/model_infer.c $(SRC_DIR)/model_train.c \
-DEIGENSCRIPT_EXT_HTTP=1 \
-DEIGENSCRIPT_EXT_MODEL=1 \
-DEIGENSCRIPT_EXT_DB=0 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' \
$(LDFLAGS)
@echo "EigenScript $(VERSION) (http+model, no db) built. Binary: $$(du -sh $(BINARY) | cut -f1)"
http: build/http/eigenscript
$(call RELINK,http)
@echo "EigenScript $(VERSION) (http+model, no db) built. Binary: $$(du -sh build/http/eigenscript | cut -f1)"

# Build with the DEFLATE codecs (inflate/deflate builtins, #684) linked
# against the system zlib. Same opt-in pattern as `make http`: the
# default build stays zero-dependency and the four builtins raise
# "compiled without zlib support" there.
zlib:
$(CC) $(CFLAGS) -o $(BINARY) $(SOURCES) \
-DEIGENSCRIPT_EXT_HTTP=0 \
-DEIGENSCRIPT_EXT_MODEL=0 \
-DEIGENSCRIPT_EXT_DB=0 \
-DEIGENSCRIPT_EXT_ZLIB=1 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' \
$(LDFLAGS) -lz
@echo "EigenScript $(VERSION) (zlib) built. Binary: $$(du -sh $(BINARY) | cut -f1)"
zlib: build/zlib/eigenscript
$(call RELINK,zlib)
@echo "EigenScript $(VERSION) (zlib) built. Binary: $$(du -sh build/zlib/eigenscript | cut -f1)"

# Raw TCP sockets on the trace tape (#414). Same opt-in pattern as gfx:
# in no default build, no extra library needed (plain POSIX sockets).
net:
$(CC) $(CFLAGS) -o $(BINARY) $(SOURCES) $(SRC_DIR)/ext_net.c \
-DEIGENSCRIPT_EXT_HTTP=0 \
-DEIGENSCRIPT_EXT_MODEL=0 \
-DEIGENSCRIPT_EXT_DB=0 \
-DEIGENSCRIPT_EXT_NET=1 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' \
$(LDFLAGS)
@echo "EigenScript $(VERSION) (net) built. Binary: $$(du -sh $(BINARY) | cut -f1)"
net: build/net/eigenscript
$(call RELINK,net)
@echo "EigenScript $(VERSION) (net) built. Binary: $$(du -sh build/net/eigenscript | cut -f1)"

gfx:
$(CC) $(CFLAGS) -o $(BINARY) $(SOURCES) $(SRC_DIR)/ext_gfx.c \
-DEIGENSCRIPT_EXT_HTTP=0 \
-DEIGENSCRIPT_EXT_MODEL=0 \
-DEIGENSCRIPT_EXT_DB=0 \
-DEIGENSCRIPT_EXT_GFX=1 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' \
$(LDFLAGS) -ldl
@echo "EigenScript $(VERSION) (gfx) built. Binary: $$(du -sh $(BINARY) | cut -f1)"
gfx: build/gfx/eigenscript
$(call RELINK,gfx)
@echo "EigenScript $(VERSION) (gfx) built. Binary: $$(du -sh build/gfx/eigenscript | cut -f1)"

test: build
cd tests && bash run_all_tests.sh
Expand Down Expand Up @@ -212,13 +266,8 @@ embed-smoke: amalgamation
# the normal -O2 build silently tolerates. ~2x slower; for testing only.
# The full suite runs leak-clean, so leave leak detection on:
# make asan && cd tests && ASAN_OPTIONS=detect_leaks=1 bash run_all_tests.sh
asan:
$(CC) -fsanitize=address,undefined -Werror=switch -g -O1 -o $(BINARY) $(SOURCES) \
-DEIGENSCRIPT_EXT_HTTP=0 \
-DEIGENSCRIPT_EXT_MODEL=0 \
-DEIGENSCRIPT_EXT_DB=0 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' \
-lm -lpthread
asan: build/asan/eigenscript
$(call RELINK,asan)
@echo "EigenScript $(VERSION) (asan+ubsan) built. Binary: $(BINARY)"

# ASan+UBSan over the EXTENSION surface — same variant as `make http`
Expand All @@ -233,43 +282,24 @@ asan:
# would make this unbuildable on a machine without postgres. ext_db.c
# therefore remains unsanitized — a separate, smaller gap.
# make asan-http && cd tests && ASAN_OPTIONS=detect_leaks=1 bash run_all_tests.sh
asan-http:
$(CC) -fsanitize=address,undefined -Werror=switch -g -O1 -o $(BINARY) $(SOURCES) \
$(SRC_DIR)/ext_http.c $(SRC_DIR)/ext_net.c \
$(SRC_DIR)/model_io.c $(SRC_DIR)/model_infer.c $(SRC_DIR)/model_train.c \
-DEIGENSCRIPT_EXT_HTTP=1 \
-DEIGENSCRIPT_EXT_MODEL=1 \
-DEIGENSCRIPT_EXT_DB=0 \
-DEIGENSCRIPT_EXT_NET=1 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' \
-lm -lpthread
asan-http: build/asan-http/eigenscript
$(call RELINK,asan-http)
@echo "EigenScript $(VERSION) (asan+ubsan, http+model+net) built. Binary: $(BINARY)"

# ThreadSanitizer build for the concurrency race gate (tests/test_tsan.sh).
# Complements ASan (which is not run with the thread checker). Run the tests
# under `setarch -R` — ThreadSanitizer needs ASLR disabled here (CLAUDE.md).
tsan:
$(CC) -fsanitize=thread -Werror=switch -g -O1 -o $(BINARY) $(SOURCES) \
-DEIGENSCRIPT_EXT_HTTP=0 \
-DEIGENSCRIPT_EXT_MODEL=0 \
-DEIGENSCRIPT_EXT_DB=0 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' \
-lm -lpthread
tsan: build/tsan/eigenscript
$(call RELINK,tsan)
@echo "EigenScript $(VERSION) (tsan) built. Binary: $(BINARY)"

# Plain -O1 -g minimal build for Valgrind/Memcheck (tests/valgrind_smoke.sh).
# No sanitizers — Valgrind shadows the uninstrumented binary at runtime, so it
# complements ASan/UBSan/TSan (uninit reads, UAF, definite/indirect leaks) on a
# system without instrumented libs. -O1 keeps optimizer-induced false positives
# down while giving usable stacks. Same minimal extension surface as asan.
valgrind:
$(CC) -Werror=switch -g -O1 -o $(BINARY) $(SOURCES) \
-DEIGS_VALGRIND \
-DEIGENSCRIPT_EXT_HTTP=0 \
-DEIGENSCRIPT_EXT_MODEL=0 \
-DEIGENSCRIPT_EXT_DB=0 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' \
-lm -lpthread
valgrind: build/valgrind/eigenscript
$(call RELINK,valgrind)
@echo "EigenScript $(VERSION) (valgrind -O1 -g) built. Binary: $(BINARY)"

# Uninitialized-read hunter (the EigenOS #UD heisenbug class, see
Expand All @@ -279,14 +309,8 @@ valgrind:
# benign zero pages. Run the suite against it, with the raw-malloc boundary
# poisoned too:
# make poison && cd tests && MALLOC_PERTURB_=170 bash run_all_tests.sh
poison:
$(CC) -g -O1 -o $(BINARY) $(SOURCES) \
-DEIGS_POISON \
-DEIGENSCRIPT_EXT_HTTP=0 \
-DEIGENSCRIPT_EXT_MODEL=0 \
-DEIGENSCRIPT_EXT_DB=0 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' \
-lm -lpthread
poison: build/poison/eigenscript
$(call RELINK,poison)
@echo "EigenScript $(VERSION) (poison 0xAA -O1 -g) built. Binary: $(BINARY)"

# Profile-guided optimization. Builds an instrumented binary, runs the
Expand All @@ -298,6 +322,7 @@ PGO_DIR ?= /tmp/eigs-pgo
PGO_RUN ?= cd $(HOME)/DMG && $(CURDIR)/$(BINARY) dmg.eigs roms/cpu_instrs.gb --cycles 200000 >/dev/null
pgo:
@rm -rf $(PGO_DIR) && mkdir -p $(PGO_DIR)
@rm -f $(BINARY) # may be a variant symlink — never write through it
$(CC) $(CFLAGS) -fprofile-generate=$(PGO_DIR) -o $(BINARY) $(SOURCES) \
-DEIGENSCRIPT_EXT_HTTP=0 \
-DEIGENSCRIPT_EXT_MODEL=0 \
Expand Down Expand Up @@ -330,6 +355,7 @@ coverage: coverage-clean
-DEIGENSCRIPT_EXT_DB=0 \
-DEIGENSCRIPT_VERSION='"$(VERSION)"' || exit 1; \
done
@rm -f $(BINARY) # may be a variant symlink — never write through it
$(CC) --coverage -o $(BINARY) $(SOURCES:.c=.o) $(LDFLAGS)
-cd tests && bash run_all_tests.sh > /dev/null 2>&1 || true
@cd $(SRC_DIR) && gcov -n -b $(notdir $(SOURCES)) > ../coverage.txt 2>&1 || true
Expand Down
6 changes: 6 additions & 0 deletions build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@ fi
# JIT_FLAGS=-DEIGENSCRIPT_JIT_FORCE_OFF=1`).
JIT_FLAGS=""

# The Makefile's objdir engine (#740) leaves src/eigenscript as a hard link
# to build/<variant>/eigenscript. Remove the name before compiling so
# build.sh writes a fresh file instead of truncating the shared inode under
# that variant's binary.
rm -f eigenscript

if [ "$1" = "lsp" ]; then
# Language server (src/eigenlsp) — the editor-intelligence half of the
# toolchain. Links eigenlsp.c against the runtime (SOURCES minus the
Expand Down
Loading
Loading