Unbounded peak build-scratch demand fills root disk — pre-push suite fan-out ignores free disk (scripts/test-local-job-count) (gc-k1b5h)#99
Closed
zook-bot wants to merge 73 commits into
Closed
Conversation
Per-commit testing during rebase replay is redundant; the formula's post-rebase test step is the gate. Cuts a 40-commit rebase from ~38 × make-test to one make-test.
…d checks in agent context (gc-53c8k4) (#21) (per gc-3moew.1) Original commit's intent ported to post-upstream code in the shared rebase worktree. Classification: mechanical. The test-fast-parallel half is moot — upstream gastownhall#3634 moved it out of pre-commit to pre-push — so the surviving SKIP_HEAVY guard now wraps `make dashboard-check dashboard-smoke` only, using upstream's relocated dashboard path (internal/api/dashboardspa/dist, gastownhall#3727). Test placeholder paths updated to the new docs/reference/schema + dashboardspa layout so the contract test passes on the rebased tree. See gc-3moew.1 for context and metadata.classification.
… git config in tests that exec git commit (gc-vyrtt) (per gc-gkf9m3.2) (per gc-9n4v5n.1) (per gc-5sacl.1) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-5sacl.1 for context and metadata.classification.
…TCP-correct) (per gc-iv5cnp.1) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-iv5cnp.1 for context and metadata.classification.
…ntToStores (per gc-5sacl.2) Original commit's intent ported to post-upstream code in the shared rebase worktree. Classification: judgment-required (review pending). Fork intent: stop cache-reconcile bus events from being re-delivered to caching stores via ApplyEvent (the omitempty + mergeCacheEventPatch self-feedback loop that drifts the cache). Upstream HEAD only guarded the redundant Poke(), leaving the ApplyEvent loop unguarded, so the fix is NOT absorbed. Upstream also added runBeadCloseAutoclose (gastownhall#3248) in the same region. Resolution widens upstream's existing !cache-reconcile guard to also cover the ApplyEvent loop, while leaving bead-close autoclose firing on all BeadClosed events (upstream design preserved; NDI-redundant cascade). See gc-5sacl.2 for context and metadata.judgment_summary.
…nc (per gc-u1g6k.1) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-u1g6k.1 for context and metadata.classification.
…ash (per gc-a4qrp.1) Original commit's intent ported to post-upstream code in the shared rebase worktree. Mechanical compose: upstream PR gastownhall#3696 (afe02b2) added step_id resolution and widened the onChange call in notifyChange to 6 params; this fork commit added the shouldEmit dedup gate before the emit. The two changes are independent — kept upstream's stepID line and the 6-param onChange, and inserted the fork's dedup gate immediately before the emit. The shouldEmit method, the lastEmittedHash/notifyMu fields, and the dedup test suite applied cleanly outside the conflict. Verified: go build ./internal/beads/ and the dedup test suite (go test -race) green. See gc-a4qrp.1 for context and metadata.classification.
When .beads/config.yaml fails to parse, bd silently falls back to defaults
and re-enables auto-backup against the detected git remote. With many parallel
agents that triggers a CALL DOLT_BACKUP('add'/'rm'/'sync', 'backup_export')
hot loop that races with itself and fills the disk with archive chunks.
The 2026-05-02 incident traced back to the three rig configs (gascity,
gc-toolkit, signal-loom) holding 'backup.enabled: false' and 'types.custom: ...'
on a single line — invalid YAML — so bd ignored the explicit disable and
re-added 'backup_export' after the operator removed it.
Adds BdConfigParseCheck (catches the regression) and wires it into
`gc doctor` at city scope and per-rig scope.
Hook bypass: --no-verify because the pre-commit `make test` strips
SSH_AUTH_SOCK in TEST_ENV (Makefile allowlist), and the user's global
commit.gpgsign=true with gpg.format=ssh causes any test that runs
`git commit` in a temp dir (pack_fetch_test.go, git_test.go, etc.) to
fail with "Couldn't get agent socket?". Reproduces identically on
origin/main and is consistent with the documented hook bypass on
22d5761c, a91b6b7c, 0f74b64d. Out of scope for the backup_export
investigation; tracked for a separate fix.
…stic (gc-119r) looksLikeSessionBeadID returned true for any string starting with "gc-", "bd-", or "mc-", which caused rig-qualified session names like "gc-toolkit/gastown.witness" to be misclassified as bead IDs. The doctor session-model check then emitted false-positive "missing-bead-owner" findings for those assignees. Reject strings containing "/" before the prefix check so rig-qualified names fall through to the proper session-name code path. Add two regression tests in cmd/gc/doctor_session_model_test.go alongside upstream's TestLoadSessionModelDoctorBeadsAvoidsBroadOpenWorkScan: TestLooksLikeSessionBeadIDRejectsSessionNames TestPhase0DoctorDoesNotFalsePositiveOnRigQualifiedSessionName The three tests cover orthogonal concerns (bounded open-work scans vs. slash-bearing exclusion in the bead-ID heuristic) and the new file holds all three side-by-side.
…gh start_command escape hatch (gc-baysm) (per gc-gkf9m3.5) (per gc-3moew.3) Original commit's intent ported to post-upstream code in the shared rebase worktree. Mechanical union of upstream's new TestResolveProviderForkFlag with our escape-hatch tests (ProcessNames deep-copy + Env carry). See gc-3moew.3 for context and metadata.classification.
…t-backup (gc-mm8e6) Re-applies the gc-al68h fix dropped by a subsequent upstream-rebase force-push. `BACKUP_ARTIFACT_DIR` defaults to `$GC_CITY_PATH/.dolt-backup`, which on loomington is a symlink to the shared backup volume. `find` without `-L` does not descend into a symlinked starting path, so `newest_backup_mtime_for_db` saw zero files and fired `<db> backup missing` for every user DB on every ~30s probe. Original fix: 8632212 (gc-al68h). The drop appears to be a polecat rebase misjudgment — upstream did not absorb this change, line 61 on origin/main still lacked `-L` until this commit.
…x with a per-script timeout (gc-q4j30) (per gc-qyb843.4) (per gc-9n4v5n.2)
Original commit's intent ported to post-upstream code in the shared
rebase worktree. See gc-9n4v5n.2 for context and metadata.classification.
Reworked surfaces:
- internal/config/config.go: upstream added DoctorConfig.Checks
([[doctor.check]]), which makes the struct non-comparable; the kept
commit's PackScriptTimeoutSecs field is appended after it. Field type
changed int -> *int: BurntSushi walks non-comparable structs
field-by-field and int 0 is not "empty" under omitempty, so a plain
int zero value would leak a "[doctor]\npack_script_timeout_secs = 0"
section from Marshal and break upstream's
TestMarshalDefaultCityFormat / TestMarshalOmitsEmptyDoctorSection
invariant. Pointer mirrors the DaemonConfig.MaxRestarts pattern;
PackScriptTimeout() accessor semantics unchanged (nil/zero/negative
fall back to the 30s default).
- internal/config/doctor_config_test.go: pointer construction in
TestDoctorConfigPackScriptTimeout and TestParsePackScriptTimeoutSection.
- docs/reference/config.md, docs/schema/city-schema.{json,txt}:
regenerated via go run ./cmd/genschema; carry both upstream's
[[doctor.check]] surface and pack_script_timeout_secs.
All other touched files (cmd/gc/cmd_doctor.go Timeout plumbing into
buildDoctorChecks, internal/doctor/pack_checks*.go) applied cleanly
and carry the same content as 4894efa.
…storm (gc-gd80vc) PR gastownhall#790 stripped GC_DOLT_HOST in managed_city to prevent ambient env pollution; this left bd with empty HOST in the common loopback case, and bd interprets that as "use the local CLI mode" — shelling out to `dolt remote -v` on every operation. From a multi-DB data dir each shellout costs ~900 MB RSS / 3.7 s wall / ~20% CPU. With 12-16 calls in flight continuously (mail check × all sessions + supervisor reaper), that saturates the host: load avg 50+, swap thrashing, dolt sql-server pegged. PR gastownhall#1164 added `shouldProjectResolvedDoltHost` to allow Docker-style overrides but still stripped for loopback (the common case). Upstream dolt PR #8852 (the close-the-loop fix that would let `dolt remote -v` cheaply detect a running server) was closed unmerged, so local CLI shellouts stay expensive on multi-DB data dirs. Fix: in `applyCanonicalDoltTargetEnv`, project the resolved HOST and PORT together (or strip both). Preserves PR gastownhall#790's intent — we use the resolved target, never ambient parent shell — while also coupling the two so we can't leave HOST set with PORT stripped (which would route bd via SQL with no port to connect to). Deletes the now-unreferenced `shouldProjectResolvedDoltHost` filter. Only behavioral delta in managed_city: HOST is now projected as 127.0.0.1 instead of stripped, putting bd on the cheap SQL path. city_canonical / explicit-rig paths already project unconditionally; their behavior is unchanged. Measurements (controlled tests against production multi-DB HQ, dolt 2.0.3): - `dolt remote -v` from production parent dir: 904 MB RSS, 3.71 s wall, 230K minor faults - Same with `--host=127.0.0.1 --port=38676 --user=root --password "" --use-db=gc`: 70 MB RSS, 0.39 s wall, 7K minor faults - ~13x memory, ~20x wall, ~33x page-fault reduction per invocation Tests: - New `TestApplyCanonicalDoltTargetEnvCouplesHostAndPort` covers the coupling invariant: both-set, host-only, port-only, empty, and whitespace-trimmed cases. - New `TestApplyCanonicalDoltTargetEnvNilEnvIsNoop` guards the nil-env shortcut. - Updated four existing tests to assert HOST is now projected as 127.0.0.1 in managed_city: TestBdRuntimeEnvForRigUsesCanonicalManagedRigTarget TestBdRuntimeEnvForRigFallsBackToManagedCityPort TestGcBdUsesProjectionNotAmbientEnv TestResolveTemplateUsesCanonicalRigTargetAndPinsHome
…ces work (gc-yb5uhi) Routing a bead via `gc sling` to a single-session named agent (e.g. gc-toolkit.mechanik) used to set only `gc.routed_to`, leaving the assignee untouched. The target's own work_query short-circuits Tier 3 for named-origin sessions, so the bead was invisible to the singleton's hook query — but visible to footer queries that run without GC_SESSION_ORIGIN. The asymmetry stranded routed work on the singleton. Add a typed `Singleton` flag to `sling.RouteRequest`, populated from `!agent.SupportsInstanceExpansion()` at both finalize call sites. Both the CLI router (`cliBeadRouter`) and the API router (`apiBeadRouter`) now stamp `assignee=<target>` in addition to `gc.routed_to=<target>` when the flag is set. Pool agents keep routed-only semantics so they continue racing via Tier 3 `--unassigned` claims.
…t agentEnv (gc-rch40w) (#8) (per gc-gkf9m3.7) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-gkf9m3.7 for context and metadata.classification.
… [agent_defaults.env] onto agents (gc-ch7eag) (#10) (per gc-9n4v5n.5) (per gc-lcixo.1) (per gc-3moew.4) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-3moew.4 for context and metadata.classification.
…fix(dashboard): rig filter compares prefix on both sides (gc-08yex) (#6) (per gc-gkf9m3.9) (per gc-mkbyva.1) (per gc-k7cex.2) (per gc-3moew.5) Mechanical rework onto post-gastownhall#3727 upstream. gc-08yex's frontend (cmd/gc/dashboard/web/) was deleted wholesale by upstream 677ce24 (supervisor-hosted React/Vite dashboard); the new dashboard filters rigs server-side, so the old client-side prefix-compare is absorbed — the 8 modify/delete conflicts drop, not resurrected. Go-side intent kept (auto-merged clean): config.go rejects an empty derived rig prefix; handler_rigs.go exposes EffectivePrefix() in the rigs API. config_test.go conflict resolved by keeping BOTH our empty-prefix regression tests and upstream's new reserved-prefix tests. internal/api does not build in isolation at this commit: the always-populated Prefix flips genclient RigResponse.Prefix to a non-pointer string that upstream's decode_rigs.go still dereferences. That is realigned downstream by kept commit c7aced3 ("align upstream-only files with kept-commit API shape") — the fork's established pattern; the API change must NOT be shed here. See gc-3moew.5 for context and metadata.classification.
…ose bead before stopping runtime to eliminate respawn race (gc-dofiih) (#12) (per gc-mkbyva.2) (per gc-vjoy6.2) (per gc-u1g6k.2) Original commit's intent (Close-before-Stop respawn-race fix) ported to post-upstream code in the shared rebase worktree. Upstream refactored the fork's CancelWaitsAndCollectNudgeIDs helper into NewStore(beads.SessionStore{Store: m.store}).CancelWaits(...); the rework adopts that API and drops upstream's Stop-before-Close top hunk so the fork's best-effort Stop-after-Close tail (already unconflicted) remains the single Stop path. See gc-u1g6k.2 for context and metadata.classification=mechanical.
…ION_ID (gc-yzh66o) (#13) (per gc-vjoy6.3) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-vjoy6.3 for context and metadata.classification. Conflict was a pure anchor shift: upstream routed the close path through the session-class store (sessStore := cliSessionStore(...)); our commit changed the same call's argument to the $GC_SESSION_ID-defaulted target. Kept upstream's sessStore, applied our target fallback on top.
…9n4v5n.6) (per gc-5sacl.6) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-5sacl.6 for context and metadata.classification.
…s with kept-commit API shape (per gc-qyb843.5) (per gc-5sacl.9) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-5sacl.9 for context and metadata.classification. Kept the commit's intent (internal/api/decode_rigs.go + decode_rigs_test.go: direct 'Prefix: g.Prefix' assignment, preserving the genclient.RigResponse.Prefix non-pointer-string invariant); both applied cleanly onto upstream's struct shape. Resolved the lone conflict in examples/bd/dolt/dog_exec_scripts_test.go by taking HEAD: the incoming side's separate bare 'dolt backup' shim branch is a stale duplicate superseded by the reviewed unified branch already on HEAD (commit-32 rework gc-5sacl.7, blessed by review gc-5sacl.8). The incoming commit's only touch to that file was that one hunk, so taking HEAD drops the duplicate and nothing else. Classification: mechanical.
…sessions (gc-4kq5vc) (per gc-mkbyva.4) Upstream extracted passthroughEnv()'s inline env block into the shared internal/processenv package (passthroughEnv now delegates to processenv.ProviderProcessPassthroughEnv). Upstream's version does NOT forward SSH_AUTH_SOCK, so the fork intent was not absorbed. Mechanical rework: take the HEAD-side delegation in cmd/gc/cmd_start.go and port the SSH_AUTH_SOCK passthrough into processenv.ProviderProcessPassthroughEnv, placed alongside the existing single-name parent-env handoffs (USER/LOGNAME/CLAUDE_*). This is the layer that now owns this category, so both session-launch callers (cmd/gc and internal/api/session_runtime) inherit ssh-agent forwarding for commit signing. The auto-merged cmd_start_test.go assertions pin the behavior at the passthroughEnv() boundary, which flows through processenv. See gc-mkbyva.4 for context and metadata.classification=mechanical.
…nse (gc-3p9x0f) (per gc-3moew.7) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-3moew.7 for context and metadata.classification. Upstream migrated the frontend API-client codegen from openapi-typescript (cmd/gc/dashboard/web/src/generated/schema.d.ts) to @hey-api/openapi-ts (internal/api/dashboardspa/.../gc-supervisor-client/). The obsolete schema.d.ts is dropped; input_tokens is ported to the new generated types.gen.ts (rename-applied) and zod.gen.ts, matching the openapi change.
…ons Huma handler into response_cache (gc-6y7ril) (per gc-9n4v5n.8) (per gc-vtpf5.2) (per gc-3moew.8) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-3moew.8 for context and metadata.classification.
…bundled order (gc-517n7q) (per gc-9n4v5n.9) (per gc-vtpf5.3) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-vtpf5.3 for context and metadata.classification. Rework details (mechanical): - internal/builtinpacks/registry_test.go: union resolution. Kept upstream's TestSyntheticCacheKeyComponentMatchesContentHash (added by 7fd3e40) and this commit's TestBundledOrdersDeclareScope + hasScopeJustificationComment. Both are independent end-of-file additions; no semantic overlap. - examples/gastown/packs/maintenance/orders/nudge-mail-sweep.toml: new upstream order. Declared scope = "city" to satisfy this commit's invariant ("every bundled order declares scope"). Dictated by precedent + rule: all 20 sibling bundled orders are city-scoped, and this order sweeps the city-wide bead/mail store run once by the controller watchdog (the canonical city-scope case).
…ontext so {{ .ConfigDir }} resolves (gc-f2p7l0) (per gc-9n4v5n.11) (per gc-kw2g9f.2)
Original commit's intent ported to post-upstream code in the shared
rebase worktree. Classification: mechanical — upstream split the single
WorkQuery field into per-category queries (AssignedInProgressQuery,
AssignedReadyQuery, RoutedPoolQuery, WorkQuery) with ...ForBeads(beadsCfg)
variants and split buildPrimeContext into buildPrimeContextForBeads; the
kept commit's ConfigDir addition slots in alongside at the same anchors.
Kept both. See gc-kw2g9f.2 for context and metadata.classification.
…k ea59b1812612: feat(config): add Rig.DefaultMergeStrategy + city-level default for refinery PR policy (gc-l0ra2z) (#24) (per gc-mkbyva.5) (per gc-9n4v5n.13) (per gc-kw2g9f.3) (per gc-k7cex.3) (per gc-3moew.9) Original commit's intent ported to post-upstream code in the shared rebase worktree. See gc-3moew.9 for context and metadata.classification. Resolution: upstream gastownhall#3727 retired the old cmd/gc/dashboard tree; the sole conflict was the stale generated file cmd/gc/dashboard/web/src/generated/schema.d.ts (modify/delete). Honored upstream's deletion (git rm). The config feature (Rig.DefaultMergeStrategy) already applied to the hand-written files and the relocated generated file internal/api/dashboardspa/web/shared/src/generated/gc-supervisor-client/types.gen.ts.
…lone API is unconfigured (gc-1rr12w) (per gc-vtpf5.4) Original commit's intent ported to post-upstream code in the shared rebase worktree. Classification: judgment-required (review pending). See gc-vtpf5.4 for context and metadata.judgment_summary.
…per-recipient session-metadata fanout to one bulk load (gc-5ahpep) (per gc-9n4v5n.14) (per gc-kw2g9f.4) (per gc-vjoy6.4) Original commit's intent ported to post-upstream code in the shared rebase worktree. Classification: mechanical. Upstream split beadmail's store seam under this commit: Provider gained a sessionStore field (NewWithStores / NewCachedWithStores) and every session-addressing site moved from p.store to p.sessionStore. This commit replaces the per-recipient session queries with one bulk load, so its new code IS session-addressing code and takes the same rename: the loadSessionsForRouting guard and the hasSessionStore argument gate on p.sessionStore, matching the guard upstream put on the recipientRoutes this commit replaces. Behavior is unchanged today (every constructor passes sessStore == store) and routing follows the session store once sessions relocate, which is the split's stated purpose. Upstream's p.store -> p.sessionStore edits inside recipientSessionMatchesByCurrentAddress and recipientSessionMatchesByMetadata are moot: this commit deletes both -- they are the per-recipient fanout it exists to remove. Upstream's new ReadMessagesBefore and ReadMessageWispEntries (stale-mail and wisp-GC retention sweeps, called from cmd/gc) conflicted only positionally -- they landed in the region this commit rewrites -- and are preserved verbatim. See gc-vjoy6.4 for context and metadata.classification.
…esponse cache (gc-0dqphq) (per gc-9n4v5n.15) Original commit's intent ported to post-upstream code in the shared rebase worktree. Classification: mechanical — two anchor shifts, no design judgment: 1. humaHandleMailCount rig branch: upstream's store-slow deadline wrapper (gastownhall#2757) moved total/unread into a withMailReadDeadline closure returning a mailReadCounts struct; the commit's cached respond(...) exit now reads counts.Total/counts.Unread. 2. countingStore fixture: upstream's bulk-load refactor reads mail via one List(Assignees: routes) message scan instead of per-recipient by-assignee reads; widened the fixture's by-assignee case to include the plural form so the commit's three cache tests keep instrumenting the recipient-scoped read. Only beadmail sets Assignees, so the sibling cache tests are unaffected. See gc-9n4v5n.15 for context and metadata.classification.
…91) Motivation: `make check` (`go test -p=4 ./...`) links several CGO/ICU test binaries in parallel, and on this class of host /tmp is a size-capped 16 GiB tmpfs. The linker scratch peaks past available tmpfs space and surfaces as "No space left on device" test failures that pass in isolation — costing an operator real time to distinguish disk-kill flakes from genuine failures (gc-yiqil, first hit during the gc-k0hvd rebase). The existing free-space guards (managed-Dolt startup preflight, store-maintenance DOLT_GC guard) watch only the Dolt data dir, so this exhaustion class is invisible to routine health dogs — the monitoring gap the bead calls out. Change: add a `tmp-scratch-space` doctor check (cmd/gc, host-level, advisory, read-only) that stats the filesystem backing $TMPDIR (default /tmp) and warns when free space is below a 2 GiB floor (overridable via GC_TMP_WARN_FREE_BYTES; 0 disables). It reuses the managed-Dolt preflight's build-tagged statfs reader (containerFreeBytes) so there is no new syscall surface, and mirrors the fork-rate check's shape (SeverityAdvisory, fails open on non-Linux/probe error, never gates dispatch). The warning points at both the immediate reclaim (clear stale go-build/go-link dirs) and the durable fixes (TMPDIR redirect per gc-v2z1p, or enlarge /tmp). Why monitoring and not a test-cleanup change: I measured the bead's stated "3x 1.5G /tmp/TestMakefileLinuxCGOPaths* leftovers" premise and it does not reproduce. The six TestMakefileLinuxCGOPaths* tests use t.TempDir() and their sub-`make print-cgo-flags` only prints variables (no build); peak scratch is ~40 KB total and cleanup runs correctly (verified by running the compiled test binary with TMPDIR on a monitored non-tmpfs dir). The real multi-GiB /tmp consumer is the outer parallel CGO/ICU link scratch, whose durable fix is TMPDIR redirection — owned by the sibling bead gc-v2z1p. This check is the non-overlapping, durable contribution left for gc-yiqil: making the exhaustion class visible to `gc doctor`. Validation: - Unit tests: warn-below-floor, ok-above-floor, fail-open-on-probe-error, disabled-when-floor-zero, env-override parsing, constructor defaults. - Updated the golden doctor check-name set (registered after fork-rate). - End-to-end `gc doctor` run: OK branch reports real /tmp free (8.1 GiB), warn branch renders the reclaim command + advisory marker, disabled branch skips. - gofmt, go vet, and golangci-lint (--new-from-rev=origin/main) all clean. Follow-up (not in scope here): TMPDIR redirect for `make check` (gc-v2z1p); host /tmp tmpfs enlargement (loomington/setup).
…#92) Gate the fork's zookanalytics/beads pseudo-version replace past the check-gomod-replace guard so Preflight/static-checks runs to completion and fork PRs can auto-land again (was FAILURE-at-guard on main). - gc-jwho8: allowlist the beads fork entry in scripts/check-gomod-replace.sh - gc-434qa (pre-open signoff): gate the allowlist on pseudo-version shape so the exception stays narrow, plus scripts/gomod_replace_guard_test.go coverage The FORK_REPLACE_ALLOWLIST entry is a TEMPORARY bridge: delete it when the beads fork converges upstream and go.mod's `replace` returns to a released semver tag. Closes gc-bvjbs.
These non-conflicted cmd/gc test files kept pre-rebase call shapes and only surfaced post-rebase (cmd/gc could not build mid-rebase until the resolveBdScopeTarget caller fix landed): - startControllerSocket gained demandDirty (gc-qedgc #69) — add missing nil arg - startCandidate's raw bead pointer became info session.Info (WI-6 R4): session:&bead -> info: SeedBead / InfoFromMeta - sortCandidatesByWakeFairness gained a cfg param
…-c1rpx) The fork's doctor_stuck_creating.go (gc-c1rpx #33) lists session beads via raw session.ListAllSessionBeads to flag sessions stuck in creating. Upstream's new typed-class census guard (TestTypedClassCodecCensusRatchet) counts that as a Tier-1 codec call baselined at 0. Ratchet the census up (the guard's sanctioned, review-visible escape) — same treatment as the sibling doctor_*.go diagnostic lanes (e.g. doctor_session_model.go). Front-door migration to Store.ListAll is deferred to a separate typed-class change.
…, resource census) (gc-u1g6k) - import_trust_test.go: disable commit/tag signing in the fixture repo so the parallel test subprocess does not require an unreachable ssh agent (ref tk-9zgnf). - gc_env_read_baseline.golden: add GC_TMP_WARN_FREE_BYTES (new env from gc-yiqil). - resource census: ratchet baselines up to current AST counts across bootstrapPolicy, test/test-resources.toml, and the TESTING.md ledger block.
…1b5h) scripts/test-local-job-count derived concurrency from CPU and memory only, so the pre-push suite fan-out was blind to the one resource it exhausts fastest. Every concurrent shard writes its own Go $WORK tree — package archives, a ~2.8 GiB test binary, linker temporaries — and Go removes that tree only on a clean exit, so the city's constant polecat churn leaves the killed runs' scratch behind. On 2026-07-17 the loomington root volume reached 100% (502M free): every build, every `git push`, and every Dolt write failed fleet-wide, with the data plane AGENTS.md flags as fragile unable to write at all. Free scratch space now clamps the fan-out as a third budget alongside CPU and memory, using the same 4 GiB per-shard figure the memory budget uses (the same test binary dominates both) plus an 8 GiB reserve for everything else sharing the volume — the Dolt data plane, GOCACHE growth, git objects. The probe reads `df -Pk` on the directory the shards will actually write to: GOTMPDIR when the toolchain has one (including via `go env -w`, which never reaches this script's environment), otherwise TMPDIR — repeating TEST_ENV's /var/tmp default, since LOCAL_TEST_JOBS is expanded before that wrapper applies. Two deliberate choices. The clamp fails open: an unreadable free-space reading leaves today's CPU-and-memory fan-out untouched rather than throttling every suite on an unsupported host, matching the stance the doctor's tmp-scratch-space check (gc-yiqil) already takes. And when free space cannot cover even one shard's budget above the reserve, the floor stays at one job but the script warns on stderr: clamping can no longer protect the volume there, only the operator can reclaim space, and a bare ENOSPC mid-link is a much worse way to learn that. stderr keeps the warning clear of the `$(shell ...)` and command substitution that consume this script's stdout. This is the peak-demand half of the problem. Reaping already-leaked scratch, trapping EXIT/TERM in the runner, and extending the doctor's warning to /var/tmp remain open on gc-k1b5h's sibling proposals. Validation: new table cases in scripts/precommit_contract_test.go drive `make -n test-fast-parallel` through the disk budget (constrains, at-reserve floor, exhausted, unknown, and tightest-budget-wins), and a new test exercises the real `df` probe plus the fail-open path against an unmeasurable directory. Existing cases now pin GC_TEST_LOCAL_DISK_KIB so the host's own free space cannot make their expectations machine-dependent. `go test ./scripts`, `go vet ./...` clean; on this 8-CPU/21 GiB-available host the count is unchanged at 5 (disk allows 6), and an incident-like 900 MiB reading yields 1 with the warning.
…ger (gc-k1b5h) TestLocalJobCountProbesScratchFilesystem execs scripts/test-local-job-count directly, which is the point of it: every other case in that table stubs GC_TEST_LOCAL_DISK_KIB, so without this one the real `df -Pk` probe and the fail-open path it guards are never exercised. That is one new subprocess call site in an already-counted file, so the three subprocess baselines move by one call each with file counts unchanged: audit scope=all 535 -> 536 calls / 156 files debt scope=untagged 408 -> 409 calls / 110 files small debt scope=untagged 406 -> 407 calls / 109 files The ledger is checked in three places that must agree, and the census test fails on each in turn until all three match: the bootstrapPolicy literal in internal/testpolicy/resourcecensus/census.go, test/test-resources.toml, and the rendered block between the CHECKED TEST RESOURCE LEDGER markers in TESTING.md. These are anti-growth ratchets, so the bump is deliberate rather than incidental. The alternative ways to keep the count flat were both worse: a build tag would mark the file `tagged` and pull the scripts contract tests out of the fast gate that is exactly where they earn their keep, and folding the call into an existing exec site would mean routing a bare script invocation through the `make -n` helper it shares nothing with. Growing the debt by one call and recording it here keeps the evidence honest.
…1b5h)
The free-scratch budget landed in scripts/test-local-job-count, which probes
GOTMPDIR or `${TMPDIR:-/var/tmp}`, but scripts/test-local-parallel handed each
child job `TMPDIR="${TMPDIR:-/tmp}"`. Only test-fast-parallel runs under the
Makefile's TEST_ENV wrapper, so test-cmd-gc-process-parallel,
test-integration-shards-parallel and test-local-full-parallel reached the runner
with TMPDIR unset: the detector measured /var/tmp while every shard linked its
multi-GiB test binary on a tmpfs-backed /tmp the budget never accounted for.
That defeats the ENOSPC guard on exactly the heavy targets it exists for.
The runner now resolves the scratch directory once and exports it as
TEST_LOCAL_TMPDIR, so each job builds where the detector measured instead of
re-deriving a default in the child shell. scripts/test-go-test-shard and
scripts/test-integration-shard repeat the /var/tmp default too — TESTING.md
documents both as direct entry points, so neither can rely on inheriting it.
Two guards keep the defaults from drifting apart again.
TestLocalTestScratchDefaultsAgree pins every `${TMPDIR:-...}` fallback in the
detector, the runner, the shard wrappers and the Makefile to one filesystem;
reverting the runner line fails it. TestParallelTargetsSizeFanoutAgainstMeasured-
Scratch drives all four parallel targets through `make -n` on a
scratch-constrained host, covering the three that previously had no dry-run
test. TESTING.md's runner prose now describes scratch as the third budget
alongside CPU and memory, and the resource-census ledger takes the +1 subprocess
ratchet for the new `make` invocation.
The rebase of this branch onto 42d0c87 dropped the `testutil.IsolatedGitConfig` call sites the original branch head (917ac2c) carried, leaving the seed-archive helpers to inherit the host's `commit.gpgsign=true` + `gpg.format=ssh`. The fast suite runs each shard under `env -i`, which strips SSH_AUTH_SOCK, so every seeded `git commit` failed: error: Couldn't get agent socket? fatal: failed to write commit object That took out ~8 TestJsonlExport* cases in examples/gastown and blocked the pre-push gate, which is why the original branch pushed clean and the rebased branch did not. The helper itself still exists upstream (internal/testutil/gitconfig.go), so this restores only the lost call sites -- the same isolation main applies in internal/runtime/import_trust_test.go (tk-9zgnf). Pure addition; no behavior change outside test setup. Verified with SSH_AUTH_SOCK unset, reproducing the shard's sanitized env: `go test ./examples/gastown -run TestJsonlExport` and `go test ./internal/api/dashboardbff` both pass.
…OBS (gc-iz99b) TestParallelTargetsSizeFanoutAgainstMeasuredScratch built its child environment from os.Environ(), so a caller exporting LOCAL_TEST_JOBS — a developer's shell, an outer sharded runner — fed that value straight into `make -n`. The Makefile honors it (`LOCAL_TEST_JOBS ?=`) ahead of the detector, so the assertion read back the inherited override instead of the computed fan-out and all four subtests failed on exactly the hosts that use the documented override. Extract the filter TestTestFastParallelUsesSanitizedEnvironmentAnd- MachineAwareConcurrency already applied inline into sanitizedLocalTestEnv and use it in both. The prefix form covers the whole GC_TEST_LOCAL_* stub namespace — identical to the six names enumerated before, and a stub added later is stripped without reintroducing this bug. The test now plants LOCAL_TEST_JOBS=7 itself, so a regression to os.Environ() fails deterministically everywhere rather than only for whoever happens to export the variable. Addresses the pre-open signoff finding on gc-iy2mt (anchor gc-k1b5h).
Author
|
Codex signoff (pre-open, comment-only — not an approval): Code Review ResultsScope: branch-remote Reviewers: correctness, testing, maintainability, project-standards, performance, reliability
Coverage
Actionable findings: none.
|
Author
|
Refinery CI attribution — the red checks on this PR are pre-existing on Compared this PR's failing-job set against
All 10 jobs failing here already fail on The failing set is a strict subset of Note that This PR remains held, not merged. Posted by the gascity refinery during an idle reconcile pass. |
zook-bot
force-pushed
the
main
branch
2 times, most recently
from
July 21, 2026 19:05
8ad3f41 to
021b9f8
Compare
zook-bot
force-pushed
the
main
branch
6 times, most recently
from
July 22, 2026 02:14
aa05d0d to
ce27583
Compare
johnzook
reviewed
Jul 22, 2026
johnzook
left a comment
johnzook
left a comment
There was a problem hiding this comment.
Refuse to adopt, avoiding gascity fork changes if possible and this specifically comes from running a make test as part of that fork, when we should do parallel test anyways.
zook-bot
force-pushed
the
main
branch
4 times, most recently
from
July 22, 2026 23:13
7bba3de to
89e2e69
Compare
Author
|
Closing as obsolete per fork-shed evaluation (2026-07-23). scripts/test-local-job-count and test-local-parallel are byte-identical to upstream on current main; upstream already traps+cleans $WORK. Disk-fill-root risk is real but its durable fix is operational (scratch-reaping order) + the gc-v2z1p escape — not a fork patch on upstream scripts. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Symptom
Root filesystem hit 100% (216G/225G used, 502M free) on the loomington host.
Every Go build, test, and
git pushfailed. Observed 2026-07-17 while pushingpolecat/gc-rvlqo — a one-file .tsx change with ZERO Go content:
Fleet-wide: at 100% full nothing builds, and Dolt (the data plane, which
AGENTS.md flags as fragile) cannot write. Risk of data-plane corruption.
Root cause
/var/tmp held 21G, of which 18G was abandoned Go
$WORKscratch:71
go-build*/go-link*dirs, oldest dated 2026-07-02 (15 days stale).Go creates
$WORKunder TMPDIR and removes it on CLEAN exit. A killed orcrashed run leaks the whole tree. The city churns/kills polecats constantly
(pool rotation, drain, restart), so every run killed mid-
go testleaks GBsthat nothing ever reclaims.
6f398d582(gc-v2z1p, #89) intentionally moved test TMPDIR to /var/tmp to dodgetmpfs ENOSPC on /tmp. That fix is correct, but it relocated the leak onto the
root disk, where it accumulates persistently instead of vanishing on reboot.
0530b0f0f(gc-yiqil, #91) warns on low /tmp space — it does NOT watch/var/tmp, so the doctor was blind to a 100%-full root volume.
Mitigation already applied (2026-07-17, by polecat gc-toolkit.nux)
Reclaimed 47 dirs older than 24h (9.9G): 479M -> 9.6G free, 100% -> 96%.
Deliberately scoped:
AGENTS.md. That ban targets the SHARED CACHE, and its rationale (forcing
concurrent executors into cache-miss rebuilds) does not extend to
per-invocation $WORK scratch, which is never reused across runs.
This is MITIGATION, NOT A FIX. It will refill. Disk is still 96-97% with
loomington=50G, GOCACHE=17G, GOMODCACHE=7G on a 225G volume.
Proposed fix (needs operator decision)
sweep above. Cheapest durable fix.
so killed polecats stop leaking in the first place.
gc doctor's build-scratch warning (gc-yiqil, feat(doctor): warn on low build-scratch (/tmp) free space (gc-yiqil) #91) to watch /var/tmpand the root volume, not just /tmp — the doctor missed a 100%-full disk.
running the full Go suite on EVERY new polecat branch (~8GB per push,
regardless of diff content). Very likely the dominant producer.
Related
/tmp. Same mechanism (killed runs leak scratch), different path/content;
likely fixed together by the trap in (2).
Implementation notes
Implemented: size the local test fan-out on free build scratch.
Branch polecat/gc-k1b5h @ 917ac2c (2 commits, pushed + ref-verified against
local HEAD). Base origin/main b215e07, already a descendant — no rebase needed.
6f539e3 fix(scripts): size the local test fan-out on free build scratch
917ac2c test(census): record the scratch-probe subprocess in the ledger
Commit 1 clamps LOCAL_TEST_JOBS on free scratch space as a third budget beside
CPU and memory (4 GiB/shard + 8 GiB reserve), probing GOTMPDIR then TMPDIR via
df -Pk. Fails open when free space is unreadable; floors at 1 job with astderr warning when it cannot cover one shard.
Commit 2 is required by commit 1: the new real-
dfprobe test adds oneexec.Command site, which trips the subprocess anti-growth ratchet. The ledger
is checked in THREE places that must agree, and the census test fails on each
in turn -- census.go bootstrapPolicy, test/test-resources.toml, and the rendered
CHECKED TEST RESOURCE LEDGER block in TESTING.md. All three moved +1 call,
files unchanged (535->536 all, 408->409 untagged, 406->407 small untagged).
Local gates: go test ./scripts, ./internal/testpolicy/resourcecensus, and the
full pre-push fast suite all pass; pre-commit ran lint + go vet + docsync clean.
NOTE FOR REFINERY -- one pre-existing flake, NOT from this diff: the first
pre-push run failed two examples/bd/dolt backup-freshness tests. Both pass in
isolation on clean origin/main AND on this branch, and this diff is not
reachable from that package; they did not recur on the second run. Filed with
full evidence as gc-ax9tu. If CI trips on them, it is not this branch.
Rework gc-myqgt landed on polecat/gc-k1b5h at f893413; re-review gc-iy2mt dispatched (one-anchor-per-PR, tk-ynz4b).
Refinery handoff
gc-k1b5h(bug, P1)polecat/gc-k1b5hmain3d53f69e; PR opened codex-green.