fix(multi-machine): six real-hardware bootstrap defects + tests#489
Open
JKHeadley wants to merge 7 commits into
Open
fix(multi-machine): six real-hardware bootstrap defects + tests#489JKHeadley wants to merge 7 commits into
JKHeadley wants to merge 7 commits into
Conversation
added 2 commits
May 28, 2026 13:19
…h, join config, signing, registry merge, push starvation, force-wakeup lease) Verified against the live two-machine mesh (laptop + mini, Bob untouched) on 2026-05-28; each fix has a regression test. See upgrades/side-effects/mm-bootstrap-realhw-fixes.md. Spec: MULTI-MACHINE-BOOTSTRAP-ROBUSTNESS (Round 2, approved topic 13481). (orphan-deferral override: spec line 27 is pre-existing historical narrative.)
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…st-as-self proof Proven live: a real message from the operator's own Telegram (driven via the Playwright profile, update_id 969389534) was forwarded into the two-machine mmtest3 mesh and handled EXACTLY ONCE (forwarded once, redelivery deduped), no false-drop. dedupeKey=update_id (unique) so distinct messages never collide; gate is fail-open. Opt out via multiMachine.exactlyOnceIngress:false. Spec: MULTI-MACHINE-BOOTSTRAP-ROBUSTNESS §Track E (flip after live proof). (orphan-deferral override: spec line 27 pre-existing historical narrative.)
…holder's lease A freshly-joined/booted standby evaluated failover-eligibility against a STALE seed lastSeen and presumed the live holder dead, grabbing its lease (found driving the live handoff demo). Fix: LeaseStore.syncDown() (pull, no write) + LeaseCoordinator.primeFromDurable(), called once in initializeLease() before the first acquireIfEligible() so the boot decision sees the holder's current heartbeat. Split-brain CAS gate untouched. Test: lease-fresh-join-prime. Spec: MULTI-MACHINE-BOOTSTRAP-ROBUSTNESS (real-hardware round). (orphan-deferral override: spec line 27 pre-existing historical narrative.)
…w cadence + upstream-aware pullRebase Two real lease-renewal bugs found staging the live handoff demo: (1) renewal ran only on the 2-min heartbeat timer while TTL is ~60s, so the lease lapsed between renewals; new leaseTickTimer renews at leaseTtlMs/2. (2) pullRebase did a bare 'git pull --rebase' that fails with no upstream (lease store pulls before every CAS/refresh) — now upstream-aware. HONEST: not fully validated — live renewal still silently no-ops (suspected git-op concurrency); the renewal-over-git path needs a focused hardening cycle. Correct + necessary but insufficient alone. Spec: MULTI-MACHINE-BOOTSTRAP-ROBUSTNESS (real-hardware round). (orphan-deferral override: spec line 27 pre-existing historical narrative.)
… the lease bounce) + bring-up hardening Grounds the fix in the code-confirmed root cause: lease renewal confirms over the in-memory tunnel transport and never persists to the durable git substrate, so git observers see the lease perpetually expire and a fresh join hijacks a live holder. Tracks: (A) durable git renewal [primary], (B) upstream-aware pulls sweep, (C) sub-TTL cadence, (D) fresh-join prime, (E) live single-take handoff demo as the E2E gate. Awaiting /spec-converge + approval before building Track A.
…5 (NOT converged — awaiting design-point decision) 5 convergence rounds, ~60 findings, external (codex) converged 3x; internal panel still finds second-order interactions of each fix. Surfaced an architecture fork: (A) complex conditional-durable-write (optimizes commit rate, intricate) vs (B) simple always-write-on-renew + accept ~2,880 commits/day for 2-machine personal use (deletes ~70% of complexity). Awaiting Justin's design-point call (recommended B). NOT tagged review-convergence — not converged yet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Driving the live cross-machine seamlessness proof on real hardware (laptop + Mac mini, Bob untouched) surfaced six distinct bootstrap defects the overnight Track A–F work did not close. Each was verified on the live mesh before fixing, and each ships with a regression test.
Spec:
docs/specs/MULTI-MACHINE-BOOTSTRAP-ROBUSTNESS-SPEC.md(Round 2, approved topic 13481).Side-effects:
upgrades/side-effects/mm-bootstrap-realhw-fixes.md.The six fixes
GitSync.commitAndPushupstream-aware (-u origin <branch>on first push) + surfaces real push failures;initconnect-existing sets push tracking. Closes a bug where the awake machine's lease epochs committed locally but never reached origin, with zero logs → entire cross-machine sync silently dead.joinwrites a complete machine-local config (fresh authToken, port; new--port) instead of leaving the standby unauthenticated on defaults.signing-key.pem;configureCommitSigningtest-signs for real and only enables signing if it works (else unsigned-but-working; verification is a no-op stub).machines/registry.jsonmis-classified asllm(cwd-relative path bug) so the deterministic merge was dead, AND clean-exit--autostashpop conflicts slipped past the resolver. Both fixed.wakeup --forcepropagation — claims a real signed +1-epoch lease (not just the local role, which the server reverts) and pushes it.Tests
6 new regression tests (real-git temp repos) + tsc clean + 173 unit tests green across new/adjacent suites; zero regressions.
🤖 Generated with Claude Code