feat(orch): live-upgrade envd inside a running sandbox at resume

[kalyazin]

feat(orch): live-upgrade envd inside a running sandbox at resume

Why

envd is the in-VM daemon every sandbox depends on, but its version is baked into
the snapshot at build time. A long-lived paused sandbox therefore resumes on
whatever envd it was built with, potentially many releases behind — there is no
way to ship an envd fix or behavioral change to the existing fleet of snapshots
short of rebuilding every template. We want to roll a new envd to running
sandboxes the same way we roll Firecracker versions: gated by a LaunchDarkly
flag, ramped by cohort, and without disturbing the customer's workload.

The hard part is doing it without a visible interruption: the swap has to keep
arbitrary customer processes — and their stdio/PTYs, timeouts, filesystem
watchers, and recently-observed exit codes — alive across the binary change.

What

Two halves: an in-guest handover mechanism in envd, and an orchestrator-side
resume-time trigger driven by a flag. They are independent — the envd half is
inert until something calls POST /upgrade.

In-guest handover (envd)

A new authenticated POST /upgrade endpoint performs a same-PID
syscall.Exec into a new envd binary and re-adopts the running workload, so
customer processes never notice the daemon changed underneath them.

• Delivery + trigger in one call. The new binary is streamed in the request
  body, written inside the guest (default /usr/bin/envd.next), then swapped.
  This uses the token-authenticated /upgrade path rather than the
  unauthenticated /files copy, which a live post-/init sandbox rejects.
• The swap. envd freezes the workload cgroups, serializes the process table
  to a tmpfs handover blob, relocates the carried fds and syscall.Execs under
  ForkLock. On success it never responds — the process image is replaced. The
  freeze lock is held across the whole handover (freeze → serialize → execve,
  via FreezeHold), not just the freeze sweep, so a concurrent /init or
  /unfreeze thaw can't slip in and unfreeze the workload mid-serialization; on
  execve it evaporates with the process image, and on any failure it's released
  before the thaw. The port scanner is left running: holding ForkLock across
  the fd relocation already serializes it against the scanner's fd/socat
  creation, so there's no CLOEXEC race to quiesce.
• Guarded fd relocation. The carried fds are dup3'd to fixed high numbers
  (so the fresh runtime's early startup can't have grabbed them) under
  ForkLock, and each target is checked free first — a collision aborts the
  upgrade (closing what it already relocated) instead of dup3 silently
  clobbering a live fd, leaving the old envd running.
• What the handover carries (tmpfs /run/e2b/envd-handover.json, so it
  never touches the rootfs diff): per-process pid / tag / cgroup / config, the
  stdio + PTY fds (dup'd with CLOEXEC cleared so they survive execve), the
  terminal-event retention cache (exit codes of processes that ended just
  before the swap), the filesystem watchers, and each process's remaining
  timeout.
• The incoming side re-adopts each process from its inherited fds, resumes
  streaming its output, and re-arms its timeout — recording a fresh deadline so a
  further chained upgrade re-carries the remaining timeout. It restores the
  retention cache (skipping any PID already re-adopted as live, so a reused PID
  resolves to the live process, not a stale exit) and re-arms the filesystem
  watchers before thawing the workload, so the workload stays frozen until
  the watches are live and no event is dropped in the gap. Re-adopted processes
  are reaped via pidfd + a pid-specific wait4 — never wait4(-1), which
  would steal os/exec's post-upgrade children — and the reaper retains a
  terminal event synchronously, before closing its event channel, so a late
  reconnect always finds the exit in the cache rather than racing the retain.
• Failure handling — the sandbox is never left worse off. On an execve
  failure the outgoing envd closes the fds it relocated (CLOEXEC-cleared — else
  they'd leak into the still-running old envd and its future children), thaws the
  workload it froze, and keeps running the old binary. The incoming resume path always thaws via a top-level defer
  (success, error, or panic) and is recover-wrapped, so a malformed blob can't
  crash envd into a systemd restart that orphans the workload. Degraded at
  worst, never hung or dead.
• One shared freeze lock. The freeze/unfreeze sweep and its serializing lock
  are factored into a single cgroups.WorkloadFreezer shared by the HTTP API
  (/freeze, /unfreeze, the /init deferred thaw) and the upgrade, so the
  upgrade's freeze can no longer interleave with the resume thaw and strand the
  workload frozen.

envd is bumped to 0.6.11, the first upgrade-aware version.

Resume-time trigger + gate (orchestrator + shared)

On resume (Create with snapshot=true, and the checkpoint-resume path),
maybeUpgradeEnvd decides against the live running envd version, delivers,
and confirms — best-effort, recover-wrapped, and bounded so it can never disrupt
resume.

• Live version, not built-with. envd advertises its running version via an
  X-Envd-Version response header on /init; the orchestrator captures it off
  the resume-path /init it already makes (no extra round-trip) and keys the
  decision, gate, from_version, and success check on it. The template
  built-with never changes across a live upgrade, so keying on it would
  re-trigger the handover on every resume.
• Flag-driven target resolution. ResolveEnvdUpgrade (shared, the resume
  analog of ResolveFirecrackerVersion) reads the envd-upgrade-target flag and
  returns the target binary's local path + version, or "" when the target is
  not strictly newer than the live version — so an already-upgraded sandbox
  (live == target) is a true no-op, skipped with no delivery or handover, and a
  downgrade is refused. Target binaries live on the node-local read-only
  /fc-envd gcsfuse mount.
• Version gate. The running envd must already speak /upgrade, so the
  trigger skips a live version below MinEnvdVersionForUpgrade (0.6.11) and
  counts it (gated{reason=old_envd}).
• Delivery + confirmation. Sandbox.CallEnvdUpgrade streams the binary over
  the authenticated /upgrade body to a fixed guest path; envd reading the body
  then exec'ing without replying (transport drop) is the expected path. After
  WaitForEnvd (which re-reads the version off the new envd's /init), success
  is confirmed by the running version actually equalling the target — a transport
  quirk can't mislabel the outcome. Only unambiguous never-reached-envd errors
  (connection refused / dial) are treated as delivery failures.

Flags / knobs:

Flag	Values	Default	Meaning
envd-upgrade-target (LD string)	off | promoted | <git-sha>	off	off: no upgrade. promoted: track node-local HOST_ENVD_PATH, upgrade when strictly newer than the live version. <sha>: pin /fc-envd/envd.<sha>.
ENVD_UPGRADE_TARGET (env fallback)	same	off	Overrides the flag fallback where there is no LD (dev), mirroring DEFAULT_FIRECRACKER_VERSION.
MinEnvdVersionForUpgrade (const)	—	0.6.11	Below this the running envd lacks /upgrade; trigger skips.

Rollout metrics

envd has no metrics pipeline, so its side reports via Loki summary logs
(handover_resumed{procs,retained}, watchers_rearmed{...}). The orchestrator
side adds:

• orchestrator.envd.upgrade.attempts{result,from_version,to_version} —
  result ∈ {success, delivery_failed, not_ready, version_mismatch, panic};
  from_version is the live running version. success is recorded only when
  the running version actually flips to to_version (not inferred from the
  transport outcome). success/total is the rollout success rate; the common
  per-resume no-op (flag off / already on target) resolves to "" and is
  deliberately not counted.
• orchestrator.envd.upgrade.duration{result} — wall-time of delivery + trigger
  • WaitForEnvd, i.e. the overhead added to the resume.
• orchestrator.envd.upgrade.gated{reason=old_envd} — targeted resumes skipped
  by the version gate (watch during a ramp).
• an envd.upgraded bool label on orchestrator.sandbox.create.duration, and an
  envd-upgrade Tempo child span.

The post-upgrade readiness re-check re-runs /init (to re-read the running
version) but is guarded to record the per-start resume KPIs — the envd-init
duration histogram + call counter and StartedAt — once per start, so a
successful upgrade doesn't double-count the init or push StartedAt (and thus
execution_time) to a later, wrong timestamp.

# rollout success rate
sum(rate(orchestrator_envd_upgrade_attempts_total{result="success"}[5m]))
  / sum(rate(orchestrator_envd_upgrade_attempts_total[5m]))

Validation

• Unit tests (root-free, in CI): a Connect after a gap-exit returns the
  retained Start+End by pid/tag (unknown → NotFound); a live PID is never served a
  stale cached exit and a reused PID keeps its successor; a stale retention timer
  can't evict a newer entry; a re-adopted process whose pidfd_open fails still
  emits a terminal event (no orphan); a watcher exported then imported keeps its
  id, delivers post-handover events, and preserves pending events;
  ResumeFromHandover always thaws on a malformed blob and on no blob; Upgrade
  refuses a target other than the fixed path; the resolver is table-driven over
  off/unset/promoted/sha × newer/same/older/missing (downgrade refused) without a
  LaunchDarkly client; /init advertises X-Envd-Version; the delivery-error
  classifier flags only never-reached-envd; the three metrics have
  description+unit entries and construct.
• Dev cluster, end-to-end (exact branch binaries; re-validated 2026-07-23 on
  the final code): this branch's envd was rebuilt at 0.6.11 (baked into the
  template) and 0.6.12 (staged as the promoted target), and this branch's
  orchestrator (0.2.0-c21705aeb) deployed to the dev client nodes. Validated
  against a live workload — a stdout-writing + tmpfs-appending background counter,
  so a broken fd-carry would SIGPIPE-kill it and line growth proves it kept
  running:
  • upgrade fires + workload survives: flag → target ⇒ resume auto-upgrades
    0.6.11→0.6.12; the running envd becomes /usr/bin/envd.next --resume-handover
    and the counter process keeps the same PID and keeps writing across the
    swap (7→20 lines) — exercising stdio fd inheritance and pidfd reaping end
    to end. The orchestrator confirms by ground truth:
    envd_upgrade_attempts_total{result="success",from_version="0.6.11",to_version="0.6.12"} = 1.
  • flag off ⇒ no upgrade: plain resume, workload preserved on 0.6.11, no
    self-upgrade in the journal.
  • gap-exit retention: a process that exits during the handover gap has
    its exit code (42) recovered by a late Connect via the carried retention
    cache.
    The rollout metrics are emitted on the same path
    (attempts{result,from,to}, duration, gated, create.duration{envd_upgraded}),
    observed on dev Prometheus with a per-upgrade duration of ≈ 345–440 ms.
• Note: the 2026-07-23 re-run above exercised the final binaries (all the
  review-driven changes: live-version decision, version-confirmed success, the
  shared freeze lock held across the handover, the freeze/serialize/exec path, and
  the once-per-start resume-KPI guard on the post-upgrade readiness re-check). The
  remaining review changes that only manifest on failure/edge paths — reaper
  retaining before closing its channel, watcher re-arm before thaw, chained-upgrade
  deadline carry, exec-fail fd cleanup, and the PID-reuse / retain-timer /
  reaper-orphan / duplicate-forwarder fixes — are covered by the unit tests above;
  they don't change the success path the e2e validated.

Key decisions

• Same-PID syscall.Exec, not restart. A fresh process would be reparented
  by systemd (Type=simple), orphaning the workload; keeping the PID means envd
  stays the workload's parent and the inherited fds/PTYs stay valid.
• pidfd + pid-specific wait4, never wait4(-1). A blanket reaper would
  steal children that os/exec spawns after the upgrade; targeted reaping keeps
  both the re-adopted and the new processes correct.
• Trigger at resume, gated + best-effort. Resume is the natural quiescent
  window and gives free cohort ramping (the resume-site LD context already
  carries envd-version/team/template). The whole path is recover-wrapped,
  version-gated, and bounded, so a failed or slow upgrade degrades to
  upgraded=false and the sandbox resumes on the old envd. Caveat: the trigger
  is currently synchronous on the resume-critical path, so on pathologically slow
  resumes (large cold-fault working sets) it can add up to its timeout budget to
  the tail; moving it off the critical path is a candidate follow-up.
• Decide and confirm on the live version, read for free off /init. The
  orchestrator can't cheaply know the running envd version, but envd can report
  it — so envd advertises pkg.Version via an X-Envd-Version header on /init
  (which the resume path already calls) and the orchestrator keys the decision,
  from_version, idempotency, and the success check on it. Re-resumes of an
  upgraded sandbox become true no-ops and success is ground-truth-accurate, with
  no new endpoint and no extra round-trip. Version comparison still relies on
  packages/envd/pkg/version.go being bumped per behavioral change (repo rule);
  the resolver refuses any target that isn't strictly newer.
• Fixed exec target. envd writes/execs the delivered binary only at a fixed
  path and refuses any other, so an authenticated-but-malformed /upgrade can't
  turn the same-PID exec into arbitrary code execution.
• getVersion injected into the shared resolver so packages/shared gains
  no dependency on the orchestrator's envd build package.
• Handover blob on tmpfs, additive-fields-only JSON, so it never enters the
  rootfs diff and an outgoing/incoming envd built at different versions stay
  compatible.
• Rollout is off by default (envd-upgrade-target=off, env fallback off);
  dev has no LaunchDarkly so the feature is inert there unless
  ENVD_UPGRADE_TARGET is set.

🤖 Generated with Claude Code

[cursor]

PR Summary

High Risk
Same-PID root exec, pre-/init auth windows, and complex freeze/handover concurrency directly affect sandbox security and workload correctness; failures can strand frozen guests or serve stale exit codes if edge cases slip through.

Overview
This PR adds live envd self-upgrade so a running sandbox can swap to a newer envd binary without restarting customer workloads. An authenticated POST /upgrade streams the new binary, freezes workload cgroups under a shared WorkloadFreezer, snapshots state into a tmpfs protobuf handover (live processes with carried stdio/PTY fds, terminal-event retention, filesystem watchers, NFS mount ledger, socat port-forwards), and syscall.Execs into the new image with --resume-handover. The incoming envd re-adopts processes via pidfd-specific reaping, re-arms watchers before thaw, keeps the workload frozen until post-upgrade /init restores the access token, and reports X-Envd-Version and X-Envd-Handover on /init. Authorization is tightened so a live-upgraded envd fails closed before /init (notably blocking unauthenticated /files), initialized flips only after the token is restored, and a timed fallback thaw exists if /init never arrives. Connect gains retained-exit serving and PID/tag reuse guards; envd version bumps to 0.6.12 with architecture docs updated.

^{Reviewed by Cursor Bugbot for commit a537fc8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[codecov]

Codecov Report

❌ Patch coverage is 37.00528% with 955 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
packages/envd/internal/services/process/upgrade.go	31.34%	214 Missing and 5 partials ⚠️
packages/orchestrator/pkg/server/sandboxes.go	0.00%	201 Missing ⚠️
packages/envd/main.go	0.00%	157 Missing ⚠️
packages/orchestrator/pkg/sandbox/envd.go	17.64%	84 Missing ⚠️
.../envd/internal/services/process/handler/readopt.go	59.73%	53 Missing and 7 partials ⚠️
.../envd/internal/services/process/handler/handler.go	20.00%	48 Missing ⚠️
packages/envd/internal/services/cgroups/freeze.go	40.42%	23 Missing and 5 partials ⚠️
packages/orchestrator/pkg/sandbox/sandbox.go	0.00%	25 Missing ⚠️
packages/orchestrator/pkg/server/main.go	0.00%	24 Missing ⚠️
packages/envd/internal/api/auth.go	0.00%	22 Missing ⚠️
... and 14 more

📢 Thoughts on this report? Let us know!

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e29fe6ffc7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Detach post-upgrade initialization from request cancellation

When the resume request is canceled after envd has read the binary but before the client observes the connection reset, the server-side handler can still complete execve, while this branch reports the exec as unconfirmed. The subsequent WaitForEnvd uses the already-canceled context, so maybeUpgradeEnvd treats the failure as recoverable and promotes the sandbox; however, the replacement envd never receives /init, remains behind the pre-init authorization gate, and the fallback timer only thaws the workload rather than restoring authentication. Finish the post-trigger readiness check with a detached, bounded context or otherwise ensure this ambiguous cancellation cannot publish an uninitialized sandbox.

Useful? React with 👍 / 👎.

[kalyazin]

Agreed. After the trigger, the readiness /init (WaitForEnvd) ran on the cancellable resume context, so an ambiguous cancellation in that window — where the execve may already have happened — would skip /init yet still return recoverably, leaving a promoted-but-uninitialized sandbox (the exec'd envd never gets its auth/env restored).

Fixed by running the readiness /init on a detached but still-bounded context (context.WithoutCancel(upCtx), keeping readyTimeout): by this point the exec is a fait accompli, so /init is always driven to completion, and the version confirmation below still distinguishes an actual exec from an untouched old envd. Folded into the orch-trigger commit (now b910b68ad).

[chatgpt-codex-connector]

Quiesce port-forward creation through exec

After this snapshot returns, ExportForwards releases the forwarder mutex before Upgrade acquires syscall.ForkLock, so envd's still-running scanner can start and register a new socat in between. That socat survives the same-PID exec but is absent from the handover; the incoming scanner then attempts a duplicate bind, and startPortForwarding leaves the failed map entry in place, preventing later scans from repairing the forward. Hold a handover gate across forwarding export through exec so no socat can be created after the snapshot.

Useful? React with 👍 / 👎.

[kalyazin]

Right — the freeze quiesces the workload but not envd's own port scanner, so between ExportForwards releasing the forwarder mutex and the execve the scan loop could spawn a socat that is then orphaned by the swap (never carried, never re-adopted) or duplicates a port the new envd re-adopts, contending on the same bind address.

Fixed by holding the forwarder mutex from the snapshot through the execve: ExportForwardsHold returns the snapshot plus a release func, and doUpgrade holds it across Upgrade (released only on the failure path; a successful execve drops the held lock with the old process). Folded into the mounts/forwards commit (now 7aa45b03f).

[chatgpt-codex-connector]

Serialize watcher event drains with the handover

If GetWatcherEvents runs after this copy but before envd executes the replacement image, it can return these pending events to the caller and clear the old watcher's queue. The handover still imports the copied slice, so the same events are returned again after upgrade; freezing workload cgroups prevents new filesystem mutations but does not stop concurrent envd RPC handlers. Gate event draining from snapshot through exec, not only watcher membership changes, to preserve consume-once behavior.

Useful? React with 👍 / 👎.

[kalyazin]

Correct — ExportWatchers snapshots each watcher's pending events under watchersMu, but GetWatcherEvents drained them under the per-watcher lock only, so a concurrent drain in the export→execve window could consume events the snapshot already carried, and the re-armed watcher would re-deliver them after the swap (double-delivery).

Fixed symmetrically to the forwards: ExportWatchersHold holds watchersMu across the execve, and GetWatcherEvents now takes watchersMu around the drain, so no drain can interleave with the snapshot. This is defense-in-depth for the resume-scoped trigger (no client is connected during resume, so GetWatcherEvents cannot fire in that window), but it closes the race for any future non-resume trigger. ExportWatchersHold folded into the handover-mechanism commit (now e4a21cea5); the doUpgrade wiring into 7aa45b03f.

[cursor]

Stale comment

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a768144. Configure here.}