Skip to content

fix(worker): guarantee & verify running code matches requested version on redeploy#144

Merged
nilsmechtel merged 3 commits into
mainfrom
fix/deploy-stale-source
Jul 21, 2026
Merged

fix(worker): guarantee & verify running code matches requested version on redeploy#144
nilsmechtel merged 3 commits into
mainfrom
fix/deploy-stale-source

Conversation

@nilsmechtel

Copy link
Copy Markdown
Collaborator

Problem

A deploy_app update could report the new version RUNNING/HEALTHY while replicas kept executing the previous version's code. Three interacting causes:

  1. In-place update reuses replicas. On an update, the worker cancelled the async deploy task and relied on same-named serve.run to overwrite. Ray Serve performs a rolling update, but at num_replicas=1 with no surge headroom (e.g. a single time-sliced GPU) the new replica stays PENDING and the old replica keeps serving stale in-memory code. On CPU (free capacity) the same bump reloads; on a saturated GPU it silently does not.
  2. Status echoes the request. get_app_status reported the requested version and never asked replicas what they had actually loaded, so a stale deployment read as healthy.
  3. Version-string source cache key. _ensure_source short-circuited its download when the on-disk .version marker equalled the requested version string, and the source directory is keyed on application_id only. A different artifact (or a reused application_id) sharing a version string served the first one's source to a brand-new replica.

Solution

  • Force fresh replicas on a content change (fix 1). When an update changes the version or the artifact, serve.delete the application before rebuilding so the slot frees and every replica is recreated from the new source. is_deployed is cleared across the delete→rebuild window so the monitor loop does not fire a redundant redeploy.
  • Verify what replicas actually loaded (fix 2). @bioengine.app now injects a bioengine_runtime_version method on every deployment that returns its boot-time BIOENGINE_ARTIFACT_* identity. ProxyDeployment.get_running_version reads it from the entry replica; the worker surfaces running_version / version_verified in get_app_status, and the monitor loop deletes a RUNNING-but-stale application so the next tick redeploys fresh.
  • Content-identity source cache key (fix 3). _ensure_source keys its marker on artifact_id@version instead of the bare version string, computed identically in the introspect and replica branches so the same content still short-circuits the flock download.

Behavioural changes

  • A deploy_app update whose version or artifact changed now briefly deregisters the app's Hypha service while replicas are recreated (previously an in-place rolling update). Updates that change neither are unaffected.
  • get_app_status gains running_version and version_verified fields.
  • Every user deployment gains a reserved bioengine_runtime_version method (name added to the reserved list).

Test plan

  • Worker-free unit checks: _ensure_source short-circuits on an identical artifact_id@version and re-materialises when the artifact differs at the same version string; _make_runtime_version returns the booted identity from env; @bioengine.app injects and reserves bioengine_runtime_version.
  • Live-cluster validation (single-machine + external-cluster) via the dev-image workflow before this PR is marked ready: deploy an app, bump its version, and confirm the served behaviour and version_verified reflect the new code on a fresh replica.

Files

  • bioengine/apps/manager.pyserve.delete on content-changing updates; _get_running_version helper; monitor mismatch self-heal; running_version/version_verified in status.
  • bioengine/apps/proxy_deployment.pyget_running_version accessor.
  • bioengine/_app/decorators.py, bioengine/_app/mixin.py — inject/reserve bioengine_runtime_version.
  • bioengine/_app/replica_init.pyartifact_id@version source cache key.

nilsmechtel and others added 3 commits July 21, 2026 10:45
…redeploy

A deploy_app update could report the new version HEALTHY while replicas kept
running old code. Three interacting causes, confirmed empirically:

- On update, deploy_app never tore the old app down; it relied on same-named
  serve.run to overwrite. Ray Serve does an in-place rolling update, but at
  num_replicas=1 with no surge headroom (e.g. a single time-sliced GPU) the
  new replica stays PENDING and the old replica keeps serving stale in-memory
  code. Fix A: on an update whose version or artifact_id changed, serve.delete
  before the rebuild so the slot frees and every replica is recreated.
- get_app_status echoed the requested version, never asking replicas what they
  loaded. Fix C: inject a bioengine_runtime_version method on every user
  deployment (returns its boot-time BIOENGINE_ARTIFACT_* env), expose it via a
  ProxyDeployment.get_running_version accessor, surface running_version /
  version_verified in get_app_status, and have the monitor loop serve.delete a
  RUNNING-but-stale replica so the next tick redeploys fresh.
- _ensure_source keyed its source cache on the bare version string, so a
  different artifact (or reused application_id) sharing a version string served
  the first one's stale source on a fresh replica. Fix B: key the marker on
  artifact_id@version.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_ensure_source's content-identity cache key and bioengine_runtime_version
both read BIOENGINE_ARTIFACT_ID, but the builder only ever set
HYPHA_ARTIFACT_ID — so artifact_id was empty and the identity collapsed
back to the bare version string. Set BIOENGINE_ARTIFACT_ID alongside
HYPHA_ARTIFACT_ID so the identity is artifact-scoped (this also revives the
BIOENGINE_LOCAL_ARTIFACT_PATH dev branch, which read the same unset var).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nilsmechtel
nilsmechtel marked this pull request as ready for review July 21, 2026 10:32
@nilsmechtel
nilsmechtel merged commit fc2797d into main Jul 21, 2026
2 checks passed
@nilsmechtel
nilsmechtel deleted the fix/deploy-stale-source branch July 21, 2026 12:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant