fix(worker): guarantee & verify running code matches requested version on redeploy#144
Merged
Conversation
…redeploy A deploy_app update could report the new version HEALTHY while replicas kept running old code. Three interacting causes, confirmed empirically: - On update, deploy_app never tore the old app down; it relied on same-named serve.run to overwrite. Ray Serve does an in-place rolling update, but at num_replicas=1 with no surge headroom (e.g. a single time-sliced GPU) the new replica stays PENDING and the old replica keeps serving stale in-memory code. Fix A: on an update whose version or artifact_id changed, serve.delete before the rebuild so the slot frees and every replica is recreated. - get_app_status echoed the requested version, never asking replicas what they loaded. Fix C: inject a bioengine_runtime_version method on every user deployment (returns its boot-time BIOENGINE_ARTIFACT_* env), expose it via a ProxyDeployment.get_running_version accessor, surface running_version / version_verified in get_app_status, and have the monitor loop serve.delete a RUNNING-but-stale replica so the next tick redeploys fresh. - _ensure_source keyed its source cache on the bare version string, so a different artifact (or reused application_id) sharing a version string served the first one's stale source on a fresh replica. Fix B: key the marker on artifact_id@version. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_ensure_source's content-identity cache key and bioengine_runtime_version both read BIOENGINE_ARTIFACT_ID, but the builder only ever set HYPHA_ARTIFACT_ID — so artifact_id was empty and the identity collapsed back to the bare version string. Set BIOENGINE_ARTIFACT_ID alongside HYPHA_ARTIFACT_ID so the identity is artifact-scoped (this also revives the BIOENGINE_LOCAL_ARTIFACT_PATH dev branch, which read the same unset var). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nilsmechtel
marked this pull request as ready for review
July 21, 2026 10:32
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A
deploy_appupdate could report the new versionRUNNING/HEALTHYwhile replicas kept executing the previous version's code. Three interacting causes:serve.runto overwrite. Ray Serve performs a rolling update, but atnum_replicas=1with no surge headroom (e.g. a single time-sliced GPU) the new replica staysPENDINGand the old replica keeps serving stale in-memory code. On CPU (free capacity) the same bump reloads; on a saturated GPU it silently does not.get_app_statusreported the requested version and never asked replicas what they had actually loaded, so a stale deployment read as healthy._ensure_sourceshort-circuited its download when the on-disk.versionmarker equalled the requested version string, and the source directory is keyed onapplication_idonly. A different artifact (or a reusedapplication_id) sharing a version string served the first one's source to a brand-new replica.Solution
serve.deletethe application before rebuilding so the slot frees and every replica is recreated from the new source.is_deployedis cleared across the delete→rebuild window so the monitor loop does not fire a redundant redeploy.@bioengine.appnow injects abioengine_runtime_versionmethod on every deployment that returns its boot-timeBIOENGINE_ARTIFACT_*identity.ProxyDeployment.get_running_versionreads it from the entry replica; the worker surfacesrunning_version/version_verifiedinget_app_status, and the monitor loop deletes aRUNNING-but-stale application so the next tick redeploys fresh._ensure_sourcekeys its marker onartifact_id@versioninstead of the bare version string, computed identically in the introspect and replica branches so the same content still short-circuits the flock download.Behavioural changes
deploy_appupdate whose version or artifact changed now briefly deregisters the app's Hypha service while replicas are recreated (previously an in-place rolling update). Updates that change neither are unaffected.get_app_statusgainsrunning_versionandversion_verifiedfields.bioengine_runtime_versionmethod (name added to the reserved list).Test plan
_ensure_sourceshort-circuits on an identicalartifact_id@versionand re-materialises when the artifact differs at the same version string;_make_runtime_versionreturns the booted identity from env;@bioengine.appinjects and reservesbioengine_runtime_version.version_verifiedreflect the new code on a fresh replica.Files
bioengine/apps/manager.py—serve.deleteon content-changing updates;_get_running_versionhelper; monitor mismatch self-heal;running_version/version_verifiedin status.bioengine/apps/proxy_deployment.py—get_running_versionaccessor.bioengine/_app/decorators.py,bioengine/_app/mixin.py— inject/reservebioengine_runtime_version.bioengine/_app/replica_init.py—artifact_id@versionsource cache key.