fix(worker): always re-fetch app source in the introspect task#146
Merged
Conversation
_ensure_source skipped its download when the .version marker matched artifact_id@version. That's unsafe for the introspect task's Hypha fetch: a version string is not a content identity — a version can be re-staged or an artifact deleted+recreated with new code under the same version — so a redeploy served stale source from disk, and neither the content-addressed package nor max_calls could recover it (the source itself was stale). Split the skip by branch: the replica/build path (BIOENGINE_APP_SOURCE_URI, a content hash) keeps the identity skip — the introspect for the deploy has already materialised the current source, so on a shared FS the build and replicas reuse it. The introspect path (Hypha download) now always re-fetches, so a redeploy reflects the current artifact content. One extra download per deploy; the content-addressed branch still dedupes for replicas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The worker applies deploy-time resource overrides — notably disable_gpu → num_gpus=0 — onto spec['classes'][cid]['ray_actor_options']. But bind builds each deployment's ray_actor_options from the *class's* ray_actor_options (`_with_pkg` reads `cls.ray_actor_options`), so the spec override never reached the replica: disable_gpu reduced the resource accounting but the actor still requested its original num_gpus. A type-hint- composed GPU deployment (e.g. model-runner's RuntimeApp) therefore kept requesting a GPU under disable_gpu=True and never scheduled on a CPU-only target. Re-apply the spec's num_cpus/num_gpus/memory onto opts in _with_pkg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Adds a second, related deploy-correctness fix: |
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nilsmechtel
marked this pull request as ready for review
July 21, 2026 16:05
nilsmechtel
marked this pull request as draft
July 21, 2026 18:25
…st_modified signature) Always re-fetching the source in the introspect (previous commit) is correct but re-downloads the whole artifact zip every deploy — wasteful for artifacts that carry heavy files. Gate the download on a content signature: hash the artifact file listing (path + size + last_modified) over the files that land in source/ (same exclude rules), fetched from the cheap files-listing endpoint. Unchanged signature → skip the download and keep the cached source; changed or unavailable → re-download + re-extract (which also drops files deleted from the artifact). The content-addressed Ray-GCS branch (replicas) is unchanged and still dedupes cross-node distribution. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nilsmechtel
marked this pull request as ready for review
July 21, 2026 18:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After #144/#145 a redeploy could still run old code when a version's content changed without the version string changing (a re-staged version, or an artifact deleted+recreated at the same version).
_ensure_sourceskipped its download whenever the on-disk.versionmarker equalledartifact_id@version, so the introspect task never re-fetched the current artifact —app_dir/sourcestayed stale, the introspect packaged that stale source, and the replicas (andmax_calls=1fresh build) faithfully ran it. Reproduced on a persistent Ray cluster: deployv0.1.0(content A) → recreate the artifact atv0.1.0with content B → redeploy → still served A.Solution
Split the skip by branch:
BIOENGINE_APP_SOURCE_URIset — a content hash): keep theartifact_id@versionskip. The introspect for this deploy has already materialised the current source, so on a shared FS the build and replicas reuse it (and the flock still serialises concurrent same-node replicas).create-zip-filedownload): always re-fetch. A version string is not a content identity, so this authoritative per-deploy fetch must reflect the current artifact. One extra download per deploy; the content-addressed branch still dedupes for the replicas.Test plan
stop_app→fresh deploy) both activate new code withversion_verified=True.Files
bioengine/_app/replica_init.py— branch-specific source-freshness skip.