fix(worker): run introspect/build Ray tasks in a fresh process (max_calls=1)#145
Merged
Conversation
…alls=1) The introspect and build Ray tasks import the user's deployment modules into their worker process. Ray pools task workers, and _load_app_class uses importlib.import_module — which returns a cached sys.modules entry, since Python resolves imports from sys.modules before sys.path. So a reused worker that built an earlier version keeps that version's classes cached, and: - introspect_app captures the stale class in the spec, and - build_and_run_application binds the stale class; because the @bioengine.app class carries closure methods (wrap_init / check_health / bioengine_runtime_version), cloudpickle serialises it by value to the replicas — so replicas run the stale class even though app_dir/source and the app_source_uri package on disk are the correct new version. max_calls=1 forces a fresh process per task (empty sys.modules → import reads the refreshed source; runtime_env env_vars applied fresh). Also switch the per-build env application from os.environ.setdefault to direct assignment so a build's BIOENGINE_ARTIFACT_VERSION / _APP_SOURCE_URI always win. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nilsmechtel
marked this pull request as ready for review
July 21, 2026 13:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Even after #144, a redeploy could keep running old code. The introspect and build Ray tasks import the user's deployment modules into their worker process, and Ray pools task workers.
_load_app_classusesimportlib.import_module, which returns a cachedsys.modulesentry (Python resolves imports fromsys.modulesbeforesys.path). So a worker that built an earlier version keeps that version's classes cached:introspect_appcaptures the stale class in the spec;build_and_run_applicationbinds the stale class — and because the@bioengine.appclass carries closure methods (wrap_init/check_health/bioengine_runtime_version), cloudpickle serialises it by value to the replicas.The replicas therefore run the stale class even though
app_dir/sourceand the content-addressedapp_source_uripackage on disk are the correct new version. This is upstream of #144's source-materialisation fixes (sys.pathinsertion is bypassed bysys.modules), and can make running code older than files on disk.Solution
max_calls=1to the introspect and buildray.remotetasks so each runs in a fresh process — emptysys.modules, so the import reads the refreshed source, andruntime_envenv_vars are applied fresh.bootstrap.pyfromos.environ.setdefaultto direct assignment, so a build'sBIOENGINE_ARTIFACT_VERSION/BIOENGINE_APP_SOURCE_URIalways win (a reused worker would otherwise keep a prior build's values).Behavioural changes
Each deploy spawns a fresh short-lived Ray worker for its introspect and build tasks (content-addressed
runtime_envpackages are already cached, so no re-upload/reinstall). No API changes.Test plan
stop_app→ freshdeploy_app) on a dev image in single-machine mode.Files
bioengine/apps/builder.py—max_calls=1on both Ray tasks.bioengine/_app/bootstrap.py— per-build env applied with overwrite semantics.