Skip to content

fix(worker): run introspect/build Ray tasks in a fresh process (max_calls=1)#145

Merged
nilsmechtel merged 2 commits into
mainfrom
fix/build-worker-sysmodules
Jul 21, 2026
Merged

fix(worker): run introspect/build Ray tasks in a fresh process (max_calls=1)#145
nilsmechtel merged 2 commits into
mainfrom
fix/build-worker-sysmodules

Conversation

@nilsmechtel

Copy link
Copy Markdown
Collaborator

Problem

Even after #144, a redeploy could keep running old code. The introspect and build Ray tasks import the user's deployment modules into their worker process, and Ray pools task workers. _load_app_class uses importlib.import_module, which returns a cached sys.modules entry (Python resolves imports from sys.modules before sys.path). So a worker that built an earlier version keeps that version's classes cached:

  • introspect_app captures the stale class in the spec;
  • build_and_run_application binds the stale class — and because the @bioengine.app class carries closure methods (wrap_init / check_health / bioengine_runtime_version), cloudpickle serialises it by value to the replicas.

The replicas therefore run the stale class even though app_dir/source and the content-addressed app_source_uri package on disk are the correct new version. This is upstream of #144's source-materialisation fixes (sys.path insertion is bypassed by sys.modules), and can make running code older than files on disk.

Solution

  • Add max_calls=1 to the introspect and build ray.remote tasks so each runs in a fresh process — empty sys.modules, so the import reads the refreshed source, and runtime_env env_vars are applied fresh.
  • Switch the per-build env application in bootstrap.py from os.environ.setdefault to direct assignment, so a build's BIOENGINE_ARTIFACT_VERSION / BIOENGINE_APP_SOURCE_URI always win (a reused worker would otherwise keep a prior build's values).

Behavioural changes

Each deploy spawns a fresh short-lived Ray worker for its introspect and build tasks (content-addressed runtime_env packages are already cached, so no re-upload/reinstall). No API changes.

Test plan

Files

  • bioengine/apps/builder.pymax_calls=1 on both Ray tasks.
  • bioengine/_app/bootstrap.py — per-build env applied with overwrite semantics.

nilsmechtel and others added 2 commits July 21, 2026 15:34
…alls=1)

The introspect and build Ray tasks import the user's deployment modules into
their worker process. Ray pools task workers, and _load_app_class uses
importlib.import_module — which returns a cached sys.modules entry, since
Python resolves imports from sys.modules before sys.path. So a reused worker
that built an earlier version keeps that version's classes cached, and:

- introspect_app captures the stale class in the spec, and
- build_and_run_application binds the stale class; because the @bioengine.app
  class carries closure methods (wrap_init / check_health /
  bioengine_runtime_version), cloudpickle serialises it by value to the
  replicas — so replicas run the stale class even though app_dir/source and
  the app_source_uri package on disk are the correct new version.

max_calls=1 forces a fresh process per task (empty sys.modules → import reads
the refreshed source; runtime_env env_vars applied fresh). Also switch the
per-build env application from os.environ.setdefault to direct assignment so a
build's BIOENGINE_ARTIFACT_VERSION / _APP_SOURCE_URI always win.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nilsmechtel
nilsmechtel marked this pull request as ready for review July 21, 2026 13:40
@nilsmechtel
nilsmechtel merged commit 725b59d into main Jul 21, 2026
1 check passed
@nilsmechtel
nilsmechtel deleted the fix/build-worker-sysmodules branch July 21, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant