Skip to content

Harden Modal run protocol and worker environment propagation #108

Open

Description

@MaxGhenis

Context

The transcript surfaced several Modal execution-protocol risks:

  • Output-directory environment variables were set only in the local launcher, so workers fell back to shared legacy output paths.
  • Detached cells could mix dataset vintages if snapshot defaults were mutable.
  • Run IDs could collide for identical submissions started within one second.

These risks make it hard to prove which raw files belong to a given rerun.

Work

  • Pass run/output directory configuration into the worker container explicitly, not only the local launcher environment.
  • Add run IDs with collision-resistant entropy beyond second-resolution timestamps.
  • Ensure worker outputs land in run-specific directories by default.
  • Add a salvage/backfill manifest path only for explicitly identified historical outputs.

Acceptance Criteria

  • A Modal rerun writes only to its run-specific artifact tree unless a deliberate override is supplied.
  • Two rapid retries cannot share the same run ID/output tree accidentally.
  • The release manifest can trace dashboard rows back to one run tree without shared-folder ambiguity.

Related: #80, #82, #84, #98, #103.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions