Skip to content

[ENHANCEMENT] stage:// storage protocol: copy models from a mounted share to node-local disk #780

Description

@weetime

What would you like to be added?

A stage:// storage protocol: the model agent copies a model from an already-mounted source directory onto the node's local disk, and inference then loads the weights from local disk.

spec:
  storage:
    storageUri: stage:///mnt/shared/qwen/Qwen3-32B   # source: a path already mounted into the agent
    path: /mnt/data/models/qwen3-32b                  # destination: node-local disk

The source is a filesystem path, not a network address — mounting the underlying share stays the node's job (fstab, autofs, or an NFS PV). The agent contains no network-protocol code, so NFS, CephFS, Lustre, GPFS and plain local disks are all handled identically.

Why is this needed?

Today a model on shared storage has to be registered as local://, and local:// is served in place: processLocalStorageModel validates that the path exists and parses the model config, but deliberately performs no copy. The pod then mounts that path as a hostPath, so the weights are read over the network.

For a large model this is paid repeatedly:

  • every pod start reads the full model over the share, with a cold page cache
  • scaling a deployment to N replicas multiplies that by N, concurrently, against one server
  • a node reboot or eviction pays it again

oci:// and hf:// already have the shape that avoids this — source is distinct from destination, and the agent lands the model on node-local disk — but they require the weights to live in object storage. Users whose models sit on an NFS export today have no way to get that behaviour without standing up an object store and importing everything into it.

stage:// fills that gap by reusing the existing source-to-destination machinery with a local copy in place of a download.

Two things worth noting for reviewers:

  1. No changes outside the model agent. Serving reads only spec.storage.path (the volume, the mount path and MODEL_PATH all derive from it) and never looks at storageUri, so the controller, the webhooks and pod construction are untouched.
  2. Atomic publication is required, not optional. The agent decides a model is present by stat-ing its path, so an interrupted copy would be indistinguishable from a complete one and would be served as truncated weights. The implementation copies into a sibling staging directory, writes a completion marker, and renames into place; the marker also backs ReuseIfExists and post-restart revalidation.

A source-root allowlist is part of the proposal rather than an afterthought: stage:// lets the author of a (cluster-scoped) model name any path the agent can read, and whatever is staged is then mounted into inference pods. Staging is disabled unless roots are configured explicitly, and containment is checked on the symlink-resolved path, per path segment.

Completion requirements

  • API change — none; spec.storage is unchanged, this is a new URI scheme
  • Tests — unit tests for URI parsing, allowlist containment (including symlink escape and sibling-prefix cases), atomic publication, reuse and re-stage semantics, and admission validation
  • Docs update
  • Design doc — happy to write an OEP if maintainers consider this significant enough

Can you help us implement this enhancement?

  • Yes, I can contribute — an implementation is ready and I will open a PR if the direction is agreeable

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions