External volumes: support FUSE-based storage (OSS/S3/gcsfuse)

[mini-cookie]

Summary

The current external-volume CSI integration is validated only against kernel-mounted filesystems (NFS, block/PD, hostpath). FUSE-based object storage (Alibaba OSS via ossfs, S3 via s3fs/mountpoint-s3, GCS via gcsfuse, JuiceFS) does not fit the existing model, and there is currently no code path for it (grep -ri fuse|ossfs|gcsfuse|mount.?pod internal/volume/ docs/ returns nothing FUSE-related). The goal of this issue is to understand the community's plans and views on supporting FUSE-based storage.

Background: how external volumes work today

• The CSI Node plugin runs as a privileged DaemonSet (docs/csi-deployment.md §3), mounting target dirs on the host (e.g. /var/lib/ateom-gvisor) with mountPropagation: Bidirectional.
• atelet connects directly to the node plugin's Unix socket and calls NodeStageVolume/NodePublishVolume when an actor is resumed (internal/volume/csi/plugin.go MountVolume, ~L176); NodeUnpublish/NodeUnstage on teardown (UnmountVolume).
• Volumes are per-actor: name substrate-<actorUID>-<volumeName>, access mode hardcoded SINGLE_NODE_WRITER (getStandardCapabilities(), with a TODO for RWX).
• The mount is created on the host and bind-mounted into the actor sandbox.

This works for NFS/block because the mount lives in the kernel and survives a node-plugin restart.

Problem: FUSE breaks three assumptions

1. Centralized DaemonSet = large blast radius. A FUSE daemon (ossfs/gcsfuse) spawned inline by the node plugin lives inside the DaemonSet container. Any plugin upgrade/restart kills every FUSE daemon on the node → all mounts become transport endpoint is not connected, with no recovery path in the current code. (The user-space client is also memory-sensitive and OOM-prone.)

2. Sidecar-in-worker-pod is only per-worker, not per-actor. Substrate workers are long-lived pods that multiplex many short-lived actors. A FUSE sidecar in the worker pod is shared by all that worker's actor volumes, so it dies with the worker pod and re-creates the shared-fate problem one level down. It also can't keep up with rapid actor churn (inject/teardown per actor).

3. FUSE inside the actor sandbox is incompatible with deep hibernation. Substrate suspends actors by taking a FULL VM snapshot (incl. process memory) and later resumes them, possibly on a different node. An in-sandbox FUSE daemon holds live kernel↔user-space FUSE connection state (/dev/fuse fd, in-flight requests) and a live network session/credentials to the object store — none of which serialize/restore cleanly through a VM memory snapshot. After resume the mount is dead. This is consistent with the current design where the mount is (re)established at resume time and is deliberately not part of the snapshot. (Separately, gVisor's /dev/fuse support is limited, making in-sandbox FUSE impractical anyway.)

Possible solutions: per-volume mount pods, re-established on resume

• On NodePublishVolume for a FUSE-class driver, launch a dedicated mount pod per volume (the pattern used by JuiceFS CSI and Alibaba Cloud CSI) instead of running the FUSE daemon inline in the node plugin. Because volumes are already per-actor, mount pods are naturally per-actor and their lifecycle is isolated from the node plugin (plugin restarts no longer break mounts).
• Tear the mount pod down on NodeUnpublishVolume (actor suspend/delete). Since the mount is re-established at each resume and is outside the snapshot, this fits the hibernation model.
• Keep the FUSE daemon host-side (mount pod), bind-mounted into the sandbox — avoids gVisor /dev/fuse limitations and keeps object-store credentials out of the actor VM.
• Document the FUSE case as an explicit exception to the "centralized node DaemonSet" guidance in docs/csi-deployment.md §3.

Open questions

• How to detect "FUSE-class" drivers (driver allowlist vs. a CSIDriverConfig field) so MountVolume can branch to the mount-pod path.
• Mount-pod scheduling/placement and image, and whether to reuse an existing driver's mount-pod support vs. wrapping a generic FUSE launcher.
• Recovery semantics if a mount pod itself crashes (restart policy, re-mount on existing target).

References

• internal/volume/csi/plugin.go (MountVolume/UnmountVolume, getStandardCapabilities)
• docs/csi-deployment.md §3 (CSI Node DaemonSet, bidirectional propagation, mount-on-resume)
• docs/csi-volumes.md (CSIDriverConfig, NFS-only examples)