[BUG] model-agent never becomes ready when the models root is mounted read-only

[weetime]

What happened?

The model agent's readiness probe fails permanently on any node where the models root directory is mounted read-only, even though the agent is fully functional on that node.

ModelAgentHealthCheck.Check (pkg/modelagent/healthz.go) ends with:

// Check if the model agent can write to the model root dir
return unix.Access(h.modelsRootDir, unix.W_OK)

On a node where the models root is a read-only mount, that call fails, /healthz returns an error, and the pod stays READY=false forever:

$ grep " /mnt/models " /proc/mounts
<server>:/export/models /mnt/models nfs4 ro,relatime,vers=4.2,...

$ touch /mnt/models/.w_test
touch: cannot touch '/mnt/models/.w_test': Read-only file system

Meanwhile the agent is doing its job correctly on that same node — it resolved both local:// models, parsed their metadata and labelled the node:

Local model path exists for model ClusterBaseModel <name>: /mnt/models/<dir>
Successfully patched node <node> with Ready state for ClusterBaseModel <name>
Successfully updated ConfigMap and cache for ClusterBaseModel <name> with status: Ready

restartCount is 0, so the liveness probe (/livez) is unaffected — only readiness.

What did you expect to happen?

The agent should be reported ready when it can actually serve its purpose on that node. Write access to the models root is not required for local:// models: the agent only stats the path and reads config.json. Requiring W_OK unconditionally makes a supported deployment shape look permanently unhealthy.

(Write access is genuinely needed for the protocols that download — oci://, hf:// — so the check is not wrong in general, only when applied unconditionally.)

How can we reproduce it (as minimally and precisely as possible)?

1. Mount a shared filesystem read-only at the model agent's --models-root-dir on a node (an NFS export mounted ro is the common case, since nothing on that node needs to write to it).
2. Register a model that lives on that share:

apiVersion: ome.io/v1beta1
kind: ClusterBaseModel
metadata:
  name: example
spec:
  storage:
    storageUri: local:///mnt/models/example
    path: /mnt/models/example

3. Observe: the node gets labelled models.ome.io/clusterbasemodel.example=Ready and the model reaches Ready, but the agent pod on that node never becomes ready.

Anything else we need to know?

The practical impact is larger than a misleading status column:

• Rolling updates of the DaemonSet stall. With the default maxUnavailable: 1, a pod that is never Available consumes the entire budget, so no other node gets updated. We hit this while rolling out an unrelated change and initially misdiagnosed it.
• Any alerting on DaemonSet readiness fires permanently for these nodes.

Read-only is arguably the correct way to mount a model share on a serving node — the weights are immutable and nothing on the node writes to them — so this is not an unusual configuration. It is also the configuration that makes local:// attractive in the first place: one copy of the weights, mounted everywhere, no per-node duplication.

I have not proposed a fix here since the right scope depends on maintainer preference (e.g. whether the requirement should follow the protocols actually in use on that node, or become a flag). Happy to send a PR once there is a direction.

Environment

• OME version: v1.2.2 (also present on main — pkg/modelagent/healthz.go is unchanged there)
• Kubernetes version: v1.29.15
• Cloud provider or hardware configuration: on-prem, NVIDIA V100
• OS: Ubuntu 22.04
• Runtime: vLLM 0.8.5.post1
• Model being served (if applicable): Qwen2.5-0.5B-Instruct via local://
• Install method: Helm (ome-crd + ome-resources)