If I correctly understand, we would like to maintain the behavior of where sky status of a failed cluster reports INIT state; when a user invokes sky status -r then corrupted cluster will be set to INIT and then user will delete its resources with sky down.
I continued to look into this and could see some specific cloud handling in this method: https://github.com/skypilot-org/skypilot/blob/master/sky/backends/cloud_vm_ray_backend.py#L3668. K8s is being handled in the generic 'else': https://github.com/skypilot-org/skypilot/blob/master/sky/backends/cloud_vm_ray_backend.py#L3849
ray down command is handled inside ray stack: https://github.com/ray-project/ray/blob/ray-2.4.0/python/ray/autoscaler/_private/commands.py#L394 which retrieves none terminated nodes: https://github.com/ray-project/ray/blob/ray-2.4.0/python/ray/autoscaler/_private/commands.py#L439. K8s provider does not return the pod since it is not Running. Hence it does not get removed.
If I correctly understand, we would like to maintain the behavior of where
sky statusof a failed cluster reportsINITstate; when a user invokessky status -rthen corrupted cluster will be set to INIT and then user will delete its resources withsky down.I continued to look into this and could see some specific cloud handling in this method: https://github.com/skypilot-org/skypilot/blob/master/sky/backends/cloud_vm_ray_backend.py#L3668. K8s is being handled in the generic 'else': https://github.com/skypilot-org/skypilot/blob/master/sky/backends/cloud_vm_ray_backend.py#L3849
ray down command is handled inside ray stack: https://github.com/ray-project/ray/blob/ray-2.4.0/python/ray/autoscaler/_private/commands.py#L394 which retrieves none terminated nodes: https://github.com/ray-project/ray/blob/ray-2.4.0/python/ray/autoscaler/_private/commands.py#L439. K8s provider does not return the pod since it is not Running. Hence it does not get removed.