[ks8_cloud_beta1] Launch hangs due to wrong NodePort being used.

[aviweit]

2023-08-17 12:46:47,031 INFO updater.py:316 -- SSH still not available (SSH command failed.), retrying in 5 seconds.
2023-08-17 12:46:52,082 VINFO command_runner.py:373 -- Running `uptime`
2023-08-17 12:46:52,082 VVINFO command_runner.py:375 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8290c29bcd/3fca379b3f/%C -o ControlPersist=300s -o ProxyCommand=ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -p 30025 -W %h:%p sky@127.0.0.1 -o Port=22 -o ConnectTimeout=10s sky@10.244.0.6 bash --login -c -i 'source ~/.bashrc; export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'`
kex_exchange_identification: Connection closed by remote host
2023-08-17 12:46:52,129 INFO updater.py:316 -- SSH still not available (SSH command failed.), retrying in 5 seconds.
2023-08-17 12:46:57,179 VINFO command_runner.py:373 -- Running `uptime`
2023-08-17 12:46:57,179 VVINFO command_runner.py:375 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8290c29bcd/3fca379b3f/%C -o ControlPersist=300s -o ProxyCommand=ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -p 30025 -W %h:%p sky@127.0.0.1 -o Port=22 -o ConnectTimeout=10s sky@10.244.0.6 bash --login -c -i 'source ~/.bashrc; export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'`
kex_exchange_identification: Connection closed by remote host
2023-08-17 12:46:57,222 INFO updater.py:316 -- SSH still not available (SSH command failed.), retrying in 5 seconds.
2023-08-17 12:47:02,269 VINFO command_runner.py:373 -- Running `uptime`
2023-08-17 12:47:02,269 VVINFO command_runner.py:375 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8290c29bcd/3fca379b3f/%C -o ControlPersist=300s -o ProxyCommand=ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -p 30025 -W %h:%p sky@127.0.0.1 -o Port=22 -o ConnectTimeout=10s sky@10.244.0.6 bash --login -c -i 'source ~/.bashrc; export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'`
kex_exchange_identification: Connection closed by remote host
2023-08-17 12:47:02,307 INFO updater.py:316 -- SSH still not available (SSH command failed.), retrying in 5 seconds.
2023-08-17 12:47:07,357 VINFO command_runner.py:373 -- Running `uptime`
2023-08-17 12:47:07,357 VVINFO command_runner.py:375 -- Full command is `ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8290c29bcd/3fca379b3f/%C -o ControlPersist=300s -o ProxyCommand=ssh -tt -i ~/.ssh/sky-key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -p 30025 -W %h:%p sky@127.0.0.1 -o Port=22 -o ConnectTimeout=10s sky@10.244.0.6 bash --login -c -i 'source ~/.bashrc; export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (uptime)'`

Steps to recreate:

1. sky local up
2. launch some tasks so that several ray clusters are created
3. sky local down
4. sky local up
5. sky status. It still lists the clusters provisioned before
6. launch a task against one of the clusters.

[aviweit]

It uses the former cluster template (e.g. ~/.sky/generated/avi.yaml) where former ProxyCommand port is set.