Improve QEMU-based container host (Docker Desktop) compatibility

[achimnol]

When using Docker Desktop for Mac or other container host based on QEMU VMs, you may see errors like:

invalid mount config for type "bind": bind source path does not exist: /tmp/backend.ai/ipc/container

One known workaround so far is to replace /tmp with a physically available host directory (such as $HOME/somewhere), but recently (since 4.15 release on Dec 1st 2022), Docker Desktop began to use virtiofs as their default filesystem driver which is not compatible with UNIX socket bind-mounts between macOS and the QEMU VM.

We need to upgrade our socket-relay container's socat version to 1.7.4 or higher (currently it's 1.7.3) and let it use VSOCK-LISTEN instead of UNIX-LISTEN with QEMU-based container hosts.

Also we need to patch libzmq and pyzmq to support VSOCK: zeromq/libzmq#4491

Reported-by: @inureyes

ref)

• https://stefano-garzarella.github.io/posts/2021-01-22-socat-vsock/

• https://man7.org/linux/man-pages/man7/vsock.7.html

  Tasks

• fix: Workaround infinite restarting of UNIX socket-relay on Docker Desktop for Mac with virtiofs #986* This will be reverted once we have the VSOCK-based socket-relay.

• Enable vsock-based communication between compute session containers and the socket-relay container (both inside the VM)* [ ] VSOCK support zeromq/libzmq#4491

• Check if how we can assign and read the CID of containers (not VMs!)* It turned out that containers share the same CID as long as they run inside a single VM. Nonetheless, we could still utilize VSOCK to implement IPC between containers without fully exposing the host network namespace.

• misc: Update socat container's alpine version to 3.17 for socat's VSOCK feature #988

• Change the socket-relay container's main command to use TCP-CONNECT=host.docker.internal with a separate network namespace instead of TCP-CONNECT=localhost with the host-networking mode (not supported in Docker Desktop) when the agent is running with Docker Desktop for Mac/Windows

• Use metadata server instead of socket proxy* To incrementally deploy of the metadata server as it requires custom routing, let's make this as an agent's local configuration and other codes relying on this feature to also recognize the option.

• We need to confirm that this migration should have little impact to the performance of other codes relying on this feature.

JIRA Issue: BA-225

[achimnol]

Another thing to consider for Docker Desktop QEMU VMs:

docker run -it --rm --privileged --pid=host \
  alpine:edge \
  nsenter -t 1 -m -u -n -i bash -c 'echo "write through" > /sys/class/block/vda/queue/write_cache'

to prevent etcd/postgres from crashing due to corrupted files upon sleep & wake-up of laptops.

ref)

• [Docker Desktop] Improve Mac File system performance docker/roadmap#7 (comment)