The path to Docker Desktop's backend socket can be too long for a unix domain socket.
When that's the case, the only way I know is to cd in the socket's parent dir. This has a nasty side effect of changing the current working directory for the whole process.