Recent combinations of macOS/clang fail at building old versions of zlib (<1.3.1) - example github issue.
Workaround:
$ TF_SYSTEM_LIBS=zlib bazel build :visqol -c opt
I had to first expunge my failed build:
Also, bazel on hombrew is too new and old versions there only go back to the 8.x line, I downloaded the version bazel suggested in the error message