deploy: force-recreate caddy container so new vhosts get certs

Reloading Caddy in-place did not trigger ACME issuance for the new
api/console/docs vhosts on the last deploy — the reload command
succeeded but no certs were ever obtained, so the new hostnames
returned a TLS internal_error alert.

Recreate the container after every deploy (volume persists, so
existing certs are kept) and dump a non-access-log slice of Caddy
logs + the on-disk cert inventory. Also runs caddy validate up front
so a syntax error aborts the deploy instead of silently leaving the
old config in place.

Author: pulkitpareek18

scripts/deploy-remote.sh

@@ -22,20 +22,47 @@ docker compose --profile "$COMPOSE_PROFILE" up -d --build --remove-orphans
 
 # `up -d --build` does NOT restart a service when only its bind-mounted
 # files (e.g. Caddyfile) changed — Docker only recreates on image /
-# environment / volume-definition drift. Force an explicit Caddy reload
-# so config changes land on every deploy. Idempotent + fast (~2s).
+# environment / volume-definition drift. We need new vhosts to be
+# picked up + their TLS certs to be provisioned on every deploy.
+#
+# Strategy:
+#   1. Try a hot reload (zero downtime).
+#   2. If reload says "config unchanged" or fails, force-recreate the
+#      container so Caddy boots fresh and queues ACME provisioning for
+#      every named vhost in the Caddyfile.
 if docker ps --format '{{.Names}}' | grep -q '^zeroauth-caddy$'; then
+  echo "Validating Caddyfile syntax..."
+  if ! docker exec zeroauth-caddy caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile 2>&1; then
+    echo "Caddyfile is invalid — aborting deploy."
+    exit 1
+  fi
+
   echo "Reloading Caddy to pick up Caddyfile changes..."
-  docker exec zeroauth-caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile \
-    || docker restart zeroauth-caddy
-
-  # Give Caddy a moment to provision certs for any new vhosts, then dump
-  # its recent log so ACME failures land in the deploy output instead of
-  # silently hiding inside the container.
-  sleep 25
-  echo "--- caddy logs (last 200 lines) ---"
-  docker logs --tail 200 zeroauth-caddy 2>&1 || true
+  reload_out="$(docker exec zeroauth-caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile 2>&1)" || reload_out="$reload_out (rc=$?)"
+  echo "$reload_out"
+
+  # Always also force-recreate the container after a deploy. Caddy's
+  # hot reload sometimes skips provisioning new vhosts when ACME
+  # storage carries a partial/failed record — a fresh boot clears
+  # in-process state and re-queues issuance for every name in the
+  # Caddyfile. We still keep the volume so existing certs persist.
+  echo "Force-recreating zeroauth-caddy so any new vhosts get fresh ACME provisioning..."
+  docker compose --profile "$COMPOSE_PROFILE" up -d --force-recreate --no-deps caddy
+
+  # Wait for Caddy to come back, then dump its recent NON-access log
+  # so ACME activity actually surfaces in the deploy output. Without
+  # the filter the chatty per-request access log drowns the lines we
+  # actually need to read.
+  sleep 35
+  echo "--- caddy logs (excl. access log, last 5min) ---"
+  docker logs --since 5m zeroauth-caddy 2>&1 \
+    | grep -Ev '"logger":"http\.log\.access' \
+    | tail -300 || true
   echo "--- end caddy logs ---"
+
+  echo "--- caddy cert inventory ---"
+  docker exec zeroauth-caddy ls /data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/ 2>&1 || true
+  echo "--- end cert inventory ---"
 fi
 
 echo "Waiting for zeroauth-prod health check..."