[fix] Report a rewritten outbound port as not reachable

.claude/lessons.md

@@ -48,6 +48,15 @@ correctly forwarded relay firewalled for hours. The tell: firewalled from the fi
 of a run, settings untouched, and a plain restart fixes it. `src/reprobe.js` re-runs
 `_updateNetworkState()` while firewalled; re-check that hook on any hyperdht bump.
 
+**`firewalled: false` and `randomized: false` together still do not mean directly
+reachable.** hyperdht offers clients a direct connection only when `dht.remoteAddress()`
+is non-null, and that is null for an unknown host, a randomized port, *or* a consistent
+public port that differs from the bound one — the last is what a NAT rewriting only
+relay-initiated flows produces, and the sampler can settle on it without ever reporting
+randomized. The tell: every client fails with `HOLEPUNCH_*` while the relay is on the
+DHT and not firewalled. Judge reachability by `remoteAddress()` (`networkInfo().publicAddress`),
+and reproduce it in tests with `dht._natAdd()` rather than a stub.
+
 **On Docker Desktop, `firewalled: true` and a 503 from the probe are the expected
 result.** The VM's NAT means the relay is genuinely not publicly reachable, so a local
 Docker run smoke-tests boot and serve — never reachability. Don't debug it as a failure.

OPERATIONS.md

@@ -73,11 +73,21 @@ it is bad, the steps:
 ```sh
 open http://localhost:9200          # or, headless:
 curl -s localhost:9200/readyz
-# {"ready":true,"firewalled":false,"publicKey":"…"}
+# {"ready":true,"state":"reachable","firewalled":false,"directlyReachable":true,"probed":true,"publicKey":"…"}
 ```
 
-`firewalled: true` → clients cannot reach you. Check, in order: the host firewall,
-the cloud security group, and whether the UDP port is actually forwarded.
+`/readyz` answers 200 only for `"state": "reachable"`, and 503 for everything else:
+
+- `firewalled` → clients cannot reach you. Check, in order: the host firewall, the
+  cloud security group, and whether the UDP port is actually forwarded.
+- `port-unstable` → the port is open, but something between the relay and the
+  internet rewrites its outbound UDP port. See **Port unstable** below.
+- `unknown` → the relay has not settled on its public address yet. See **Unknown
+  after a network change** below.
+
+Use `/readyz` for readiness (take the relay out of rotation), never for liveness:
+restarting the container does not fix a port rewritten on the host, so a liveness
+probe on it would restart-loop. The shipped Docker `HEALTHCHECK` uses `/healthz`.
 
 The verdict can also simply be wrong for a while. HyperDHT probes reachability once
 at startup — it needs 3 of 5 remote nodes to ping the port back — and on its own
@@ -88,24 +98,57 @@ and 5 minutes, then every 5 minutes. A relay that flips to reachable shortly aft
 a restart, with `reachability re-probe passed` in the log, was that case. It does not apply under
 `MIRALL_RELAY_ASSUME_REACHABLE`, where nothing was probed to begin with.
 
-Two states the page separates that `/readyz` does not:
-
-- **Assumed reachable.** `MIRALL_RELAY_ASSUME_REACHABLE` makes `firewalled` read
-  `false` whether or not anything was measured. The page says so in words;
-  `/readyz` reports it as `probed: false` and `/metrics` as
-  `relay_reachability_probed 0`. Anything consuming only `firewalled` — a
-  platform health check, an uptime probe — is reporting a fact nobody
-  established.
-- **Symmetric NAT.** A NAT that assigns a different external port per destination
-  reports `firewalled: false` and is still unusable as a relay. The page raises it
-  from `dht.randomized`; nothing else does.
+**Assumed reachable.** `MIRALL_RELAY_ASSUME_REACHABLE` makes `firewalled` read
+`false` whether or not anything was measured. The page says so in words; `/readyz`
+reports it as `probed: false` and `/metrics` as `relay_reachability_probed 0`.
+Anything consuming only `firewalled` — a platform health check, an uptime probe —
+is reporting a fact nobody established.
+
+**Port unstable.** Not firewalled is not enough. HyperDHT offers clients a direct
+connection only when the public port other nodes observe is the port the relay is
+bound to. When something rewrites the relay's *outbound* port, nodes the relay
+contacts see a different port from nodes that contact it, and HyperDHT stops
+advertising a public address: every client then has to hole-punch to the relay,
+which fails for exactly the peers that need one. It shows in one of two shapes,
+both reported as `port-unstable`:
+
+- a different external port per destination (`portRandomized: true` in
+  `status.json`, as behind a symmetric NAT);
+- one consistent but wrong port (`publicPort` in `status.json` differs from the
+  bound port), which is what a NAT rewriting only relay-initiated flows produces.
+
+`/readyz`, `status.json` `state`, the page (**Port unstable**, with the steps), the
+log (`outbound UDP port is being rewritten`, at `warn`) and
+`relay_reachability_state{state="port-unstable"}` all raise it. The causes, in the
+order to check them: a NAT or tunnel between the host and the internet rewrites
+outbound ports (cloud NAT gateway, CGNAT, a VPN or WireGuard tunnel); stale
+conntrack state on the host after a network or container change; Docker's NAT, when
+the container publishes the port (`-p …/udp`) instead of using host networking.
+Fixes are in [The relay is up but nothing connects](#the-relay-is-up-but-nothing-connects).
+
+**It can come and go.** Only flows the relay *starts* are rewritten; replies on
+flows other nodes started keep the real port. Once enough nodes have contacted a
+running relay, HyperDHT can settle on the real port again, advertise it, and direct
+connections then genuinely work, so `/readyz` reads `reachable` and is right to. It
+breaks again after a restart, and can when the relay re-samples its address (a
+known node is re-sampled every 30 minutes). The same slow re-sampling means a
+rewrite that begins on a relay that is already running can take about 30 minutes to
+show. To check for one, restart the relay and read `/readyz` in the first minute.
+
+**Unknown after a network change.** When the host's public IP changes, the relay
+spends a few minutes re-learning it, and HyperDHT advertises no public address
+until it settles, so `/readyz` is 503 with `"state": "unknown"` meanwhile. That is
+expected on a line with a dynamic IP and logged at `info`; a `warn` follows only if
+it has not settled after 10 minutes. Alert on it only when it lasts (the example
+rule in `deploy/prometheus-scrape.example.yml` waits 10 minutes).
 
 **Docker networking.** `network_mode: host` is the reliable choice. Publishing
-UDP with `-p 49737:49737/udp` works on many hosts but Docker's userland proxy and
-conntrack can rewrite the external mapping in ways that degrade hole-punch success
-and are miserable to diagnose. If you must publish rather than share the host
-network, verify with `scripts/probe.js` from a *different* machine, not just
-`/readyz`.
+UDP with `-p 49737:49737/udp` also worked when measured (2026-09-21, a cloud VM with
+its public IP on the interface: `reachable` for 30 minutes and probes bridged), but
+it puts Docker's NAT and its conntrack state in the outbound path, where a clash can
+rewrite the port and read as **Port unstable**. If you must publish rather than
+share the host network, also verify with `scripts/probe.js` from a *different*
+machine.
 
 **`MIRALL_RELAY_ASSUME_REACHABLE`** skips hyperdht's own probing. Set it only when
 you know the host is public — setting it while actually firewalled produces a
@@ -145,6 +188,7 @@ actually matter:
 | Signal | Why it matters |
 |---|---|
 | `relay_dht_firewalled == 1` | You are advertising a key nobody can reach. |
+| `relay_reachability_state{state="reachable"} == 0` for 10 minutes | Peers cannot connect directly: firewalled, **Port unstable**, or the public address has not settled. The wait tolerates the few minutes an IP change takes. |
 | `relay_reachability_probed == 0` | Reachability was asserted, not measured — so the row above is forced to 0 and can never fire. |
 | `relay_ready == 0` | Not listening or not bootstrapped. |
 | `relay_links_active` near `maxActiveLinks` | About to start refusing connections. |
@@ -265,11 +309,29 @@ restart. Both take effect for new links immediately.
 
 ### The relay is up but nothing connects
 
-In order: the status page's reachability verdict (or `/readyz` → `firewalled`); UDP
+In order: the status page's reachability verdict (or `/readyz` → `state`); UDP
 reachability from outside; whether the published public key matches the one on the
 page (a lost seed is the usual cause — check the **Identity seed** line);
 `scripts/probe.js` from another machine.
 
+For `"state": "port-unstable"` (see [Port unstable](#3-networking-and-reachability)):
+
+1. **A NAT or tunnel in front of the host.** It must keep the relay's UDP source
+   port unchanged. If it cannot, run the relay where it has a public IP, or forward
+   on a NAT that preserves ports.
+2. **Stale conntrack state** after a network or container change. Restart the
+   relay; if it is still `port-unstable` after a few minutes, clear the relay's
+   entries on the host:
+   - host networking or bare metal: `conntrack -D -p udp --orig-port-src <MIRALL_RELAY_PORT>`;
+   - any container platform (StartOS, Umbrel): `conntrack -D -p udp -s <container IP>`,
+     and see that platform's package docs.
+3. **Docker port publishing.** Switch to `network_mode: host`. Until then, clearing
+   the container's stale mappings can help:
+   `conntrack -D -p udp -s $(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container>)`.
+
+It recovers on its own once the rewrite stops: the log says `outbound UDP port is
+stable again` and `/readyz` returns 200.
+
 ### Restart
 
 Shutdown is graceful on SIGTERM: the meter stops, blind-relay sessions close, the

README.md

@@ -102,13 +102,24 @@ open http://localhost:9200
 
 # …or, on a headless host, ask the same questions over curl:
 curl -s localhost:9200/readyz
-# {"ready": true, "firewalled": false, "publicKey": "…"}
+# {"ready": true, "state": "reachable", "firewalled": false, "directlyReachable": true, "probed": true, "publicKey": "…"}
 ```
 
 `"firewalled": true` — **Not reachable** on the page — means clients cannot reach
 you. Open UDP 49737 and try again. A firewalled relay starts, logs an error,
 returns 503 on `/readyz`, and serves nothing.
 
+`"state": "port-unstable"` — **Port unstable** on the page — means the port is
+open, but something between the relay and the internet rewrites its outbound UDP
+port, so peers cannot connect to it directly. `/readyz` returns 503. The causes are
+a NAT or tunnel in front of the host, stale connection-tracking state after a
+network change, or Docker port publishing instead of host networking;
+`OPERATIONS.md` §3 has the fixes, and explains why it can come and go.
+
+`"state": "unknown"` means the relay is still learning its public address, which
+is normal for a few minutes after startup or a change of public IP. `/readyz`
+returns 503 until it settles.
+
 ### Running it locally (Docker Desktop) — smoke test only
 
 ```sh
@@ -124,10 +135,10 @@ docker run -d --name mirall-relay \
 open http://localhost:9200
 ```
 
-On a laptop this will report **`"firewalled": true`** and `/readyz` will return
-**503**, and that is correct: your machine is behind NAT, and on macOS/Windows
+On a laptop this will report **`"firewalled": true`**, or **Port unstable** or
+**Unknown**, and `/readyz` will return **503**, and that is correct: your machine is behind NAT, and on macOS/Windows
 Docker Desktop adds a Linux VM in between. The container is healthy and every
-endpoint works, but **it is not a usable relay** — no peer can hole-punch to it.
+endpoint works, but **it is not a usable relay** — no peer can connect to it.
 Use this to check the image, the endpoints and that the seed persists; use a host
 with a public IP for anything real.
 
@@ -335,7 +346,7 @@ Bound to `127.0.0.1:9200` by default.
 | `/status.json` | Everything the page shows, as data. |
 | `/qr.svg` | The public key as a scannable square, to save or print. |
 | `/healthz` | Process is up. |
-| `/readyz` | Listening, bootstrapped, and **not** firewalled. 503 otherwise. `probed` says whether that verdict was measured. |
+| `/readyz` | Listening, bootstrapped, and **directly reachable**: not firewalled, with a stable public address. 503 otherwise. `state` says which (`reachable`, `firewalled`, `port-unstable`, `unknown`, `starting`, `stopped`); `probed` says whether the verdict was measured. Use it for readiness, never liveness: a restart does not fix a rewritten port. |
 | `/metrics` | Prometheus. See `deploy/prometheus-scrape.example.yml` for the alerts worth having. |
 | `/.well-known/mirall-relay.json` | Public key, region, operator, caps. |
 | `/admin/` | **Members page.** Add, show and revoke invites. The page itself is a static shell — see below. |

deploy/prometheus-scrape.example.yml

@@ -13,6 +13,11 @@ scrape_configs:
 #   relay_dht_firewalled == 1
 #     The relay cannot be reached. It is advertising a key nobody can use.
 #
+#   relay_reachability_state{state="reachable"} == 0   for: 10m
+#     Peers cannot connect to this relay directly: firewalled, port unstable
+#     (something rewrites its outbound UDP port), or its public address has not
+#     settled. The `for` tolerates the few minutes an IP change takes.
+#
 #   relay_reachability_probed == 0
 #     Reachability was ASSERTED with MIRALL_RELAY_ASSUME_REACHABLE, not measured,
 #     so relay_dht_firewalled is forced to 0. Alert on the assertion itself,

docker-compose.yml

@@ -17,7 +17,7 @@ services:
     #
     # (b) explicit publishing. Comment out network_mode above and uncomment this.
     #     Verify at http://127.0.0.1:9200 -> "Reachable", or with
-    #     `curl 127.0.0.1:9200/readyz` -> "firewalled": false.
+    #     `curl 127.0.0.1:9200/readyz` -> "state": "reachable".
     #     NOTE: also set MIRALL_RELAY_ADMIN_HOST to 0.0.0.0 below. Under bridge
     #     networking Docker forwards the published port to the CONTAINER's
     #     address, which a socket listening on the container's own loopback never

src/metrics.js

@@ -59,6 +59,14 @@ export function makeMetrics ({ collectDefault = true } = {}) {
       name: 'relay_reachability_probed',
       help: '1 when relay_dht_firewalled came from hyperdht probing, 0 when it was asserted with MIRALL_RELAY_ASSUME_REACHABLE'
     }),
+    // One-hot, so an alert is one rule on state="reachable" and no series vanishes
+    // when the state moves.
+    reachabilityState: new client.Gauge({
+      registers,
+      name: 'relay_reachability_state',
+      help: '1 for the current reachability state (reachable, firewalled, port-unstable, unknown, starting, stopped), 0 for the others',
+      labelNames: ['state']
+    }),
     ready: new client.Gauge({
       registers,
       name: 'relay_ready',
@@ -137,6 +145,15 @@ export function mirrorMembers (metrics, roster) {
   metrics.m.members.set({ state: 'revoked' }, roster.total - roster.active)
 }
 
+export const REACHABILITY_STATES = ['reachable', 'firewalled', 'port-unstable', 'unknown', 'starting', 'stopped']
+
+// Set on scrape from the same function /readyz and the page use, so the three
+// cannot disagree and the gauge cannot go stale between transitions.
+export function mirrorReachability (metrics, state) {
+  if (!metrics?.m.reachabilityState) return
+  for (const each of REACHABILITY_STATES) metrics.m.reachabilityState.set({ state: each }, each === state ? 1 : 0)
+}
+
 // Mirror blind-relay's `server.stats` into the registry. Called immediately
 // before rendering /metrics so a scrape always reflects the live library state
 // rather than whatever a timer last happened to copy.

src/operator/access-copy.js

@@ -9,6 +9,7 @@ const REACHABILITY = {
   reachable: { text: 'Reachable', tone: 'good' },
   assumed: { text: 'Assumed reachable', tone: 'warn' },
   firewalled: { text: 'Not reachable', tone: 'bad' },
+  'port-unstable': { text: 'Port unstable', tone: 'bad' },
   starting: { text: 'Starting', tone: 'idle' },
   stopped: { text: 'Stopped', tone: 'idle' },
   unknown: { text: 'Unknown', tone: 'idle' }

src/operator/http/public-routes.js

@@ -1,5 +1,5 @@
-import { mirrorMembers, mirrorRelayStats } from '../../metrics.js'
-import { capabilityDoc, statusSnapshot } from '../../status.js'
+import { mirrorMembers, mirrorReachability, mirrorRelayStats } from '../../metrics.js'
+import { capabilityDoc, reachabilityState, statusSnapshot } from '../../status.js'
 import { loadAssets, uiPaths, etagFor, renderPage, standaloneQr } from '../status/page.js'
 import { cached, json, notFound, send } from './responses.js'
 
@@ -46,11 +46,16 @@ export function createPublicRoutes ({ cfg, metrics, relay, firewall, roster }) {
         return json(res, 200, { ok: true })
 
       case '/readyz': {
-        const firewalled = relay.firewalled
-        const ok = relay.ready && firewalled === false
+        // Not firewalled is not enough: hyperdht offers clients a direct connection
+        // only with a stable public address, and without one every client must
+        // hole-punch, which fails for the peers that need a relay.
+        const state = reachabilityState(relay)
+        const ok = relay.ready && state === 'reachable'
         return json(res, ok ? 200 : 503, {
           ready: relay.ready,
-          firewalled,
+          state,
+          firewalled: relay.firewalled,
+          directlyReachable: !!relay.networkInfo().publicAddress,
           probed: !cfg.assumeReachable,
           publicKey: relay.publicKeyZ32
         })
@@ -62,6 +67,7 @@ export function createPublicRoutes ({ cfg, metrics, relay, firewall, roster }) {
       case '/metrics': {
         mirrorRelayStats(metrics, relay.relayStats())
         mirrorMembers(metrics, roster)
+        mirrorReachability(metrics, reachabilityState(relay))
         const body = await metrics.registry.metrics()
         return send(res, 200, body, metrics.registry.contentType, { 'cache-control': 'no-store' })
       }