Replace the home-relay watchdog with in-place relay failover (flexaccess-iroh v0.0.8)

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "ezvpn"
-version = "0.0.46"
+version = "0.0.47"
 edition = "2024"
 description = "IP-over-QUIC VPN tunnel via iroh P2P"
 readme = "README.md"
@@ -34,7 +34,7 @@ ipnet = { version = "2", features = ["serde"] }
 # Shared iroh transport layer (RelayConfig + per-relay probe, endpoint
 # build/rebuild, home-relay watchdog, endpoint-bound auth transcript). No
 # `mdns` feature: ezvpn runs no local-network discovery.
-flexaccess-iroh = { git = "https://github.com/flexaccessdev/flexaccess-iroh", tag = "v0.0.3" }
+flexaccess-iroh = { git = "https://github.com/flexaccessdev/flexaccess-iroh", tag = "v0.0.8" }
 # Redirected to the fork below via `[patch.crates-io]`, which also covers the
 # shared crate's own `iroh` dependency so the graph holds a single `iroh`.
 iroh = "1.1.0"

README.md

@@ -722,12 +722,12 @@ the same IP during reconnects. Reassignment is expected mainly after server
 restart or allocation state changes.
 
 A **server** with custom relays watches its own home-relay registration: if it
-has no connected home relay for 60s it re-checks the network, and if that has
-not helped by 180s it rebuilds its endpoint in place (same node id) — the
-in-process equivalent of a restart, so relay-only clients (the mobile apps,
-anything off the LAN) are not stranded with connect timeouts until someone
-restarts the service. See
-[`docs/Architecture.md`](docs/Architecture.md#relay-watchdog-server-custom-relays).
+has no connected home relay for 60s and iroh has not re-homed it on its own,
+it takes the wedged relay out of its relay map and homes on another configured
+relay in place — same node id, same sockets, nothing torn down — so clients off
+the LAN (the mobile apps) are not stranded with connect timeouts until someone
+restarts the service. The relay is put back once it is connectable again. See
+[`docs/Architecture.md`](docs/Architecture.md#relay-failover-server-custom-relays).
 
 ## Relay and Address Lookup
 
@@ -751,10 +751,16 @@ The short version as it applies to `ezvpn`:
   attaches to the connection as dial hints. These are **required** for
   connectivity in that mode — with lookup off there is no published record to
   fall back on — so configure both sides with the full relay list.
-- Every configured custom relay is probed individually at startup and **all**
-  must come online, so a dead backup relay fails startup instead of hiding until
-  you need it. `relay_auth_token` (custom relays only) is validated by the same
-  probe.
+- A custom relay set is **at least two distinct relays**: the server rides out
+  a relay outage by moving onto another configured relay, so one relay is
+  rejected at startup.
+- Every configured custom relay is probed individually at startup. Startup
+  fails only when **none** comes online; a relay that does not is named in a
+  warning and left out of the relay map, so a relay that answers probes but
+  refuses connections cannot keep the process from ever coming online. The
+  server's failover puts it back once it is connectable; a client keeps it out
+  for its session. `relay_auth_token` (custom relays only) is validated by the
+  same probe.
 
 iroh address lookup is endpoint-ID resolution, not real/VPN DNS: it does not
 affect client DNS resolution, and the client does not push DNS or match domains

docs/Architecture.md

@@ -702,54 +702,45 @@ sequenceDiagram
 - Jitter: 0-500ms added to prevent thundering herd
 - Counter reset: Resets to 0 after successful tunnel operation
 
-### Relay Watchdog (Server, Custom Relays)
+### Relay Failover (Server, Custom Relays)
 
 Implemented in the shared
 [flexaccess-iroh](https://github.com/flexaccessdev/flexaccess-iroh) crate
-(`flexaccess_iroh::relay_watchdog`), driven by the serve loop in
-`VpnServer::run`. A custom-relay server is dialable from off the LAN only while
-it is **registered on its home relay** (n0 address lookup is off; clients dial
-by relay hint, and a relay forwards QUIC Initials only to endpoints connected to
-it). iroh has been observed to silently lose its home relay for good after a
-routine relay reconnect (relays behind Cloudflare tunnels reset idle WebSockets
-roughly hourly): no dial retries, no warnings, no registration on any relay —
-the server stops being reachable through the relays until the process
-restarts, while LAN clients that find it over mDNS keep working and hide the
-outage. Relay-only clients see connect timeouts.
-
-The watchdog observes `Endpoint::home_relay_status()` and escalates like the
-client's reconnect loop:
-
-1. no connected home relay for `RELAY_OUTAGE_NUDGE` (60s) → log a warning and
-   call `Endpoint::network_change()` (forces a fresh net report and relay
-   re-selection — enough when only the bookkeeping went stale);
-2. still none at `RELAY_OUTAGE_REBUILD` (180s from the outage start) → the
-   serve loop closes the endpoint (bounded by `REBUILD_CLOSE_TIMEOUT`, 5s; a
-   slower close finishes in the background), binds a fresh one with the
-   **same identity** (`server_rebuild_factory`: no per-relay probe, online
-   wait tolerated failing), and accepts on it. The TUN device, address pools,
-   client registries, and status socket carry over; the old endpoint's
-   connections end with it and those clients reconnect on their own. The TUN
-   reader's self-encapsulation filter (the endpoint's local UDP ports) is
-   re-read for the fresh sockets. A failed rebuild is retried every
-   `REBUILD_RETRY` (30s).
-
-A rebuild only helps when iroh's bookkeeping went stale; when the relay itself
-is unreachable the fresh endpoint never registers either, and rebuilding again
-every three minutes would keep dropping the LAN clients that still work. The
-watchdog therefore reports whether the endpoint held a home relay at any point
-(`RelayOutage::relay_seen`), and the serve loop doubles the rebuild deadline
-for each consecutive endpoint that never did (`rebuild_deadline`: 180s, 6m,
-12m, 24m, then capped at `REBUILD_DEADLINE_MAX`, 30m). An endpoint that
-registers resets the escalation to the usual 180s. The 60s nudge is unaffected.
+(`flexaccess_iroh::relay_failover::fail_over_home_relay`), run by
+`VpnServer::run` alongside its accept loop; the design is documented once in
+[iroh-common-architecture/relay-failover.md](https://github.com/flexaccessdev/iroh-common-architecture/blob/main/relay-failover.md).
+A custom-relay server is dialable from off the LAN only while it is
+**registered on its home relay** (n0 address lookup is off; clients dial by
+relay hint, and a relay forwards QUIC Initials only to endpoints connected to
+it). iroh re-homes on its own when a relay is really down, but not when the
+relay keeps answering net-report probes while relay connections to it fail
+(relays behind Cloudflare tunnels that reset idle WebSockets): net_report keeps
+preferring it, the relay connection never re-establishes, and the server is
+registered nowhere until the process restarts.
+
+The failover watches `Endpoint::home_relay_status()`. After 60 s
+(`RELAY_OUTAGE_FAILOVER`) without a connected home relay it takes the wedged
+relay **out of the endpoint's relay map**; the forced net report can only
+prefer a relay still in the map, so the endpoint homes on another configured
+relay **in place**: same node id, same sockets, same direct paths, same
+established connections, same TUN device. Nothing is rebuilt. The removed
+relay is probed every 90 s (`RELAY_RESTORE_INTERVAL`) and put back once it is
+connectable again. Relays that failed the **startup** probe are handled the
+same way: the endpoint is bound without them (`CreatedEndpoint::relays_left_out`,
+which `run` receives) and the failover restores them, so a process that
+starts during such an outage still comes online on the relay that works.
 
 A reconnect at any point resets the outage clock. Non-home relays are connected
 on demand and dropped after a minute idle, which is normal and never counts as
-an outage. With the default relays the watchdog is not armed: reachability
-there rests on n0 publishing/resolution, not on one relay registration.
-
-The watchdog is shared with flextunnel through that crate: fix it there, tag a
-release, and bump the tag here.
+an outage. With the default relays the failover is pending forever:
+reachability there rests on n0 publishing/resolution, not on one relay
+registration. A custom relay set must therefore hold at least two distinct
+relays, which `RelayConfig` enforces at startup.
+
+The failover is shared with tunnel-rs and flextunnel through that crate: fix it
+there, tag a release, and bump the tag here. Until flexaccess-iroh v0.0.7 this
+was a watchdog that rebuilt the endpoint (dropping every connection) after a
+`network_change()` nudge that did nothing on a stable host; both are gone.
 
 ### Client Network Consistency Check (Reconnect)
 

src/config/file_config.rs

@@ -48,6 +48,9 @@ pub struct ServerAuthConfig {
 #[serde(deny_unknown_fields)]
 pub struct VpnServerIrohConfig {
     pub secret_file: Option<PathBuf>,
+    /// Custom relay server URLs. At least two distinct relays: a server keeps
+    /// working through a relay outage by moving onto another configured relay,
+    /// so one relay is rejected. Both sides must list the full set.
     pub relay_urls: Option<Vec<String>>,
     /// Optional shared bearer token sent to the custom relays as
     /// `Authorization: Bearer <token>`. Only valid together with `relay_urls`.
@@ -77,6 +80,9 @@ pub struct ClientAuthConfig {
 #[serde(deny_unknown_fields)]
 pub struct VpnClientIrohConfig {
     pub server_node_id: Option<String>,
+    /// Custom relay server URLs. At least two distinct relays: a server keeps
+    /// working through a relay outage by moving onto another configured relay,
+    /// so one relay is rejected. Both sides must list the full set.
     pub relay_urls: Option<Vec<String>>,
     /// Optional shared bearer token sent to the custom relays as
     /// `Authorization: Bearer <token>`. Only valid together with `relay_urls`.
@@ -629,7 +635,7 @@ auth_key_file = "./client.key"
 
 [iroh]
 server_node_id = "2xnbkpbc7izsilvewd7c62w7wnwziacmpfwvhcrya5nt76dqkpga"
-relay_urls = ["https://relay.example.com"]
+relay_urls = ["https://relay.example.com", "https://relay2.example.com"]
 "#,
         )
         .unwrap();
@@ -640,7 +646,12 @@ relay_urls = ["https://relay.example.com"]
         let iroh = config.iroh.as_ref().unwrap();
         assert_eq!(
             iroh.relay_urls.as_deref(),
-            Some(&["https://relay.example.com".to_string()][..])
+            Some(
+                &[
+                    "https://relay.example.com".to_string(),
+                    "https://relay2.example.com".to_string()
+                ][..]
+            )
         );
 
         let resolved = VpnClientConfigBuilder::new()
@@ -650,9 +661,13 @@ relay_urls = ["https://relay.example.com"]
             .unwrap();
         assert_eq!(
             resolved.relay_config,
-            RelayConfig::from_urls(&["https://relay.example.com".to_string()]).unwrap()
+            RelayConfig::from_urls(&[
+                "https://relay.example.com".to_string(),
+                "https://relay2.example.com".to_string()
+            ])
+            .unwrap()
         );
-        assert_eq!(resolved.relay_config.custom_urls().len(), 1);
+        assert_eq!(resolved.relay_config.custom_urls().len(), 2);
     }
 
     #[test]
@@ -663,7 +678,7 @@ role = "vpnclient"
 
 [iroh]
 server_node_id = "2xnbkpbc7izsilvewd7c62w7wnwziacmpfwvhcrya5nt76dqkpga"
-relay_urls = ["https://relay.example.com"]
+relay_urls = ["https://relay.example.com", "https://relay2.example.com"]
 relay_auth_token = "shared-secret"
 "#,
         )

src/main.rs

@@ -25,9 +25,8 @@ use ezvpn::config::file_config::{
     load_vpn_client_config, load_vpn_server_config,
 };
 use ezvpn::runtime::LockRole;
-use ezvpn::transport::endpoint::{
-    create_client_endpoint, create_server_endpoint, load_secret, server_rebuild_factory,
-};
+use ezvpn::transport::endpoint::{create_client_endpoint, create_server_endpoint, load_secret};
+use flexaccess_iroh::endpoint::CreatedEndpoint;
 use ezvpn::transport::{
     CongestionConfig, CongestionControl, parse_congestion_initial_window, set_congestion_config,
 };
@@ -182,7 +181,8 @@ enum ClientAction {
         #[arg(short = 'n', long)]
         server_node_id: Option<String>,
 
-        /// Custom relay server URL(s)
+        /// Custom relay server URLs (at least two, for failover); the server
+        /// moves onto another one when its home relay is lost
         #[arg(long = "relay-url")]
         relay_urls: Vec<String>,
 
@@ -953,17 +953,10 @@ async fn run_vpn_server(resolved: ResolvedVpnServerConfig) -> Result<()> {
     // for VPN traffic; relays are only the automatic fallback when a direct
     // connection fails. A single endpoint serves both relay modes; internet
     // discovery follows the mode (on for default relays, off for custom).
-    let endpoint = create_server_endpoint(&resolved.relay_config, secret_key.clone())
-        .await
-        .context("Failed to create iroh endpoint")?;
-    // The relay watchdog's remedy of last resort: a fresh endpoint with the
-    // same identity. Only a custom-relay server hangs its reachability on one
-    // home-relay registration (n0 discovery is off, clients dial by relay
-    // hint), so the watchdog is armed for custom relays only.
-    let rebuild = resolved
-        .relay_config
-        .is_custom()
-        .then(|| server_rebuild_factory(resolved.relay_config.clone(), secret_key));
+    let CreatedEndpoint { endpoint, relays_left_out } =
+        create_server_endpoint(&resolved.relay_config, secret_key)
+            .await
+            .context("Failed to create iroh endpoint")?;
 
     log::info!("VPN Server Node ID: {}", endpoint.id());
     log::info!(
@@ -977,7 +970,7 @@ async fn run_vpn_server(resolved: ResolvedVpnServerConfig) -> Result<()> {
         .context("Failed to create VPN server")?;
 
     server
-        .run(endpoint, rebuild)
+        .run(endpoint, &resolved.relay_config, &relays_left_out)
         .await
         .map_err(|e| anyhow::anyhow!("VPN server error: {}", e))
 }

src/transport/endpoint.rs

@@ -2,25 +2,23 @@
 //! [`flexaccess_iroh::endpoint`] builder — the VPN ALPN, its QUIC transport
 //! tuning, the client/server identity rules, the bounded connect, and the
 //! server's secret-key file. Relay configuration, the per-relay startup probe,
-//! and the creation-vs-rebuild policy come from the shared crate.
+//! and the bind-and-come-online policy come from the shared crate.
 
 use crate::error::{VpnError, VpnResult};
 use crate::transport::build_quic_transport_config;
 use crate::tunnel::signaling::VPN_ALPN;
 use anyhow::{Context, Result};
 use base64::{Engine, engine::general_purpose::STANDARD as BASE64};
 use flexaccess_iroh::endpoint::{
-    EndpointOptions, create_endpoint, endpoint_builder, rebuild_endpoint,
+    CreatedEndpoint, EndpointOptions, create_endpoint, endpoint_builder,
 };
 use iroh::{
     Endpoint, EndpointAddr, EndpointId, SecretKey,
     endpoint::{Builder as EndpointBuilder, Connection},
 };
 use std::path::Path;
-use std::sync::Arc;
 use std::time::Duration;
 
-pub use flexaccess_iroh::endpoint::EndpointFactory;
 pub use flexaccess_iroh::relay::RelayConfig;
 
 /// Deadline for establishing the QUIC connection to the VPN server.
@@ -96,9 +94,7 @@ fn base_builder(relay_config: &RelayConfig, publish_address: bool) -> Result<End
 }
 
 /// A server endpoint builder: persistent identity (published on the default
-/// relays) and the VPN ALPN. Binding policy is the caller's —
-/// [`create_server_endpoint`] and [`server_rebuild_factory`] each layer their
-/// own.
+/// relays) and the VPN ALPN.
 fn server_builder(relay_config: &RelayConfig, secret: SecretKey) -> Result<EndpointBuilder> {
     Ok(base_builder(relay_config, true)?
         .alpns(vec![VPN_ALPN.to_vec()])
@@ -112,30 +108,25 @@ fn server_builder(relay_config: &RelayConfig, secret: SecretKey) -> Result<Endpo
 /// resolve it by endpoint ID (iroh's relay failover re-homes and republishes on
 /// its own). With custom relays discovery is off, so clients reach the server
 /// through the relay hints they attach to its `EndpointAddr` (see
-/// `VpnClient::resolve_server_addr`). Strict first-creation policy: every
-/// custom relay is probed and the endpoint must come online.
-pub async fn create_server_endpoint(relay_config: &RelayConfig, secret: SecretKey) -> Result<Endpoint> {
+/// `VpnClient::resolve_server_addr`). Every custom relay is probed (startup
+/// fails only if none answers), the endpoint is bound without the relays that
+/// failed and must come online. The relays left out come back in
+/// [`CreatedEndpoint::relays_left_out`] for the home-relay failover
+/// (`VpnServer::run`) to restore once they are connectable.
+pub async fn create_server_endpoint(
+    relay_config: &RelayConfig,
+    secret: SecretKey,
+) -> Result<CreatedEndpoint> {
     create_endpoint(relay_config, server_builder(relay_config, secret)?).await
 }
 
-/// The rebuild recipe for the server endpoint, used when the relay watchdog
-/// (`flexaccess_iroh::relay_watchdog`) gives up on the current one. Same
-/// identity as the original, so the server's node id — what clients dial —
-/// never changes. Tolerant rebuild policy (see [`rebuild_endpoint`]): no
-/// relay probe, and the online wait may fail — the watchdog trips again if the
-/// relays stay unreachable, with a lengthening deadline so a dead relay does
-/// not churn the endpoint every few minutes (see `VpnServer::run`).
-pub fn server_rebuild_factory(relay_config: RelayConfig, secret: SecretKey) -> EndpointFactory {
-    Arc::new(move || {
-        let relay_config = relay_config.clone();
-        let secret = secret.clone();
-        Box::pin(async move { rebuild_endpoint(server_builder(&relay_config, secret)?).await })
-    })
-}
-
 /// Create a client endpoint: ephemeral identity, never published (the client
 /// only dials out; its credential is the auth keypair, not the endpoint id).
-/// Strict first-creation policy.
+/// Same relay probe and online wait as the server. A relay that failed the
+/// probe stays out for the client's lifetime: it lives one session and runs
+/// no failover.
 pub async fn create_client_endpoint(relay_config: &RelayConfig) -> Result<Endpoint> {
-    create_endpoint(relay_config, base_builder(relay_config, false)?).await
+    let CreatedEndpoint { endpoint, .. } =
+        create_endpoint(relay_config, base_builder(relay_config, false)?).await?;
+    Ok(endpoint)
 }