Remove the server watchdog and require a self-hosted address lookup with custom relays

Cargo.toml

@@ -1,8 +1,8 @@
 [package]
 name = "flexaccess-iroh"
-version = "0.0.3"
+version = "0.0.6"
 edition = "2024"
-description = "Shared iroh transport layer for FlexAccess applications: relay configuration and probing, endpoint building and rebuilding, the home-relay watchdog, and the endpoint-bound public-key auth transcript"
+description = "Shared iroh transport layer for FlexAccess applications: relay configuration and probing, the self-hosted address lookup custom relays require, endpoint building and rebuilding, and the endpoint-bound public-key auth transcript"
 repository = "https://github.com/flexaccessdev/flexaccess-iroh"
 
 [features]
@@ -20,21 +20,25 @@ base64 = "0.22"
 # consumers see exactly the version this crate signs and verifies with.
 flexaccess-keys = { git = "https://github.com/flexaccessdev/flexaccess-keys", tag = "v0.0.2", default-features = false }
 futures = "0.3"
+# Random bytes for a fresh lookup secret.
+getrandom = "0.4"
 # A range, deliberately: every consumer resolves one `iroh` 1.1.x for its own
 # workspace and this crate compiles against it. A consumer on a fork of iroh
 # redirects this dependency too with `[patch.crates-io]` (see README.md).
 iroh = ">=1.1.0, <1.2.0"
 log = "0.4"
 rustls = { version = "0.23", default-features = false, features = ["ring", "std"] }
 tokio = { version = "1", features = ["macros", "rt", "sync", "time"] }
+url = "2"
 
 # iroh's mDNS lookup sends raw UDP multicast, which iOS rejects without the
 # multicast entitlement, so it is never even a dependency there.
 [target.'cfg(not(target_os = "ios"))'.dependencies]
 iroh-mdns-address-lookup = { version = "0.5", optional = true }
 
 [dev-dependencies]
-# Test double for iroh's `Watcher`-based status APIs (the relay watchdog tests
-# drive a plain `Watchable`); the same crate iroh itself re-exports `Watcher` from.
-n0-watcher = "1"
-tokio = { version = "1", features = ["full", "test-util"] }
+env_logger = "0.11"
+# An in-process relay for the lookup integration test, plain HTTP on
+# localhost (the same server `iroh-relay --dev` runs).
+iroh-relay = { version = ">=1.1.0, <1.2.0", features = ["server"] }
+tokio = { version = "1", features = ["full"] }

README.md

@@ -5,7 +5,7 @@ Shared iroh transport layer for FlexAccess applications, as a Rust crate.
 The programs built on iroh in this org — [tunnel-rs], [ezvpn], [flextunnel] —
 share one transport foundation. Its design is documented once in
 [iroh-common-architecture]; this crate is that design as code, so a fix to the
-relay watchdog or the relay probe lands here once instead of being ported by
+relay probe or endpoint construction lands here once instead of being ported by
 hand into every repo.
 
 [tunnel-rs]: https://github.com/flexaccessdev/tunnel-rs
@@ -17,9 +17,9 @@ hand into every repo.
 
 | Module | Contents |
 |---|---|
-| `relay` | `RelayConfig` (default vs custom relays, which also decides whether n0 internet discovery is on), the shared relay auth token, the strict per-relay startup probe |
-| `endpoint` | the common endpoint builder, `create_endpoint` (strict first creation) vs `rebuild_endpoint` (tolerant mid-run replacement), `RebuildableEndpoint` |
-| `relay_watchdog` | the server-side home-relay watchdog: nudge with `network_change()`, then ask for a rebuild |
+| `relay` | `RelayConfig` (default vs custom relays, which also decides where address lookup happens), the shared relay auth token, the strict per-relay startup probe |
+| `lookup` | the self-hosted address lookup service that custom relays **require**: the `lks1-` secret format (lowercase z-base-32 with a CRC-32, so a typo fails at config load) and the `lookup_url` / `lookup_secret` pair the crate turns into `<url>/<secret>/pkarr` |
+| `endpoint` | the common endpoint builder, `create_endpoint` (strict first creation, including the foreground first publish to the lookup service) vs `rebuild_endpoint` (tolerant mid-run replacement), `RebuildableEndpoint` |
 | `auth` | the endpoint-bound public-key auth transcript over the [flexaccess-keys] format; each application passes its own domain-separation context |
 
 Deliberately **not** in it: ALPNs, handshake wire formats, QUIC transport
@@ -31,13 +31,39 @@ takes the resulting `iroh::SecretKey` / `flexaccess_keys` values.
 
 [flexaccess-keys]: https://github.com/flexaccessdev/flexaccess-keys
 
+## Custom relays need a lookup service
+
+Custom relays turn n0's address lookup off, so without a replacement a
+server that moves to another relay is unreachable to every client that only
+knows the old one. `RelayConfig::resolve` therefore **rejects** custom relay
+URLs without a `lookup_url` and `lookup_secret`: one self-hosted
+`iroh-dns-server` behind a reverse proxy that only serves
+`/<lookup_secret>/…`. Servers publish their relay URL there at startup (in the
+foreground, so a wrong secret or a dead service stops the program with the
+reason) and iroh keeps republishing; clients resolve peers from it. The
+deployment recipe is in
+[self-hosting.md](https://github.com/flexaccessdev/iroh-common-architecture/blob/main/self-hosting.md),
+the design in
+[relays-and-address-lookup.md](https://github.com/flexaccessdev/iroh-common-architecture/blob/main/relays-and-address-lookup.md#custom-relays).
+
+## Server relay recovery
+
+Servers rely on iroh 1.1.x for relay reconnects and re-homing, and on the
+lookup service to tell clients where they went; they keep the same endpoint
+during relay outages. The former server watchdog has been removed. Its
+history, the failure it covered, and the conditions for bringing it back are
+in
+[home-relay-watchdog.md](https://github.com/flexaccessdev/iroh-common-architecture/blob/main/home-relay-watchdog.md).
+
+The client-side `RebuildableEndpoint` remains available for reconnect escalation.
+
 ## Depending on it
 
 ```toml
 [dependencies]
-flexaccess-iroh = { git = "https://github.com/flexaccessdev/flexaccess-iroh", tag = "v0.0.3" }
+flexaccess-iroh = { git = "https://github.com/flexaccessdev/flexaccess-iroh", tag = "v0.0.6" }
 # or, with mDNS local-network discovery on every endpoint (compiled out on iOS):
-flexaccess-iroh = { git = "...", tag = "v0.0.3", features = ["mdns"] }
+flexaccess-iroh = { git = "...", tag = "v0.0.6", features = ["mdns"] }
 ```
 
 The `flexaccess_keys` crate is re-exported so a consumer signs and verifies

src/endpoint.rs

@@ -10,18 +10,27 @@
 //! [`endpoint_builder`], then hand it to [`create_endpoint`] (first creation:
 //! strict) or [`rebuild_endpoint`] (mid-run replacement: tolerant).
 
+use crate::lookup::LookupConfig;
 use crate::relay::{RELAY_CONNECT_TIMEOUT, RelayConfig, probe_custom_relays};
 use anyhow::{Context, Result};
 use futures::future::BoxFuture;
 use iroh::{
-    Endpoint, EndpointId,
-    address_lookup::{DnsAddressLookup, PkarrPublisher},
+    Endpoint, EndpointId, TransportAddr,
+    address_lookup::{
+        DEFAULT_PKARR_TTL, DnsAddressLookup, EndpointData, EndpointInfo, PkarrPublisher,
+        PkarrRelayClient, PkarrResolver,
+    },
     endpoint::{Builder as EndpointBuilder, QuicTransportConfig, presets},
 };
 use log::info;
 use std::sync::Arc;
 use std::time::Duration;
 
+/// How long the first publish of a server's address record may take. It is a
+/// single HTTP PUT to the lookup service; a Cloudflare-tunnelled service
+/// answers in well under a second.
+pub const LOOKUP_PUBLISH_TIMEOUT: Duration = Duration::from_secs(10);
+
 /// What an application decides about every endpoint it builds.
 #[derive(Debug, Clone)]
 pub struct EndpointOptions {
@@ -30,40 +39,44 @@ pub struct EndpointOptions {
     /// product-specific by design — a VPN's datagram path and a proxy's
     /// stream path want different settings.
     pub transport_config: QuicTransportConfig,
-    /// Whether to publish this endpoint's address to n0's pkarr DNS when on the
-    /// default relays (a no-op with custom relays, where internet discovery is
-    /// off). A server with a persistent identity publishes so clients can
-    /// resolve it by id; a client that only dials out should not advertise
-    /// itself.
+    /// Whether to publish this endpoint's address record: to n0's pkarr
+    /// service on the default relays, to the configured lookup service with
+    /// custom relays. A server with a persistent identity publishes so clients
+    /// can resolve it by id; a client that only dials out should not
+    /// advertise itself. Only the relay URL is ever published, never IP
+    /// addresses.
     pub publish_address: bool,
     /// Reach peers **only** through the configured relays: the direct IP
-    /// transports are dropped and no address lookup of any kind (n0 internet
-    /// discovery, mDNS) is added, so nothing can ever produce a direct path.
-    /// A testing and reference mode for a self-hosted relay deployment; only
-    /// meaningful with custom relays (the default relays are rate-limited).
+    /// transports are dropped and no local-network discovery (mDNS) is added,
+    /// so nothing can ever produce a direct path. The address lookup service
+    /// stays, since it carries relay URLs only. A testing and reference mode
+    /// for a self-hosted relay deployment; only meaningful with custom relays
+    /// (the default relays are rate-limited).
     pub relay_only: bool,
 }
 
 /// Create a base endpoint builder with the common configuration.
 ///
-/// iroh *internet* discovery (n0 pkarr publishing + DNS-based lookup of
-/// `_iroh.<endpoint-id>.dns.iroh.link`, see
-/// <https://docs.iroh.computer/concepts/address-lookup>) follows the relay mode:
+/// Internet address lookup (pkarr publishing of the home relay and
+/// resolution of a peer's, see
+/// <https://docs.iroh.computer/concepts/address-lookup>) follows the relay
+/// mode:
 ///
-/// - [`RelayConfig::Default`]: the n0 lookup stack is enabled — DNS resolution
-///   is always on, and pkarr publishing is added only when
-///   [`EndpointOptions::publish_address`] is set.
-/// - [`RelayConfig::Custom`]: n0 internet discovery is disabled — nothing is
-///   published to or resolved from n0's public infrastructure. Dialers reach
-///   peers through relay hints attached to the peer's `EndpointAddr`: iroh
-///   sends QUIC Initials to every configured relay, so the handshake succeeds
-///   via whichever relay the peer is homed on.
+/// - [`RelayConfig::Default`]: the n0 lookup stack — DNS resolution of
+///   `_iroh.<endpoint-id>.dns.iroh.link` is always on, and pkarr publishing
+///   to n0 is added only when [`EndpointOptions::publish_address`] is set.
+/// - [`RelayConfig::Custom`]: the configured self-hosted lookup service —
+///   pkarr resolution over HTTP is always on, and pkarr publishing is added
+///   only when `publish_address` is set. Nothing is published to or resolved
+///   from n0's infrastructure. Dialers may still attach relay hints to the
+///   peer's `EndpointAddr`; the lookup record is what reaches them once the
+///   peer has moved to a relay the hints do not name.
 ///
 /// With the `mdns` feature, mDNS local-network discovery is added independent
 /// of the relay mode (except on iOS, where it is compiled out).
 ///
-/// [`EndpointOptions::relay_only`] overrides all of that: the IP transports
-/// are cleared and no address lookup at all is added.
+/// [`EndpointOptions::relay_only`] then clears the IP transports and skips
+/// mDNS; the internet lookup stays.
 pub fn endpoint_builder(relay_config: &RelayConfig, options: EndpointOptions) -> EndpointBuilder {
     // iroh 1.x requires the crypto provider to be set explicitly on the
     // builder when starting from the `Empty` preset — the `tls-ring` feature
@@ -73,18 +86,29 @@ pub fn endpoint_builder(relay_config: &RelayConfig, options: EndpointOptions) ->
         .transport_config(options.transport_config)
         .crypto_provider(Arc::new(rustls::crypto::ring::default_provider()));
 
-    if options.relay_only {
-        info!("Relay-only mode: no direct paths and no address lookup");
-        return builder.clear_ip_transports();
+    match relay_config.lookup() {
+        Some(lookup) => {
+            let pkarr_url = lookup.pkarr_url();
+            if options.publish_address {
+                builder = builder.address_lookup(PkarrPublisher::builder(pkarr_url.clone()));
+            }
+            builder = builder.address_lookup(PkarrResolver::builder(pkarr_url));
+            info!(
+                "Address lookup via {} (custom relays; nothing goes to n0)",
+                lookup.display_host()
+            );
+        }
+        None => {
+            if options.publish_address {
+                builder = builder.address_lookup(PkarrPublisher::n0_dns());
+            }
+            builder = builder.address_lookup(DnsAddressLookup::n0_dns());
+        }
     }
 
-    if relay_config.is_custom() {
-        info!("Internet discovery disabled (custom relays configured)");
-    } else {
-        if options.publish_address {
-            builder = builder.address_lookup(PkarrPublisher::n0_dns());
-        }
-        builder = builder.address_lookup(DnsAddressLookup::n0_dns());
+    if options.relay_only {
+        info!("Relay-only mode: no direct paths and no local-network discovery");
+        return builder.clear_ip_transports();
     }
     #[cfg(all(feature = "mdns", not(target_os = "ios")))]
     {
@@ -94,6 +118,60 @@ pub fn endpoint_builder(relay_config: &RelayConfig, options: EndpointOptions) ->
     builder
 }
 
+/// Publish the endpoint's address record to the lookup service now, in the
+/// foreground, and fail if the service rejects it.
+///
+/// iroh's own publisher does the same in the background and keeps
+/// republishing for the life of the endpoint, but it only logs failures and
+/// retries forever. A server that cannot publish is unreachable to every
+/// client that does not already know its relay, so the first publish is done
+/// here where a wrong `lookup_secret` (a `404` from the reverse proxy), a
+/// wrong host, or a service that is down stops the program with the reason.
+/// The record carries the relay URLs only, never IP addresses, exactly like
+/// the background publisher's.
+///
+/// Requires the endpoint to be online (it has a home relay to publish).
+pub async fn publish_address_record(endpoint: &Endpoint, lookup: &LookupConfig) -> Result<()> {
+    let addr = endpoint.addr();
+    let relays: Vec<TransportAddr> = addr
+        .relay_urls()
+        .map(|url| TransportAddr::Relay(url.clone()))
+        .collect();
+    if relays.is_empty() {
+        anyhow::bail!("Endpoint has no home relay to publish (is it online?)");
+    }
+    let relay_list: Vec<String> = addr.relay_urls().map(ToString::to_string).collect();
+    let info = EndpointInfo::from_parts(endpoint.id(), EndpointData::new(relays));
+    let packet = info
+        .to_pkarr_signed_packet(endpoint.secret_key(), DEFAULT_PKARR_TTL)
+        .map_err(|e| anyhow::anyhow!("{e:#}"))
+        .context("Failed to sign the address record")?;
+    let dns_resolver = endpoint
+        .dns_resolver()
+        .map_err(|e| anyhow::anyhow!("{e:#}"))
+        .context("Endpoint has no DNS resolver")?
+        .clone();
+    let client = PkarrRelayClient::new(lookup.pkarr_url(), endpoint.tls_config().clone(), dns_resolver);
+    let host = lookup.display_host();
+    match tokio::time::timeout(LOOKUP_PUBLISH_TIMEOUT, client.publish(&packet)).await {
+        Ok(Ok(())) => {
+            info!(
+                "Published address record to the lookup service at {host} (relay: {})",
+                relay_list.join(", ")
+            );
+            Ok(())
+        }
+        Ok(Err(e)) => anyhow::bail!(
+            "Failed to publish the address record to the lookup service at {host}: {e:#}. \
+             Check lookup_url and lookup_secret, and that the service is up (a wrong secret is a 404)"
+        ),
+        Err(_) => anyhow::bail!(
+            "Publishing the address record to the lookup service at {host} timed out after {}s",
+            LOOKUP_PUBLISH_TIMEOUT.as_secs()
+        ),
+    }
+}
+
 /// Wait for a freshly bound endpoint to come online (relay/discovery ready),
 /// bounded by [`RELAY_CONNECT_TIMEOUT`]. Does not close the endpoint on
 /// failure; the caller decides (creation closes and fails, a rebuild carries
@@ -113,18 +191,32 @@ pub async fn wait_online(endpoint: &Endpoint) -> Result<()> {
 }
 
 /// First creation of an endpoint: log the relay setup, probe every custom
-/// relay (fail if any is unreachable — configuration validation), bind, and
-/// require the endpoint to come online. On failure after binding the endpoint
-/// is closed before the error propagates (dropping a bound endpoint without
-/// `close()` is fatal under `panic = "abort"`).
-pub async fn create_endpoint(relay_config: &RelayConfig, builder: EndpointBuilder) -> Result<Endpoint> {
+/// relay (fail if any is unreachable — configuration validation), bind,
+/// require the endpoint to come online, and — for an endpoint that publishes
+/// its address (`publishes_address`, the same value the builder was given as
+/// [`EndpointOptions::publish_address`]) on custom relays — publish its
+/// record to the lookup service in the foreground, failing if the service
+/// rejects it (see [`publish_address_record`]). On failure after binding the
+/// endpoint is closed before the error propagates (dropping a bound endpoint
+/// without `close()` is fatal under `panic = "abort"`).
+pub async fn create_endpoint(
+    relay_config: &RelayConfig,
+    builder: EndpointBuilder,
+    publishes_address: bool,
+) -> Result<Endpoint> {
     relay_config.log_status();
     probe_custom_relays(relay_config).await?;
     let endpoint = builder.bind().await.context("Failed to create iroh endpoint")?;
     if let Err(e) = wait_online(&endpoint).await {
         endpoint.close().await;
         return Err(e);
     }
+    if publishes_address && let Some(lookup) = relay_config.lookup()
+        && let Err(e) = publish_address_record(&endpoint, lookup).await
+    {
+        endpoint.close().await;
+        return Err(e);
+    }
     Ok(endpoint)
 }
 
@@ -136,8 +228,9 @@ pub async fn create_endpoint(relay_config: &RelayConfig, builder: EndpointBuilde
 ///   would block recovery through the one relay that still answers.
 /// - **The online wait is tolerated failing.** A fresh endpoint is no worse
 ///   than the wedged one it replaces — LAN peers can still find it over mDNS —
-///   and whatever tripped the rebuild (the relay watchdog, a client's
-///   reconnect escalation) trips again if the relays stay unreachable.
+///   and the client's reconnect escalation retries if the relays stay unreachable.
+/// - **No foreground publish.** A rebuilt endpoint that publishes leaves it
+///   to iroh's background publisher, which retries until the service answers.
 pub async fn rebuild_endpoint(builder: EndpointBuilder) -> Result<Endpoint> {
     let endpoint = builder.bind().await.context("Failed to create iroh endpoint")?;
     if let Err(e) = wait_online(&endpoint).await {
@@ -147,8 +240,7 @@ pub async fn rebuild_endpoint(builder: EndpointBuilder) -> Result<Endpoint> {
 }
 
 /// Recipe producing a fresh, fully bound endpoint — how a
-/// [`RebuildableEndpoint`] replaces itself mid-session, or how a server
-/// replaces a wedged endpoint when the relay watchdog gives up on it.
+/// [`RebuildableEndpoint`] replaces itself mid-session.
 pub type EndpointFactory = Arc<dyn Fn() -> BoxFuture<'static, Result<Endpoint>> + Send + Sync>;
 
 /// Bound wait on the old endpoint's graceful close during a rebuild. The close