Skip to content

Replace the home-relay watchdog with in-place relay failover #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andrewchen5678 merged 3 commits into from
Sep 5, 2026
Merged

Replace the home-relay watchdog with in-place relay failover #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
66382ef
Replace the home-relay watchdog with in-place relay failover
Sep 5, 2026
44d212b
Document that the two-relay minimum counts distinct relays
Sep 5, 2026
de88179
Let failover act on a second wedged relay while the first is still out
Sep 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
[package]
name = "flexaccess-iroh"
version = "0.0.3"
version = "0.0.7"
edition = "2024"
description = "Shared iroh transport layer for FlexAccess applications: relay configuration and probing, endpoint building and rebuilding, the home-relay watchdog, and the endpoint-bound public-key auth transcript"
description = "Shared iroh transport layer for FlexAccess applications: relay configuration and probing, endpoint building, in-place home-relay failover, and the endpoint-bound public-key auth transcript"
repository = "https://github.com/flexaccessdev/flexaccess-iroh"

[features]
Expand Down Expand Up @@ -34,7 +34,7 @@ tokio = { version = "1", features = ["macros", "rt", "sync", "time"] }
iroh-mdns-address-lookup = { version = "0.5", optional = true }

[dev-dependencies]
# Test double for iroh's `Watcher`-based status APIs (the relay watchdog tests
# Test double for iroh's `Watcher`-based status APIs (the relay failover tests
# drive a plain `Watchable`); the same crate iroh itself re-exports `Watcher` from.
n0-watcher = "1"
tokio = { version = "1", features = ["full", "test-util"] }
12 changes: 6 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Shared iroh transport layer for FlexAccess applications, as a Rust crate.
The programs built on iroh in this org — [tunnel-rs], [ezvpn], [flextunnel] —
share one transport foundation. Its design is documented once in
[iroh-common-architecture]; this crate is that design as code, so a fix to the
relay watchdog or the relay probe lands here once instead of being ported by
relay failover or the relay probe lands here once instead of being ported by
hand into every repo.

[tunnel-rs]: https://github.com/flexaccessdev/tunnel-rs
Expand All @@ -17,9 +17,9 @@ hand into every repo.

| Module | Contents |
|---|---|
| `relay` | `RelayConfig` (default vs custom relays, which also decides whether n0 internet discovery is on), the shared relay auth token, the strict per-relay startup probe |
| `endpoint` | the common endpoint builder, `create_endpoint` (strict first creation) vs `rebuild_endpoint` (tolerant mid-run replacement), `RebuildableEndpoint` |
| `relay_watchdog` | the server-side home-relay watchdog: nudge with `network_change()`, then ask for a rebuild |
| `relay` | `RelayConfig` (default vs custom relays, which also decides whether n0 internet discovery is on; custom relays must number at least two distinct URLs; duplicates are collapsed before the count), the shared relay auth token, the per-relay startup probe (fails only when no relay is reachable) |
| `endpoint` | the common endpoint builder and `create_endpoint` |
| `relay_failover` | the server-side home-relay failover: after 60 s without a connected home relay, take the wedged relay out of the relay map so the forced net report homes the endpoint on another configured relay, in place; put it back once a probe shows it connectable |
| `auth` | the endpoint-bound public-key auth transcript over the [flexaccess-keys] format; each application passes its own domain-separation context |

Deliberately **not** in it: ALPNs, handshake wire formats, QUIC transport
Expand All @@ -35,9 +35,9 @@ takes the resulting `iroh::SecretKey` / `flexaccess_keys` values.

```toml
[dependencies]
flexaccess-iroh = { git = "https://github.com/flexaccessdev/flexaccess-iroh", tag = "v0.0.3" }
flexaccess-iroh = { git = "https://github.com/flexaccessdev/flexaccess-iroh", tag = "v0.0.7" }
# or, with mDNS local-network discovery on every endpoint (compiled out on iOS):
flexaccess-iroh = { git = "...", tag = "v0.0.3", features = ["mdns"] }
flexaccess-iroh = { git = "...", tag = "v0.0.7", features = ["mdns"] }
```

The `flexaccess_keys` crate is re-exported so a consumer signs and verifies
Expand Down
228 changes: 9 additions & 219 deletions src/endpoint.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,23 @@
//! Endpoint construction shared by every FlexAccess program: the common
//! builder, the creation-vs-rebuild policy, and a rebuildable endpoint handle.
//! builder and the bind-and-come-online policy.
//!
//! Identity is the application's: it reads and decodes its own secret-key
//! file (or generates an ephemeral key) and binds the resulting
//! [`iroh::SecretKey`] on the builder itself.
//!
//! Applications layer their own ALPNs, hooks, identity, and QUIC transport
//! tuning onto the [`iroh::endpoint::Builder`] returned by
//! [`endpoint_builder`], then hand it to [`create_endpoint`] (first creation:
//! strict) or [`rebuild_endpoint`] (mid-run replacement: tolerant).
//! [`endpoint_builder`], then hand it to [`create_endpoint`].

use crate::relay::{RELAY_CONNECT_TIMEOUT, RelayConfig, probe_custom_relays};
use anyhow::{Context, Result};
use futures::future::BoxFuture;
use iroh::{
Endpoint, EndpointId,
Endpoint,
address_lookup::{DnsAddressLookup, PkarrPublisher},
endpoint::{Builder as EndpointBuilder, QuicTransportConfig, presets},
};
use log::info;
use std::sync::Arc;
use std::time::Duration;

/// What an application decides about every endpoint it builds.
#[derive(Debug, Clone)]
Expand Down Expand Up @@ -96,9 +93,8 @@ pub fn endpoint_builder(relay_config: &RelayConfig, options: EndpointOptions) ->

/// Wait for a freshly bound endpoint to come online (relay/discovery ready),
/// bounded by [`RELAY_CONNECT_TIMEOUT`]. Does not close the endpoint on
/// failure; the caller decides (creation closes and fails, a rebuild carries
/// on).
pub async fn wait_online(endpoint: &Endpoint) -> Result<()> {
/// failure; the caller decides.
async fn wait_online(endpoint: &Endpoint) -> Result<()> {
info!(
"Waiting for endpoint to come online (timeout: {}s)...",
RELAY_CONNECT_TIMEOUT.as_secs()
Expand All @@ -112,10 +108,10 @@ pub async fn wait_online(endpoint: &Endpoint) -> Result<()> {
}
}

/// First creation of an endpoint: log the relay setup, probe every custom
/// relay (fail if any is unreachable — configuration validation), bind, and
/// require the endpoint to come online. On failure after binding the endpoint
/// is closed before the error propagates (dropping a bound endpoint without
/// Create an endpoint: log the relay setup, probe every custom relay (fail
/// only if none is reachable; see [`probe_custom_relays`]), bind, and require
/// the endpoint to come online. On failure after binding the endpoint is
/// closed before the error propagates (dropping a bound endpoint without
/// `close()` is fatal under `panic = "abort"`).
pub async fn create_endpoint(relay_config: &RelayConfig, builder: EndpointBuilder) -> Result<Endpoint> {
relay_config.log_status();
Expand All @@ -127,209 +123,3 @@ pub async fn create_endpoint(relay_config: &RelayConfig, builder: EndpointBuilde
}
Ok(endpoint)
}

/// Mid-run replacement of an endpoint, the recipe behind an
/// [`EndpointFactory`]. Differs from [`create_endpoint`] deliberately:
///
/// - **No per-relay probe.** At creation the probe validates the configuration
/// (fail fast if *any* relay is down); during an outage that strictness
/// would block recovery through the one relay that still answers.
/// - **The online wait is tolerated failing.** A fresh endpoint is no worse
/// than the wedged one it replaces — LAN peers can still find it over mDNS —
/// and whatever tripped the rebuild (the relay watchdog, a client's
/// reconnect escalation) trips again if the relays stay unreachable.
pub async fn rebuild_endpoint(builder: EndpointBuilder) -> Result<Endpoint> {
let endpoint = builder.bind().await.context("Failed to create iroh endpoint")?;
if let Err(e) = wait_online(&endpoint).await {
log::warn!("Rebuilt endpoint: {e:#}; continuing (local discovery may still work)");
}
Ok(endpoint)
}

/// Recipe producing a fresh, fully bound endpoint — how a
/// [`RebuildableEndpoint`] replaces itself mid-session, or how a server
/// replaces a wedged endpoint when the relay watchdog gives up on it.
pub type EndpointFactory = Arc<dyn Fn() -> BoxFuture<'static, Result<Endpoint>> + Send + Sync>;

/// Bound wait on the old endpoint's graceful close during a rebuild. The close
/// runs as its own task and is never cancelled (dropping a bound endpoint
/// without `close()` is fatal under panic=abort); the bound only keeps the
/// caller's reconnect loop from stalling behind it, letting a slow close
/// finish in the background.
const REBUILD_CLOSE_TIMEOUT: Duration = Duration::from_secs(5);

/// An endpoint handle that can be **rebuilt** from scratch mid-session.
///
/// `Endpoint::network_change()` re-binds dead UDP transports, but a wedged
/// endpoint can be broken beyond what a rebind repairs: a relay link lost to a
/// ping timeout that never re-establishes, stale cached paths for the peer,
/// dead discovery state. A process restart always recovers because it builds a
/// brand-new endpoint; [`Self::rebuild`] gives a reconnect loop that same
/// remedy in-process — fresh sockets, fresh relay connections, fresh discovery
/// — without dropping anything else the process holds (bound listeners, a
/// control socket).
///
/// The handle is `Clone` and shared: a reconnect loop escalates to
/// [`Self::rebuild`] after repeated failures, while the embedder logs
/// [`Self::id`] and [`Self::close`]s whatever endpoint is current at teardown.
#[derive(Clone)]
pub struct RebuildableEndpoint {
/// The live endpoint, swapped by [`Self::rebuild`]. Std lock: accessors
/// clone the handle out synchronously and never hold it across an await.
current: Arc<std::sync::RwLock<Current>>,
factory: EndpointFactory,
/// Serializes [`Self::rebuild`]'s build-and-swap: the handle is shared,
/// and two callers noticing the same outage must not each build an
/// endpoint and have the second discard (and close) the first's good one.
rebuilding: Arc<tokio::sync::Mutex<()>>,
}

/// The installed endpoint plus how many rebuilds produced it, so a rebuild
/// caller can tell whether one already happened while it waited its turn.
struct Current {
generation: u64,
endpoint: Endpoint,
}

impl RebuildableEndpoint {
/// Wrap a bound endpoint with the recipe that rebuilds it.
pub fn from_parts(endpoint: Endpoint, factory: EndpointFactory) -> Self {
Self {
current: Arc::new(std::sync::RwLock::new(Current {
generation: 0,
endpoint,
})),
factory,
rebuilding: Arc::new(tokio::sync::Mutex::new(())),
}
}

/// A clone of the current endpoint handle. Take it fresh per use: a handle
/// held across a [`Self::rebuild`] keeps pointing at the old, closed
/// endpoint.
pub fn endpoint(&self) -> Endpoint {
self.current.read().expect("endpoint lock").endpoint.clone()
}

/// The current endpoint id. Changes on rebuild for an ephemeral identity;
/// stable when the factory binds a fixed secret.
pub fn id(&self) -> EndpointId {
self.endpoint().id()
}

fn generation(&self) -> u64 {
self.current.read().expect("endpoint lock").generation
}

/// Swap in a freshly built endpoint and close the old one. On error the
/// current endpoint stays in place, so the caller can simply retry with it.
///
/// Concurrent calls coalesce: a caller that arrives while another rebuild
/// is in flight waits for it and, if it installed a fresh endpoint,
/// returns `Ok` without building another — its trigger was the same dead
/// endpoint, and [`Self::endpoint`] now yields the replacement. Only if
/// the in-flight rebuild failed does the waiter build one itself.
pub async fn rebuild(&self) -> Result<()> {
let seen = self.generation();
let old = {
let _serialized = self.rebuilding.lock().await;
if self.generation() != seen {
info!(
"Endpoint already rebuilt by a concurrent caller; endpoint id: {}",
self.id()
);
return Ok(());
}
let fresh = (self.factory)().await?;
let mut current = self.current.write().expect("endpoint lock");
current.generation += 1;
std::mem::replace(&mut current.endpoint, fresh)
};
// Graceful close on its own task: bounded wait here, but the task is
// never cancelled (see [`REBUILD_CLOSE_TIMEOUT`]).
let mut close = tokio::task::spawn(async move { old.close().await });
if tokio::time::timeout(REBUILD_CLOSE_TIMEOUT, &mut close)
.await
.is_err()
{
log::warn!("Old endpoint's close is slow; leaving it to finish in the background");
}
info!("Endpoint rebuilt; endpoint id: {}", self.id());
Ok(())
}

/// Close the current endpoint gracefully (session teardown).
pub async fn close(&self) {
self.endpoint().close().await;
}
}

#[cfg(test)]
mod tests {
use super::*;

#[tokio::test]
async fn rebuildable_endpoint_swaps_and_closes_the_old_one() {
// Hermetic: loopback-only endpoints, no relays, no discovery.
fn loopback() -> EndpointBuilder {
Endpoint::builder(presets::Empty)
.relay_mode(iroh::RelayMode::Disabled)
.crypto_provider(Arc::new(rustls::crypto::ring::default_provider()))
}
let first = loopback().bind().await.unwrap();
let first_id = first.id();
let handle = RebuildableEndpoint::from_parts(
first.clone(),
Arc::new(|| Box::pin(async { loopback().bind().await.map_err(Into::into) })),
);
assert_eq!(handle.id(), first_id);

handle.rebuild().await.unwrap();
assert_ne!(handle.id(), first_id, "an ephemeral rebuild gets a new id");
assert!(first.is_closed(), "the replaced endpoint is closed");
handle.close().await;
assert!(handle.endpoint().is_closed());
}

#[tokio::test]
async fn concurrent_rebuilds_coalesce_into_one() {
use std::sync::atomic::{AtomicUsize, Ordering};

fn loopback() -> EndpointBuilder {
Endpoint::builder(presets::Empty)
.relay_mode(iroh::RelayMode::Disabled)
.crypto_provider(Arc::new(rustls::crypto::ring::default_provider()))
}
let builds = Arc::new(AtomicUsize::new(0));
let first = loopback().bind().await.unwrap();
let handle = RebuildableEndpoint::from_parts(first.clone(), {
let builds = builds.clone();
Arc::new(move || {
builds.fetch_add(1, Ordering::SeqCst);
Box::pin(async {
// Hold the build long enough for the second caller to
// queue behind it.
tokio::time::sleep(std::time::Duration::from_millis(200)).await;
loopback().bind().await.map_err(Into::into)
})
})
});

let a = handle.clone();
let b = handle.clone();
let (ra, rb) = tokio::join!(a.rebuild(), async {
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
b.rebuild().await
});
ra.unwrap();
rb.unwrap();
assert_eq!(builds.load(Ordering::SeqCst), 1, "the second caller joined the first");
assert!(first.is_closed());
assert!(!handle.endpoint().is_closed(), "the one fresh endpoint is live");

// A rebuild after the coalesced one is a new outage: it builds again.
handle.rebuild().await.unwrap();
assert_eq!(builds.load(Ordering::SeqCst), 2);
handle.close().await;
}
}
17 changes: 9 additions & 8 deletions src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,14 @@
//! across repos:
//!
//! - [`relay`]: the default-vs-custom [`relay::RelayConfig`] (which also decides
//! whether n0 internet discovery is on), the shared relay auth token, and the
//! strict per-relay startup probe.
//! - [`endpoint`]: the common endpoint builder, the bind-and-come-online policy
//! for first creation versus a mid-run rebuild, and a
//! [`endpoint::RebuildableEndpoint`] handle.
//! - [`relay_watchdog`]: the server-side home-relay watchdog that nudges, then
//! asks for a rebuild, when iroh silently loses its home relay.
//! whether n0 internet discovery is on, and requires at least two distinct
//! custom relays, duplicates collapsed), the shared relay auth token, and the
//! per-relay startup probe.
//! - [`endpoint`]: the common endpoint builder and the bind-and-come-online
//! policy.
//! - [`relay_failover`]: the server-side home-relay failover that moves the
//! endpoint onto another configured relay, in place, when its home relay is
//! lost and iroh does not recover it on its own.
//! - [`auth`]: the endpoint-bound public-key authentication transcript over the
//! shared [`flexaccess_keys`] format; each application supplies its own
//! domain-separation context.
Expand All @@ -32,6 +33,6 @@
pub mod auth;
pub mod endpoint;
pub mod relay;
pub mod relay_watchdog;
pub mod relay_failover;

pub use flexaccess_keys;
Loading
Loading