Skip to content

Wait for object storage at startup instead of exiting - #95

Merged
jaredLunde merged 1 commit into
mainfrom
jared/wait-for-storage-at-startup
Oct 2, 2026
Merged

jaredLunde merged 1 commit into
mainfrom
jared/wait-for-storage-at-startup

Conversation

@jaredLunde

Copy link
Copy Markdown
Contributor

GlideFS exited at startup if object storage was unreachable for about 2 seconds. Its unit's fail-closed restart cap then kept it down.

What happened

On the homelab boot of Sep 20, the network wasn't up yet. The startup compatibility check (a test PUT to S3) is GlideFS's first storage call:

  1. It failed with "Temporary failure in name resolution".
  2. object_store gave up after about 2s of retries, and the daemon exited.
  3. StartLimitBurst=1 refused the restart.

GlideFS stayed down for 11 days. The network side is fixed in beyond (beyondoss/beyond#364: network-online.target now waits for a default gateway). This PR makes the daemon itself tolerate a network blip during any restart.

Change

wait_for_if_match_support retries the check until storage answers:

  • Unreachable (transport or DNS errors): wait with backoff, 1s → 30s cap, and log each attempt at warn.
  • Unsupported (the provider answered and lacks conditional writes): fail at once. This is a misconfiguration, and no retry fixes it.

The wait happens in build_router_only, before ublk recovery, device registration and listeners. Nothing waits while holding kernel state, so the restart cap still guards the spin loop it exists for. systemd's TimeoutStartSec (600s) bounds the wait.

Tests

The storage_compatibility tests use a store that's unreachable for its first writes:

  • the single attempt fails as unreachable, and the wait passes once storage answers;
  • an unsupported provider fails without waiting;
  • the two existing tests still pass.

Clippy reports nothing new in the changed code.

🤖 Generated with Claude Code

https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr

The startup compatibility check (a test PUT to S3) is glidefs's first
storage call, and it runs before anything touches kernel state. If
storage was unreachable, object_store gave up after about 2s of retries
and the daemon exited. The unit's fail-closed restart cap
(StartLimitBurst=1) then left it down. On the homelab boot of Sep 20 the
network wasn't up yet ("Temporary failure in name resolution"), and
glidefs stayed down for 11 days.

The check now distinguishes "unreachable" (a transport or DNS error:
wait it out with backoff from 1s, capped at 30s) from "unsupported" (the
provider answered and lacks conditional writes: fail at once). systemd's
TimeoutStartSec bounds the wait. The restart cap still guards what it
exists for, since nothing waits while holding kernel state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr
@jaredLunde
jaredLunde merged commit 09702b5 into main Oct 2, 2026
18 of 23 checks passed
@jaredLunde
jaredLunde deleted the jared/wait-for-storage-at-startup branch October 2, 2026 05:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant