Repository navigation
Wait for object storage at startup instead of exiting - #95
Merged
Merged
Conversation
The startup compatibility check (a test PUT to S3) is glidefs's first
storage call, and it runs before anything touches kernel state. If
storage was unreachable, object_store gave up after about 2s of retries
and the daemon exited. The unit's fail-closed restart cap
(StartLimitBurst=1) then left it down. On the homelab boot of Sep 20 the
network wasn't up yet ("Temporary failure in name resolution"), and
glidefs stayed down for 11 days.
The check now distinguishes "unreachable" (a transport or DNS error:
wait it out with backoff from 1s, capped at 30s) from "unsupported" (the
provider answered and lacks conditional writes: fail at once). systemd's
TimeoutStartSec bounds the wait. The restart cap still guards what it
exists for, since nothing waits while holding kernel state.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GlideFS exited at startup if object storage was unreachable for about 2 seconds. Its unit's fail-closed restart cap then kept it down.
What happened
On the homelab boot of Sep 20, the network wasn't up yet. The startup compatibility check (a test PUT to S3) is GlideFS's first storage call:
StartLimitBurst=1refused the restart.GlideFS stayed down for 11 days. The network side is fixed in beyond (beyondoss/beyond#364:
network-online.targetnow waits for a default gateway). This PR makes the daemon itself tolerate a network blip during any restart.Change
wait_for_if_match_supportretries the check until storage answers:warn.The wait happens in
build_router_only, before ublk recovery, device registration and listeners. Nothing waits while holding kernel state, so the restart cap still guards the spin loop it exists for. systemd'sTimeoutStartSec(600s) bounds the wait.Tests
The
storage_compatibilitytests use a store that's unreachable for its first writes:Clippy reports nothing new in the changed code.
🤖 Generated with Claude Code
https://claude.ai/code/session_0152kQKDdP8XRhoeqisJYpWr