Keep the shim from staying dead after a boot that outran DNS - #33
Merged
Merged
Conversation
nginx resolves the vendor's addresses while parsing its configuration, so a host that reaches nginx before a resolver is answering fails the config test and exits. Nothing tries again, and nginx -t passes cleanly once DNS is up, so it reads as a mystery rather than a race. The fail-open is what hides it. Health stops answering, the watchdog withdraws the redirect, the device reports to the vendor directly, and the vendor's app looks perfectly healthy while Home Assistant records nothing. On the reference install a routine host reboot cost 3h20m of local monitoring before anyone looked. Ship a systemd drop-in: ordered after nss-lookup.target for the ordinary case, and restarting on failure for the rest, since reaching that target does not guarantee a resolver is answering. The start limit is disabled so it never gives up -- if the configuration is genuinely broken the retries change nothing, because the redirect stays withdrawn either way. The docs already noted that nginx resolves the vendor only at start or reload, but drew only the stale-address conclusion. They now draw the fatal one too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #32.
The shim's nginx resolves
www.pumpspy.comwhile parsing its config, so aboot that reaches nginx before a resolver answers ends at
[emerg] host not found in upstream— and nginx exits with nothing to retryit.
nginx -tpasses once DNS is up, which is what makes it read as a mystery.Hit for real on 2026-08-23 after host maintenance rebooted the shim LXC: it
failed this way at 14:39:45 (something restarted it a minute later) and again
at 18:03:43, where nothing did. The shim was down 3h20m.
The fail-open is what hid it — health stopped answering, the watchdog withdrew
the NAT rules, the device went straight to the vendor, and the PumpSpy app
looked healthy throughout. The only symptom was Home Assistant recording
nothing.
What this adds
shim/nginx-service-override.conf, installed as/etc/systemd/system/nginx.service.d/override.conf:After=nss-lookup.target— the ordinary case.Restart=on-failure+RestartSec=10s— the actual fix, since reaching thattarget does not guarantee a resolver is answering.
StartLimitIntervalSec=0— never give up. A genuinely broken config losesnothing by retrying: health stays silent and the redirect stays withdrawn,
which is the fail-open outcome regardless.
Three tests in
tests/test_shim_config.py, each verified to fail with thecorresponding directive removed.
docs/fail-open-shim.mdgains the installstep and a section explaining why booting is the dangerous case — the existing
text noted that nginx resolves the vendor only at start/reload but drew only
the stale-address conclusion.
Verification
Installed on the live shim and confirmed through systemd rather than by reading
the file back:
systemctl show nginxreportsRestart=on-failure,RestartUSec=10s,StartLimitIntervalUSec=0, and both targets inAfter.200 tests pass.