Skip to content

Give up on a hanging vendor before the device gives up on us - #35

Merged
matuszeg merged 1 commit into
mainfrom
forward-timeout
Aug 24, 2026
Merged

matuszeg merged 1 commit into
mainfrom
forward-timeout

Conversation

@matuszeg

Copy link
Copy Markdown
Owner

Closes #29.

upstream_session() created its session with no ClientTimeout, so it
inherited aiohttp's five-minute default. Fine while this only relayed. Not fine
since #21, where a token request is relayed first and answered locally only if
that relay fails — a vendor that hangs rather than refuses would hold the
minted answer far past the point where the device has stopped listening.

Two corrections to the issue, both from measurement

The device's patience is 10 seconds, not 11. The shim log shows
device_got=499 at 10.003 s — the device closing the request. 11 s is its
retry interval; 10 s is where it stops listening.

The bigger problem was not the wait. asyncio.TimeoutError is not an
aiohttp.ClientError, so a timeout escaped _relay's handlers entirely. Three
consequences at once, and the middle one is the worst:

  • the device got aiohttp's own 504 instead of the honest 502,
  • the message was never parsed, losing a reading already in hand — which is the
    whole point of running this locally,
  • and no vendor failure was recorded, so the outage shape that most needs a
    locally minted token would never have opened the gate to get one.

Adding the timeout without catching it would have made that reachable rather
than theoretical.

What shipped

  • VENDOR_TIMEOUT_SECONDS = 4 on the session — ~3× the slowest healthy vendor
    response measured at the shim (0.1–1.3 s), and two attempts still fit inside
    the device's 10 s.
  • _relay catches TimeoutError beside ClientError, with a written-out
    reason since a bare TimeoutError stringifies to nothing.
  • A hang is not retried. The retry exists for the vendor's fast
    FIN-with-no-response, where replaying costs milliseconds; replaying a hang
    doubles the only thing that matters. This falls out of catching
    ServerDisconnectedError alone, and is now pinned by a test rather than left
    implicit.

Tests

A hanging_upstream fixture — accepts the request, answers nothing, holds the
connection — since the existing flaky_upstream models the hangup, which is
a different failure. Four new tests, each verified against the unfixed code:
the two hang tests do not merely fail without the timeout, they sit for five
minutes, which is the defect stated as an experiment.

213 pass. core/ still imports no homeassistant.

The session forwarding to the vendor had no ClientTimeout, so it
inherited aiohttp's five-minute default. That was tolerable while this
only relayed -- the device gives up and retries on its own, and nothing
here depended on answering quickly.

It is no longer only that. Since the local token answer, a token request
is relayed first and answered locally only if that relay fails, so a
vendor that hangs rather than refuses would hold the minted answer far
past the point where the device has stopped listening. The device
abandons a request after about ten seconds -- device_got=499 at 10.003s
in the proxy's access log -- so the answer whose entire value is
arriving in time would not arrive at all.

Four seconds, roughly three times the slowest healthy vendor response
measured at the shim, leaving room for the existing retry without either
attempt approaching the device's limit. A hang is deliberately not
retried: the retry is for a vendor that answers nothing quickly, where
replaying costs milliseconds, and replaying a hang only doubles the
wait.

The second half matters more than the timeout. A timeout is not a
ClientError, so it escaped the relay's handlers entirely: the device got
aiohttp's 504 rather than an honest 502, the message was never parsed --
losing a reading already in hand, which is the whole point of running
this locally -- and no vendor failure was recorded, so the outage shape
that most needs a locally minted token would never have got one.
@matuszeg
matuszeg merged commit afd4c5f into main Aug 24, 2026
3 checks passed
@matuszeg
matuszeg deleted the forward-timeout branch August 24, 2026 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Forwarding has no timeout, so a hanging vendor delays the local token answer past the device's patience

1 participant