[bug] stage_router no longer falls back to the capable target when the connection to the efficient upstream times out (0.3.0; 0.2.0 fell back)

[mill101]

Summary

With 0.3.0, if a stage_router route's efficient target is down (connection refused), the request is retried for ~91 s and then fails with 504 upstream_timeout. The capable target is never tried. With 0.2.0 and the same config, the request falls back to the capable target (reason="unavailable") and succeeds.

This matters for setups where the efficient tier is a local model that may be stopped: in 0.3.0 the route becomes unusable instead of degrading to the capable tier.

Versions

• switchyard-server 0.3.0 (cargo install --locked switchyard-server --version 0.3.0) — fails
• switchyard-server 0.2.0 (git 9b6efb94) — falls back
• Ubuntu 24.04.4 LTS on WSL2
• x86_64; inbound: Chat Completions (non-streaming); upstreams: OpenAI-compatible (openai_chat)

Minimal config

schema_version = 1

[llm_clients.up]
format = "openai_chat"
base_url = "http://127.0.0.1:18100/v1"   # any OpenAI-compatible server that answers

[llm_clients.down]
format = "openai_chat"
base_url = "http://127.0.0.1:18199/v1"   # nothing listens here -> connection refused

[targets.good]
id = "mock-good"
llm_client = "up"

[targets.dead]
id = "mock-dead"
llm_client = "down"

[routes.fb_dead]
id = "fb-dead"
type = "stage_router"
efficient_target = "dead"
capable_target = "good"
picker = "efficient_first"
confidence_threshold = 0.5
recent_turn_window = 3

Steps

RUST_LOG=info switchyard-server --config min.toml --host 127.0.0.1 --port 4005
curl -s -w '\nHTTP %{http_code} %{time_total}s\n' -H 'content-type: application/json' \
  -d '{"model":"fb-dead","stream":false,"max_tokens":8,"messages":[{"role":"user","content":"ping"}]}' \
  http://127.0.0.1:4005/v1/chat/completions

Observed

	0.3.0	0.2.0
HTTP	504 after 91.37 s	200 after 91.37 s
Body	{"error":{"message":"error sending request","type":"upstream_error","code":"upstream_timeout"}}	answer from mock-good
Log	LLM request failed … status=504 requested_model="fb-dead" selected_model=""	model call failed; trying next candidate from=mock-dead to=mock-good reason="unavailable"

Other cases behaved identically in both versions (429/500 → retried 3× then fell back; 403 → fell back; 400/404 → no fallback; streaming fallback on 429 works).

What seems to happen (from reading 0.3.0 source)

1. switchyard-llm-client client.rs: AttemptFailure::is_retryable treats LlmClientError::Transport as retryable, so a refused connection is retried until the response deadline.
2. When the deadline expires, deadline_error() returns LlmClientError::Timeout ("response did not finish within … ms, retries included").
3. run.rs fallback_reason() maps Transport to RoutingFallbackReason::Unavailable, but has no arm for Timeout, so the route stops instead of trying the next candidate.

So a persistent transport failure never reaches fallback_reason() as Transport; it arrives as Timeout and falls through to None.

Is this intended?

This looks related to #702 (merged 2026-09-16), whose description says:

After retries, the HTTP driver stops routing on client errors; a timeout never tries another candidate.

Compatibility: HTTP client errors stop routing even when timeout_ms is unset or an advisor has fail_open = true.

So stopping on a timeout is clearly intended. What I'm unsure about is whether an upstream that refuses the connection was meant to fall under this too. Before 0.3.0 that case fell back as Unavailable, and fallback_reason() still has a Transport → Unavailable arm, but with the new retry loop a persistent transport failure seems to always end as Timeout, so that arm is effectively never reached for a down upstream.

For deployments whose efficient tier is a local model that may be stopped, this changes stage_router from "degrade to the capable tier" to "fail after ~91 s". If this is intended, it would help to document it (release notes / routing docs), since the 0.3.0 notes mention fallback correctness but not this change. If it is not intended, some options:

• keep the last Transport error (→ Unavailable) when the deadline expires after transport-only failures, or
• stop retrying a candidate on connection-refused and hand off to the next candidate.

Separately, ~91 s before giving up on a refused connection seems long for a local upstream; is there a per-client setting to shorten it?

Happy to test a patch.

[ting-hong-shieh]

I reproduced this on main (9cf6fadf). With timeout_ms unset, refused connections still fall back. The failure in the report is most likely a connect timeout rather than a refusal, and since #702 a timeout stops routing.

Results (debug build of main, Ubuntu on WSL2 in NAT mode, the config from this issue):

Efficient upstream	Result
Nothing listening, so the connection is refused	HTTP 200 from mock-good in 0.83 s; log: trying next candidate from=mock-dead to=mock-good reason="unavailable"
Listener that drops SYNs (accept queue full), default retries	HTTP 504 after 91.4 s of server handling time
Same, max_retries = 0	HTTP 504 after about 30 s, with the same body as the report: {"error":{"message":"error sending request","type":"upstream_error","code":"upstream_timeout"}}

Why the report points to a connect timeout

• A refused connection fails immediately, so 3 attempts with 250 ms + 500 ms of backoff finish in under a second. The reported 91.37 s fits 3 attempts that each wait 30 s for the connect to time out.
• The 30 s is reqwest's default TCP_USER_TIMEOUT on Linux (reqwest 0.13 sets tcp_user_timeout to 30 s), which also limits SYN retransmits.
• Your table shows 91.37 s for 0.2.0 as well. So 0.2.0 appears to have hit the same connect timeouts before falling back, and the upstream was not refusing connections there either.
• timeout_ms is not set, so the client has no response deadline and deadline_error() cannot produce the error.
• error sending request comes from reqwest. A deadline expiry would say response did not finish within … ms, retries included.

Code path

1. reqwest reports the OS ETIMEDOUT from the connect as is_timeout() == true (and is_connect() == true).
2. convert_reqwest_error maps it to LlmClientError::Timeout.
3. fallback_reason returns None for Timeout, so the capable target is never tried. Before fix(client): enforce response deadlines and stop routing on errors #702 (0.2.0), fallback classification mapped Timeout to Unavailable, which is why 0.2.0 fell back.

timeout_ms also affects refused connections. If it is shorter than the retries plus backoff (about 750 ms by default), even a refused connection ends as a deadline Timeout and does not fall back.

Question for maintainers

Should a timeout that occurs before a connection is established fall back the way Transport does? The failing attempt never sent the request. A refused connection is in the same position, and it already falls back as Transport.

If yes, convert_reqwest_error could return Transport when is_connect() is true, checked before is_timeout().

• Other connect errors (refused, DNS, TLS handshake) already map to Transport, so this only reclassifies connect-phase timeouts.
• When no candidate is left, the client would get 502 upstream_error instead of 504 upstream_timeout.
• This does not cover timeout_ms expiring during the connect. The deadline drops the in-flight attempt, so there is no reqwest error left to inspect. That case would need its own decision.

Per-client connect timeout

There isn't a Switchyard setting today. On Linux, reqwest's default TCP user timeout caps each connect attempt at 30 s. timeout_ms bounds the whole response, including every attempt and retry delay.

I have two local tests: a routing test showing that a refused connection falls back, and a client test (Linux only) showing that a connect timeout without timeout_ms is currently reported as Timeout rather than Transport. I can open a PR once the intended behavior is agreed.

[mill101]

Thanks for digging into this — you're right, and the report mislabeled the failure.

I checked my environment: WSL2 is in mirrored networking mode, where 127.0.0.1 is bridged to the Windows host, so a SYN to a closed port on 127.0.0.1 is silently dropped instead of refused:

Target (nothing listening)	Result
127.0.0.1:18199 (used in the report)	no reply, connect timed out after 40 s
127.0.0.2:18199	refused immediately

Re-running the same config with the efficient upstream on 127.0.0.2:18199 (a real refusal), both 0.3.0 and 0.2.0 fall back in 0.76 s (reason="unavailable", HTTP 200 from mock-good). So refused connections are fine, and the 91.37 s in both versions was 3 × 30 s connect timeouts, as you said.

On your question: +1 for treating a timeout before the connection is established like Transport. My real setup has the efficient tier on another machine on the LAN (a local vLLM box) that may be powered off; there the connect times out rather than being refused, and with 0.3.0 the route now fails after ~91 s instead of falling back to the capable tier. Reclassifying connect-phase timeouts via is_connect() would restore that.

I'll update the issue title to say "connect timeout" instead of "refuses connections". Happy to test a PR.

[ting-hong-shieh]

@mill101 I opened #854 with the fix: a connection that times out before it is established is now treated like a refused connection, so the route falls back to the capable target.

To try it:

cargo install --locked --git https://github.com/ting-hong-shieh/Switchyard --branch fix/853-connect-timeout-fallback switchyard-server

With your original config (efficient upstream at 127.0.0.1:18199 in mirrored mode), fb-dead should return HTTP 200 from mock-good, and the log should show trying next candidate from=mock-dead to=mock-good reason="unavailable".

Fallback only starts after every attempt has timed out. On Linux each connect attempt gives up after about 30 s, so with the default max_retries = 2 that is still about 90 s. Setting max_retries = 0 on the efficient client cuts it to about 30 s. A timeout_ms shorter than that total still returns 504 without fallback; that part is unchanged.

If you can, please also check your LAN setup with the vLLM box powered off. Thanks for offering to test.