Hydration failures and builder submission timeouts cannot be attributed from the logs

[0w3n-d]

Status

Two of the relay's highest-volume failures cannot be attributed to anything
from the logs, which blocks fixing either.

Hydration failures. simulator/tile.rs:340 logs
error!(%e, "hydration failed in sim tile") and nothing else. Measured on
relay-mainnet-1-aws-fr over 19 minutes on 2026-09-01 (build
a737fa6f): 8,235 failures, roughly 624k/day, against 34.2M
helix_hydration_cache_hits — a ~1.7% miss rate. Only 389 distinct
(index, hash) pairs, so a small set of transactions fails repeatedly, in
bursts.

That is consistent with at least three different causes, each needing a
different fix:

• one builder's dehydration disagreeing with the relay about what the relay
  already holds,
• the per-slot cache clear in on_new_slot racing submissions that reference
  the previous slot's transactions,
• the submission carrying the full transaction never reaching the sim tile at
  all (dropped upstream, or lost to a ring overrun).

The log carries no builder pubkey, no slot and no block hash, so there is no
way to tell which. Related and probably sharing a root cause: 273
cannot hydrate submission rejections and 4,834 bad order ref merging
rejects in the same window.

Builder submission timeouts. api/router.rs:146 fired 12,015 times in the
same window (~911k/day), every one of them on /relay/v1/builder/blocks. The
tower timeout is 5s while the handler's own wait is 3s, so a slow pipeline
should surface as submit_block.rs:99 — that fired only 487 times. Roughly
96% of these requests spend more than 5s somewhere outside the inner wait, and
the builder receives a bare 408 with no logged reason.

The metrics rule out the obvious: helix_submission_client_to_server_latency_us
is 1.3 ms mean with all 111,373 samples under 5s, and body read, decode and
signature verification are about 1 ms each. helix_submission_trace_latency_us
shows simulation at 248 ms mean, 500 ms p50 and 5 s p99, which is the
standing suspect — but the timeout log records only uri and request_id, so
it cannot be correlated with body size or client.

Why

Both are among the largest error sources on the relay, and neither can be
root-caused from what is logged today. This is the shortest path to fixing
them; writing speculative fixes first risks another deploy cycle spent
guessing.

Affected surface

crates/relay/src/simulator/tile.rs — add submission identity to the
hydration failure.
crates/relay/src/api/router.rs — add non-secret request attribution to the
timeout.

Steps (each becomes one PR)

• Step 1: Add builder_pubkey, slot and block_hash to the hydration failure log, and content_length plus user_agent to the router timeout log. Deliberately not the API key from HEADER_API_KEY: it identifies the builder but is a credential and must not reach the logs. (tests: none — this changes log fields only, with no behaviour to assert; verification is the deployed output) (PR: )
• Step 2: Once a day of attributed data exists, root-cause each and open the real fixes. No code in this step. (tests: n/a) (PR: )

Open questions

Should hydration failures also be counted per builder in the simulator's
existing report_slot_stats line, rather than only logged per event? That
would answer "is it one builder?" without grepping, and it fits the direction
of #542. It is more code than step 1 and would slow the deploy, so it is left
out for now.

Is there a non-secret way to identify the builder at the router layer? The API
key is present in the headers but must not be logged; mapping it to a builder
id needs cache access that the HandleErrorLayer closure does not have.
user_agent is the best available proxy.