getPayload for a block we never served returns 500 after holding the request ~600ms

[0w3n-d]

getPayload for a block we never served returns 500 after holding the request ~600ms

What's wrong

mev-boost broadcasts the signed blinded block to every configured relay, not only the
winner, so we are routinely asked for payloads we never served. We answer those with a
5xx, and only after holding the request for most of a second.

Measured on relay-mainnet-loadbalancer-aws-fr, 2026-09-11.

Outcome per slot over 80 min (399 slots, 151 with getPayload traffic):

	slots
served 200/202 and delivered	67
500-only, and we did not win the slot	60
400s / client aborts only	16
500-only but we did win the slot	5
served but not yet in the delivered list (data-API lag)	3

getPayload status mix over 3 h (1354 requests): 500 485 (36%), 400 376 (28%),
202 371 (27%), client-abort 90 (6.6%), 200 32 (2.4%).

Attributing the 500s by response-body length (internal server error is 21 bytes,
no execution payload for this request is 37):

error	n	share	upstream p50	p90
InternalServerError	377	76%	0.598 s	0.928 s
NoExecutionPayloadFound	62	13%	0.666 s	0.925 s
pingora's own error body (233/249 B)	54	11%	0.708 s	0.817 s

Three separate defects:

1. Wrong status class. Being asked for a payload we never advertised is not a server
   error. At ~3,000/day this is our noisiest error class and it inflates our error rate on
   every public relay monitor.
2. The dedup path erases the reason. get_payload.rs:307 is
   response.ok_or(ProposerApiError::InternalServerError), so a primary failure reaches
   every duplicate caller as a generic internal error. With 10,492
   dedup get_payload, waiting for primary per day, that fan-out is why
   InternalServerError outnumbers NoExecutionPayloadFound 6:1.
3. We hold the request rather than answering. auctioneer/mod.rs:469 parks a
   getPayload for an unknown block as pending_payload. The only ways out are the payload
   arriving by gossip or on_new_slot resolving it at the next slot boundary
   (context.rs:222). That is the 598 ms p50 / 928 ms p90 spent before we answer, on
   roughly half of all slots.

Holding is correct when our own builder or cross-region gossip might still deliver. It is
not correct for a block hash we never served — we can answer that immediately.

This also explains the {'500': 1, '400': 1} pattern present in nearly every failed slot:
one 500 from the hold, plus a 400 GetPayloadAlreadyReceived because only one pending
payload is kept per slot.

And it plausibly explains the client aborts: getPayload has a 6.18% abort rate (35 of 566
over 40 min) whose p90 is 1.23 s, sitting inside the 500-latency band.

Repro

Call POST /eth/v2/builder/blinded_blocks with a validly signed blinded block for a slot
whose winning block was served by a different relay. Expect a prompt 404; observe a 500
after ~600 ms. In production it happens unprompted on every slot we do not win.

Suspected cause

ProposerApiError::NoExecutionPayloadFound maps to INTERNAL_SERVER_ERROR in
proposer/error.rs:208, and the pending-payload hold has no early exit for a block hash
this relay never served.

Affected surface

• crates/relay/src/api/proposer/error.rs -> status mapping for NoExecutionPayloadFound.
• crates/relay/src/api/proposer/get_payload.rs:307 -> dedup path discards the primary's error.
• crates/relay/src/auctioneer/mod.rs:469, crates/relay/src/auctioneer/context.rs:222 ->
  pending-payload hold and its slot-boundary resolution.

Steps (each becomes one PR)

• Step 1: Map NoExecutionPayloadFound to 404 instead of 500, and give the dedup path
  the primary's real error rather than InternalServerError.
  (tests: the status mapping for NoExecutionPayloadFound; a dedup waiter observes the
  primary's error variant, not a generic one) (PR: )
• Step 2: Answer immediately when the requested block hash was never served by this
  relay, instead of parking it as pending_payload. Keep the hold for a hash we did
  serve but do not yet hold locally.
  (tests: a never-served hash resolves without waiting for a slot rollover; a served-but-
  absent hash still parks and is still resolved by a later gossiped payload) (PR: )

Open questions

• 5 slots we did deliver had every getPayload 500 on the FR load balancer. Most likely NV
  or TK served them and this is a per-region artefact, but that is unconfirmed and it is
  the only case in the data that would be a real miss. Check the same slots on the other
  two load balancers before closing this.
• Should a never-served hash be 404 or 204? 404 is more accurate; check what mev-boost and
  commit-boost do with each before changing the wire behaviour.