What's wrong
get_payload returns InternalServerError to proposers about 5,127 times a
day, and the relay logs no reason for any of them.
The 5,127 figure comes from the #[tracing::instrument(..., err)] event on
api/proposer/get_payload.rs:137/:210, which reports
error=internal server error with the request id and slot and nothing else.
Measured on titan_relay.log.2026-08-30 (mainnet,
relay-mainnet-1-aws-fr). For comparison, on the same day the explicit error
logs on that path — failed sending request to worker (:297), failed to receive payload response from auctioneer (:316), timed out waiting for payload response from auctioneer (:321) — fired zero times.
So the 500s are being produced somewhere that logs nothing, and we cannot tell
from the logs whether a proposer lost a block because of us.
The same instrument(err) event also reports these ordinary client
conditions, which are expected and are covered by the logging cleanup issue,
not by this one: request for past slot (4,109), proposer not registered
(1,331), no execution payload for this request (941), duplicate payload request (57), already delivering payload (38).
Repro
Not yet known — identifying the path is Step 1.
Suspected cause
ProposerApiError::InternalServerError is returned from several places on the
get_payload path. The three that log are silent in production, so the
returns that fire must be the ones without a log line. Enumerating the
variants that map to Display = internal server error will narrow it.
Affected surface
crates/relay/src/api/proposer/get_payload.rs,
crates/relay/src/api/proposer/error.rs.
Steps (each becomes one PR)
Open questions
Do any of these 5,127 coincide with a missed slot? Cross-referencing against
delivered-payload records would say whether this is a cosmetic 500 after the
payload was already delivered, or a real delivery failure.
What's wrong
get_payloadreturnsInternalServerErrorto proposers about 5,127 times aday, and the relay logs no reason for any of them.
The 5,127 figure comes from the
#[tracing::instrument(..., err)]event onapi/proposer/get_payload.rs:137/:210, which reportserror=internal server errorwith the request id and slot and nothing else.Measured on
titan_relay.log.2026-08-30(mainnet,relay-mainnet-1-aws-fr). For comparison, on the same day the explicit errorlogs on that path —
failed sending request to worker(:297),failed to receive payload response from auctioneer(:316),timed out waiting for payload response from auctioneer(:321) — fired zero times.So the 500s are being produced somewhere that logs nothing, and we cannot tell
from the logs whether a proposer lost a block because of us.
The same
instrument(err)event also reports these ordinary clientconditions, which are expected and are covered by the logging cleanup issue,
not by this one:
request for past slot(4,109),proposer not registered(1,331),
no execution payload for this request(941),duplicate payload request(57),already delivering payload(38).Repro
Not yet known — identifying the path is Step 1.
Suspected cause
ProposerApiError::InternalServerErroris returned from several places on theget_payloadpath. The three that log are silent in production, so thereturns that fire must be the ones without a log line. Enumerating the
variants that map to
Display=internal server errorwill narrow it.Affected surface
crates/relay/src/api/proposer/get_payload.rs,crates/relay/src/api/proposer/error.rs.Steps (each becomes one PR)
ProposerApiError::InternalServerErrorfromget_payload, and give each a distinct log or a distinct error variant so the cause is identifiable from the logs. (tests: unit test per newly distinguished variant) (PR: )Open questions
Do any of these 5,127 coincide with a missed slot? Cross-referencing against
delivered-payload records would say whether this is a cosmetic 500 after the
payload was already delivered, or a real delivery failure.