Enhancement Task
When TiDB cannot connect to PD, the PD client service-discovery loop can repeatedly
emit an INFO log for every failed membership refresh attempt:
[2026/07/08 22:22:25.978 +08:00] [INFO] [service_discovery.go:896] ["[pd] cannot update member from this url"] [keyspaceName=keyspace1] [url=http://192.168.206.72:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 192.168.206.72:2379: i/o timeout\" target:192.168.206.72:2379 status:TRANSIENT_FAILURE: ..."]
During prolonged PD unavailability or network failures, repeated copies of this
per-attempt message can cause a log storm, increase log volume, and obscure more
useful diagnostic information. The membership-refresh retries themselves are
necessary for recovery and should remain unchanged.
Why this becomes a log storm
The message is emitted once for every failed PD URL inside updateMember(). A
membership refresh is then multiplied by several retry and scheduling layers:
updateMemberLoop() executes updateMember() through a backoff with a 20 ms
base interval, a 100 ms maximum interval, and a one-second total backoff
budget. When connection failures return quickly, one wake-up can invoke
updateMember() about twelve times.
- Each invocation tries every configured PD URL. With three unreachable PD
endpoints, a single wake-up can therefore emit about 36 copies of the INFO
message before the batch-level warning is logged.
- Membership checks are not triggered only by the one-minute periodic ticker.
A failed service-mode check schedules one every three seconds. Failed TSO,
region, store, keyspace, meta-storage, resource-manager, router, and HTTP
requests can also call ScheduleCheckMemberChanged().
- TSO reconnection is a particularly hot path: stream creation and request
failures schedule membership checks while TiDB continues requesting
timestamps for active SQL transactions.
checkMembershipCh has capacity one and coalesces concurrent notifications,
but it does not rate-limit sustained failures. A new notification can remain
queued while the current retry batch runs, causing another batch to start as
soon as the current one finishes.
Representative TSO call path:
TiDB statement starts or activates a transaction
-> client-go requests a timestamp from the PD client
-> TSO stream creation or request fails
-> ScheduleCheckMemberChanged()
-> updateMemberLoop()
-> Backoffer.Exec(updateMember)
-> updateMember tries every PD URL
-> "[pd] cannot update member from this url"
Ordinary PD RPC failures follow a similar path through the common response error
handler. Multiple TiDB instances, keyspaces, and PD client instances multiply
the resulting log volume further. Sampling the final per-URL message controls
the output from all of these producers without slowing the recovery paths.
Add a reusable sampled logger to the PD client and use it only for
[pd] cannot update member from this url.
Expected behavior:
- Retain the first five matching messages per minute for each service-discovery
client.
- Suppress additional matching messages until the next sampling window.
- Keep sampling budgets independent between service-discovery clients.
- Preserve the batch-level
[pd] failed to update member warning without
sampling.
- Do not change retry frequency, timeouts, returned errors, service discovery,
or recovery behavior.
Enhancement Task
When TiDB cannot connect to PD, the PD client service-discovery loop can repeatedly
emit an INFO log for every failed membership refresh attempt:
During prolonged PD unavailability or network failures, repeated copies of this
per-attempt message can cause a log storm, increase log volume, and obscure more
useful diagnostic information. The membership-refresh retries themselves are
necessary for recovery and should remain unchanged.
Why this becomes a log storm
The message is emitted once for every failed PD URL inside
updateMember(). Amembership refresh is then multiplied by several retry and scheduling layers:
updateMemberLoop()executesupdateMember()through a backoff with a 20 msbase interval, a 100 ms maximum interval, and a one-second total backoff
budget. When connection failures return quickly, one wake-up can invoke
updateMember()about twelve times.endpoints, a single wake-up can therefore emit about 36 copies of the INFO
message before the batch-level warning is logged.
A failed service-mode check schedules one every three seconds. Failed TSO,
region, store, keyspace, meta-storage, resource-manager, router, and HTTP
requests can also call
ScheduleCheckMemberChanged().failures schedule membership checks while TiDB continues requesting
timestamps for active SQL transactions.
checkMembershipChhas capacity one and coalesces concurrent notifications,but it does not rate-limit sustained failures. A new notification can remain
queued while the current retry batch runs, causing another batch to start as
soon as the current one finishes.
Representative TSO call path:
Ordinary PD RPC failures follow a similar path through the common response error
handler. Multiple TiDB instances, keyspaces, and PD client instances multiply
the resulting log volume further. Sampling the final per-URL message controls
the output from all of these producers without slowing the recovery paths.
Add a reusable sampled logger to the PD client and use it only for
[pd] cannot update member from this url.Expected behavior:
client.
[pd] failed to update memberwarning withoutsampling.
or recovery behavior.