Skip to content

client: sample repeated member update failure logs #11020

Description

@D3Hunter

Enhancement Task

When TiDB cannot connect to PD, the PD client service-discovery loop can repeatedly
emit an INFO log for every failed membership refresh attempt:

[2026/07/08 22:22:25.978 +08:00] [INFO] [service_discovery.go:896] ["[pd] cannot update member from this url"] [keyspaceName=keyspace1] [url=http://192.168.206.72:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 192.168.206.72:2379: i/o timeout\" target:192.168.206.72:2379 status:TRANSIENT_FAILURE: ..."]

During prolonged PD unavailability or network failures, repeated copies of this
per-attempt message can cause a log storm, increase log volume, and obscure more
useful diagnostic information. The membership-refresh retries themselves are
necessary for recovery and should remain unchanged.

Why this becomes a log storm

The message is emitted once for every failed PD URL inside updateMember(). A
membership refresh is then multiplied by several retry and scheduling layers:

  1. updateMemberLoop() executes updateMember() through a backoff with a 20 ms
    base interval, a 100 ms maximum interval, and a one-second total backoff
    budget. When connection failures return quickly, one wake-up can invoke
    updateMember() about twelve times.
  2. Each invocation tries every configured PD URL. With three unreachable PD
    endpoints, a single wake-up can therefore emit about 36 copies of the INFO
    message before the batch-level warning is logged.
  3. Membership checks are not triggered only by the one-minute periodic ticker.
    A failed service-mode check schedules one every three seconds. Failed TSO,
    region, store, keyspace, meta-storage, resource-manager, router, and HTTP
    requests can also call ScheduleCheckMemberChanged().
  4. TSO reconnection is a particularly hot path: stream creation and request
    failures schedule membership checks while TiDB continues requesting
    timestamps for active SQL transactions.
  5. checkMembershipCh has capacity one and coalesces concurrent notifications,
    but it does not rate-limit sustained failures. A new notification can remain
    queued while the current retry batch runs, causing another batch to start as
    soon as the current one finishes.

Representative TSO call path:

TiDB statement starts or activates a transaction
  -> client-go requests a timestamp from the PD client
  -> TSO stream creation or request fails
  -> ScheduleCheckMemberChanged()
  -> updateMemberLoop()
  -> Backoffer.Exec(updateMember)
  -> updateMember tries every PD URL
  -> "[pd] cannot update member from this url"

Ordinary PD RPC failures follow a similar path through the common response error
handler. Multiple TiDB instances, keyspaces, and PD client instances multiply
the resulting log volume further. Sampling the final per-URL message controls
the output from all of these producers without slowing the recovery paths.

Add a reusable sampled logger to the PD client and use it only for
[pd] cannot update member from this url.

Expected behavior:

  • Retain the first five matching messages per minute for each service-discovery
    client.
  • Suppress additional matching messages until the next sampling window.
  • Keep sampling budgets independent between service-discovery clients.
  • Preserve the batch-level [pd] failed to update member warning without
    sampling.
  • Do not change retry frequency, timeouts, returned errors, service discovery,
    or recovery behavior.

Metadata

Metadata

Assignees

No one assigned

    Labels

    contributionThis PR is from a community contributor.type/enhancementThe issue or PR belongs to an enhancement.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions