client: reduce membership refreshes during transport outages

[rleungx]

What problem does this PR solve?

During a sustained PD transport outage, asynchronous membership-check triggers can keep checkMembershipCh populated. Each trigger starts a fresh one-second GetMembers retry batch, so fast failures can produce continuous RPC bursts and per-URL logs.

This is related to #11020, but intentionally addresses both the repeated RPCs and the logs rather than adding a process-wide sampler.

What is changed and how does it work?

• Preserve the complete existing fast retry batch before changing behavior.
• Enter a transport-degraded mode only when every current URL failed with a structured transport error and every URL has an observable IDLE, CONNECTING, or TRANSIENT_FAILURE gRPC connection.
• While degraded:
  • coalesce asynchronous schedule requests without issuing GetMembers;
  • inspect local ClientConn state every 100 ms;
  • call Connect() for IDLE connections;
  • resume the existing retry batch when a connection is READY, missing, shut down, or the URL set changes;
  • run one real membership safety sweep on the existing one-minute tick.
• Keep synchronous CheckMemberChanged immediate and unsuppressed.
• Track failures per service-discovery client and URL:
  • retain the first detailed error and log structured error-class changes;
  • aggregate repeated failures into a one-minute summary;
  • log recovery for the URL that actually recovers.

The controller uses only structured signals (ErrGRPCDial, gRPC status codes, PD error types, URL identity, and gRPC connectivity state). Error-message text is not used for control decisions or classification.

Compatibility and correctness

• No public API or configuration changes.
• updateMember remains a compatibility wrapper and returns the existing error unchanged.
• Callback and synchronous refresh behavior are unchanged.
• Semantic, application-level, unobserved, or otherwise uncertain failures fall back to the existing retry behavior.
• Router and TSO service-discovery implementations are not changed.
• No new goroutine, healthy-mode ticker, dependency, or Prometheus metric.

Tests

• go test ./servicediscovery -count=1
• go test -race ./servicediscovery -count=1
• make static in client/
• Regression test covering the initial retry batch, sustained scheduled triggers, and immediate synchronous checks.
• Deterministic tests for strict degraded eligibility, READY/IDLE/missing/SHUTDOWN states, URL replacement, error contracts, logging, cleanup, and concurrent failure tracking.

Benchmarks on darwin/amd64:

BenchmarkMemberRefreshControllerInspect       10.97 ns/op    0 B/op    0 allocs/op
BenchmarkMemberFailureTrackerSuppression      17.28 ns/op    0 B/op    0 allocs/op

Summary by CodeRabbit

• New Features

  • Enhanced service discovery member refresh with stricter degraded-mode behavior when all URLs fail by transport and observed connection states are inactive.
  • Added connection-state inspection during degraded operation to proactively reconnect idle connections.
  • Introduced detailed failure tracking with categorized summaries and recovery/suppression logging.

• Bug Fixes

  • Improved classification and handling of member update failures (dial/connection issues, RPC/response/header errors, timeouts, cluster mismatches, missing leaders).
  • Suppressed scheduled refreshes during persistent transport failures while keeping synchronous checks available.

• Tests

  • Added coverage for degraded-mode logic, allocation-free inspection, failure fingerprinting, recovery semantics, concurrency safety, and refreshed update error behavior.

[ti-chi-bot]

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign yisaer for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6eb6f4da-f28f-4a57-8e2a-796d7b75c985

📥 Commits

Reviewing files that changed from the base of the PR and between b630e07 and 989aee5.

📒 Files selected for processing (4)

• client/servicediscovery/member_refresh_controller.go
• client/servicediscovery/member_refresh_controller_test.go
• client/servicediscovery/service_discovery.go
• client/servicediscovery/service_discovery_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• client/servicediscovery/service_discovery.go

---

📝 Walkthrough

Walkthrough

Member refresh gains failure classification, per-URL episode tracking, degraded-mode connection inspection, scheduled-refresh suppression, recovery logging, and service-client integration. Tests cover error contracts, state transitions, concurrency, logging, allocation behavior, and benchmarks.

Changes

Member refresh failure control

Layer / File(s)	Summary
Failure classification and degraded-mode decisions client/servicediscovery/member_refresh_controller.go, client/servicediscovery/member_refresh_controller_test.go	Failure fingerprints distinguish dial, RPC, response, cluster, and leader failures; degraded mode and connection-state inspection decisions are tested.
Failure episodes and summaries client/servicediscovery/member_refresh_controller.go, client/servicediscovery/member_refresh_controller_test.go	The tracker records attempts, suppressed errors, fingerprint changes, recovery, cleanup, and sorted summaries with concurrency coverage.
Member refresh loop integration client/servicediscovery/service_discovery.go, client/servicediscovery/service_discovery_test.go	The update loop polls connection states, classifies member fetch failures, suppresses scheduled retries during transport failures, logs recovery and summaries, and preserves the existing error contract.
Performance validation client/servicediscovery/member_refresh_controller_test.go	Benchmarks cover connection inspection and repeated failure suppression paths.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant updateMemberLoop
  participant memberRefreshController
  participant PD
  participant memberFailureTracker
  participant serviceDiscovery
  updateMemberLoop->>memberRefreshController: evaluate member URLs and connection states
  memberRefreshController->>PD: request member updates when retry is required
  PD-->>serviceDiscovery: members or classified failure
  serviceDiscovery->>memberFailureTracker: record failure or recover URL
  memberFailureTracker-->>serviceDiscovery: suppression or summary state
  serviceDiscovery->>updateMemberLoop: update clients and retry scheduling

Loading

Possibly related PRs

• tikv/pd#11021: Both changes modify the member update failure logging and handling path in service_discovery.go.

Suggested labels: release-note

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is detailed, but it misses the required Issue Number line and does not follow several template sections like commit-message, checklist, and release note.	Add an Issue Number line linking the related issue and fill in the template sections for commit-message, checklist items, and release note.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: reducing membership refreshes during transport outages.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

[FORMAT CHECKER NOTIFICATION]

Notice: To remove the do-not-merge/needs-linked-issue label, please provide the linked issue number on one line in the PR body, for example: Issue Number: close #123 or Issue Number: ref #456, multiple issues should use full syntax for each issue and be separated by a comma, like: Issue Number: close #123, ref #456.

_{📖 For more info, you can check the "Linking issues" section in the CONTRIBUTING.md.}