Skip to content

resource_group client: narrow degraded fallback conditions for GetResourceGroup#11009

Merged
ti-chi-bot[bot] merged 8 commits into
tikv:masterfrom
ystaticy:rg_degraded_RUSettings
Jul 20, 2026
Merged

resource_group client: narrow degraded fallback conditions for GetResourceGroup#11009
ti-chi-bot[bot] merged 8 commits into
tikv:masterfrom
ystaticy:rg_degraded_RUSettings

Conversation

@ystaticy

@ystaticy ystaticy commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: ref #11010

What is changed and how does it work?

Refine the degraded fallback behavior of GetResourceGroup when degradedRUSettings is configured.

Currently, degraded fallback is too broad and may hide original errors from RM. We should only use degraded fallback for transient RM failures, while preserving original errors for non-retryable cases.

Expected behavior:

  • transient RM unavailability -> degraded group
  • RM recovery -> real group
  • resource group not found -> original error
  • caller cancellation/deadline -> original context error
  • generic non-retryable error -> original error
  • nonexistent SwitchGroup target -> do not switch

It is also helpful to preserve the underlying error from ErrClientGetResourceGroup so upper layers can correctly inspect errors such as context.Canceled and context.DeadlineExceeded.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Code changes

Side effects

  • Possible performance regression
  • Increased code complexity
  • Breaking backward compatibility

Related changes

Release note

None.

Summary by CodeRabbit

Summary by CodeRabbit

  • Bug Fixes

    • Improved resource group error reporting by surfacing the most relevant underlying failure cause.
    • Reliably preserved caller Canceled and DeadlineExceeded errors, even when the server reports gRPC deadline failures.
    • Refined degraded-mode fallback to apply only to retryable scenarios, with better “not found” and permission-denied handling.
  • Tests

    • Expanded unit test coverage for degraded fallback and error classification.
    • Updated an integration test to wait for resource group readiness before proceeding.

Signed-off-by: ystaticy <y_static_y@sina.com>
@ti-chi-bot ti-chi-bot Bot added do-not-merge/needs-linked-issue dco-signoff: yes Indicates the PR's author has signed the dco. release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 15, 2026
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The change preserves underlying resource-group errors, maps gRPC cancellation and deadline statuses to context errors, and updates controller fallback decisions to classify errors before creating degraded resource groups.

Changes

Resource group error handling

Layer / File(s) Summary
Error contract and cause preservation
client/errs/errno.go
ErrClientGetResourceGroup stores an underlying error, derives its display cause with fallbacks, and supports unwrapping.
Client error normalization and validation
client/resource_manager_client.go, client/resource_manager_client_test.go
GetResourceGroup preserves caller cancellation and deadline errors while retaining injected gRPC causes; tests verify both behaviors.
Controller fallback selection and coverage
client/resource_group/controller/global_controller.go, client/resource_group/controller/global_controller_test.go, tests/integrations/mcs/resourcemanager/resource_manager_test.go
Controller helpers unwrap and classify errors before selecting degraded fallback; tests cover transient, missing-group, cancellation, deadline, permission, switching, recovery, asynchronous visibility, and unconfigured-degraded cases.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant GlobalController
  participant ResourceManagerClient
  participant DegradedResourceGroup
  Caller->>GlobalController: request resource-group controller
  GlobalController->>ResourceManagerClient: GetResourceGroup(context)
  ResourceManagerClient-->>GlobalController: wrapped error
  GlobalController->>GlobalController: normalize and classify error
  GlobalController->>DegradedResourceGroup: create degraded group when eligible
  GlobalController-->>Caller: degraded controller or normalized error
Loading

Possibly related issues

  • tikv/pd issue 11010 — Covers narrowed degraded fallback logic and preservation of underlying GetResourceGroup errors.

Possibly related PRs

  • tikv/pd#11011 — Updates the same integration test path to wait for GetResourceGroup success before continuing.

Suggested labels: contribution, ok-to-test, lgtm, approved

Suggested reviewers: rleungx, bufferflies

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: narrowing degraded fallback for GetResourceGroup in the resource_group client.
Description check ✅ Passed The description includes the required issue reference, change summary, checklist headings, and release note placeholder, with only minor detail missing.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@client/resource_group/controller/global_controller_test.go`:
- Around line 342-345: Update the newController helper to accept the subtest
testing.T and create a local require.New(t) within each invocation instead of
using the parent re. Update every t.Run call that invokes newController to pass
its subtest t, preserving the existing controller creation and assertion
behavior.

In `@client/resource_group/controller/global_controller.go`:
- Around line 579-580: Update the error handling around
normalizeGetResourceGroupError to check context.Canceled and
context.DeadlineExceeded on the original error before normalization. Then
classify the unwrapped gRPC status so codes.DeadlineExceeded reaches the
existing degraded-mode path, while preserving the current cancellation handling.

In `@client/resource_manager_client.go`:
- Around line 162-173: Update the error mapping in
client/resource_manager_client.go:162-173 around the resource-group client
method to convert codes.DeadlineExceeded to ctx.Err() only when the caller
context is done; otherwise preserve the original gRPC deadline error. Ensure
client/resource_group/controller/global_controller.go:579-588 continues routing
server-generated deadline errors through the degraded fallback, and add coverage
in client/resource_group/controller/global_controller_test.go:397-425 for a
server-returned codes.DeadlineExceeded case.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 80d51619-73e1-4ed0-9026-61aacf4a2b6e

📥 Commits

Reviewing files that changed from the base of the PR and between 6a2787b and d797561.

📒 Files selected for processing (5)
  • client/errs/errno.go
  • client/resource_group/controller/global_controller.go
  • client/resource_group/controller/global_controller_test.go
  • client/resource_manager_client.go
  • client/resource_manager_client_test.go

Comment thread client/resource_group/controller/global_controller_test.go Outdated
Comment thread client/resource_group/controller/global_controller.go Outdated
Comment thread client/resource_manager_client.go
@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.35714% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.26%. Comparing base (5c811b2) to head (e8cd1e7).
⚠️ Report is 22 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #11009      +/-   ##
==========================================
+ Coverage   79.20%   79.26%   +0.06%     
==========================================
  Files         541      541              
  Lines       75450    76035     +585     
==========================================
+ Hits        59762    60272     +510     
- Misses      11456    11518      +62     
- Partials     4232     4245      +13     
Flag Coverage Δ
unittests 79.26% <80.35%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ystaticy ystaticy changed the title change getResourceGroup ,degraded fallback conditions resource_group client: narrow degraded fallback conditions for GetResourceGroup Jul 15, 2026
ystaticy added 3 commits July 15, 2026 17:26
Signed-off-by: ystaticy <y_static_y@sina.com>
Signed-off-by: ystaticy <y_static_y@sina.com>
Signed-off-by: ystaticy <y_static_y@sina.com>
@ystaticy

Copy link
Copy Markdown
Contributor Author

/test pull-unit-test-next-gen-2

@ystaticy

Copy link
Copy Markdown
Contributor Author

/retest-required

1 similar comment
@ystaticy

Copy link
Copy Markdown
Contributor Author

/retest-required

@ystaticy

Copy link
Copy Markdown
Contributor Author

/retest

Signed-off-by: ystaticy <y_static_y@sina.com>
@ystaticy

Copy link
Copy Markdown
Contributor Author

/retest

@ystaticy

Copy link
Copy Markdown
Contributor Author

/test pull-unit-test-next-gen-3

Comment on lines +161 to +164
switch ctxErr := ctx.Err(); ctxErr {
case context.Canceled, context.DeadlineExceeded:
causeErr = ctxErr
}

@D3Hunter D3Hunter Jul 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you considering context cause err? when will ctx.Err() return other type of errors

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ctx.Err() only returns context.Canceled or context.DeadlineExceeded, so this branch is only used to normalize caller-side cancellation or timeout. I am intentionally not using context.Cause here to keep the change minimal and avoid changing the error chain.

func unwrapGetResourceGroupError(err error) error {
for err != nil {
var groupErr *errs.ErrClientGetResourceGroup
if !goerrors.As(err, &groupErr) || groupErr.Err == nil || groupErr.Err == err {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will groupErr be nil?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, groupErr.Err == nil is possible. For example, NewResourceGroupNotExistErr constructs ErrClientGetResourceGroup with only Cause set and no Err field, so there is no inner error to unwrap. The groupErr.Err == nil check handles this case by stopping unwrap and returning the wrapper as-is.

Comment thread client/resource_group/controller/global_controller.go Outdated
Comment thread client/resource_group/controller/global_controller_test.go Outdated
Comment thread client/resource_group/controller/global_controller_test.go
ystaticy added 2 commits July 17, 2026 17:21
Signed-off-by: ystaticy <y_static_y@sina.com>
Signed-off-by: ystaticy <y_static_y@sina.com>
@ystaticy

Copy link
Copy Markdown
Contributor Author

/retest

Signed-off-by: ystaticy <y_static_y@sina.com>
@ti-chi-bot ti-chi-bot Bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jul 17, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@lhy1024: Your lgtm message is repeated, so it is ignored.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ystaticy

Copy link
Copy Markdown
Contributor Author

/retest

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 20, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lhy1024, rleungx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot

ti-chi-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

[LGTM Timeline notifier]

Timeline:

  • 2026-07-17 10:10:12.586253683 +0000 UTC m=+967598.622348739: ☑️ agreed by lhy1024.
  • 2026-07-20 04:29:07.247302715 +0000 UTC m=+1206333.283397841: ☑️ agreed by rleungx.

@ti-chi-bot
ti-chi-bot Bot merged commit 0b37df9 into tikv:master Jul 20, 2026
41 of 43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved dco-signoff: yes Indicates the PR's author has signed the dco. lgtm release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants