pkg/utils/etcdutil: reload watcher after compaction

[rleungx]

What problem does this PR solve?

Issue Number: Close #11032

When an etcd watch revision has been compacted, LoopWatcher resumes from the
compact revision without reloading current state. Changes in the compacted
interval are not replayed, so consumer caches can remain stale.

Paginated loads can also observe different etcd revisions, and an earlier
callback error can be overwritten by a later successful callback.

What is changed and how does it work?

• Reload current state before restarting a compacted watch.
• Read every page of one load from the same etcd revision.
• Preserve the first load callback error and only advance the watch revision
  after a successful load.
• Add regression coverage for missed compacted changes, paginated snapshot
  consistency, and callback error propagation.

Reload LoopWatcher state from a consistent etcd snapshot after watch
compaction. Preserve the first callback error and only advance the watch
revision after a successful load.

Check List

Tests

• Unit test

Release note

Fix an issue where LoopWatcher consumers could miss etcd changes after watch compaction.

Summary by CodeRabbit

• Bug Fixes
  • Improved etcd-backed watcher consistency by enabling deleted-key reconciliation across TSO, metadata, and scheduling watchers (store/config/rules/labels/affinity/group), including after compaction events.
  • Strengthened compaction handling with safer reload/retry behavior, and more reliable snapshot key reconciliation.
  • Improved metadata snapshot consistency by making resource-group/service-limit updates generation-aware to avoid conflicting concurrent changes.
  • Optimized rule watcher updates by skipping patch commits when no relevant changes are detected.
• Tests
  • Expanded coverage for deleted-key reconciliation, compaction reload/backoff behavior, lifecycle hook error handling, and single-revision loading; added metadata snapshot reconciliation tests/benchmarks.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jmpotato for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

LoopWatcher now reloads consistent etcd snapshots after compaction, reconciles deleted keys, and preserves callback errors. Metadata watchers coordinate generation-aware reconciliation with concurrent resource-manager writes.

Changes

LoopWatcher consistency

Layer / File(s)	Summary
Consistent reload and deletion reconciliation pkg/utils/etcdutil/etcdutil.go	LoopWatcher uses consistent revisions, bounded compaction backoff, lifecycle hooks, deferred event state, and deleted-key reconciliation.
Watcher load and compaction tests pkg/utils/etcdutil/etcdutil_test.go	Tests cover lifecycle errors, consistent loads, compaction recovery, reconciliation retries, callback failures, and backoff.
Deleted-key reconciliation wiring pkg/keyspace/..., pkg/mcs/..., pkg/mcs/scheduling/..., pkg/tso/..., server/server.go	Service watchers enable deleted-key reconciliation before starting watch loops.
Rule watcher filtering and test wiring pkg/mcs/scheduling/server/rule/*	Rule mutations now gate patch processing; tests inject checker controllers and centralize embedded-etcd cleanup.
Metadata snapshot generations pkg/mcs/resourcemanager/server/*	Metadata loads and API writes use generation-aware state, cache reconciliation, and locked service-limit updates.

Estimated code review effort: 5 (Critical) | ~100 minutes

Sequence Diagram(s)

sequenceDiagram
  participant MetadataWatcher
  participant Manager
  participant KeyspaceManager
  participant ResourceGroupCache
  MetadataWatcher->>Manager: Begin metadata snapshot
  MetadataWatcher->>KeyspaceManager: Apply generation-aware metadata updates
  Manager->>ResourceGroupCache: Coordinate concurrent API writes
  MetadataWatcher->>Manager: Finish snapshot
  Manager->>KeyspaceManager: Reconcile entries absent from snapshot

Loading

Possibly related PRs

• tikv/pd#10259: Provides the metadata watcher scaffold extended here with generation-aware load handling.
• tikv/pd#10938: Also changes embedded-etcd setup in pkg/mcs/scheduling/server/rule/watcher_test.go.

Suggested labels: contribution, ok-to-test

Suggested reviewers: jmpotato, lhy1024, amoebaprotozoa, niubell

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	Several metadata-reconciliation and resourcemanager changes appear unrelated to #11032's LoopWatcher compaction fix.	Split the unrelated metadata/resourcemanager changes into separate PRs, keeping this one focused on LoopWatcher compaction recovery.
Docstring Coverage	⚠️ Warning	Docstring coverage is 8.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately describes the main LoopWatcher compaction fix.
Description check	✅ Passed	The PR description matches the template, includes the issue number, change summary, tests, and release note.
Linked Issues check	✅ Passed	The main etcd compaction and snapshot-consistency fixes satisfy #11032's requirements.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 88.76404% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.27%. Comparing base (39b6220) to head (b837a0c).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #11033      +/-   ##
==========================================
+ Coverage   79.25%   79.27%   +0.02%     
==========================================
  Files         541      542       +1     
  Lines       76037    76382     +345     
==========================================
+ Hits        60262    60554     +292     
- Misses      11534    11566      +32     
- Partials     4241     4262      +21     

Flag	Coverage Δ
unittests	79.27% <88.76%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rleungx]

/retest

[coderabbitai]

🧹 Nitpick comments (1)

pkg/utils/etcdutil/etcdutil.go (1)

378-381: 🩺 Stability & Availability | 🔵 Trivial | ⚖️ Poor tradeoff

Consider enforcing the "call before StartWatchLoop" contract at runtime.

SetLoadHooks/SetReconcileDeletedKeys mutate preLoadFn/postLoadFn/reconcileDeletedKeys/loadedKeys without synchronization, relying solely on the doc comment for callers to invoke them before StartWatchLoop. If ever called after the watch goroutine starts, this is a data race. Consider guarding with a started flag (panic/log on misuse) instead of doc-only enforcement.

Also applies to: 933-938, 943-946

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/utils/etcdutil/etcdutil.go` around lines 378 - 381, Enforce the
pre-StartWatchLoop contract at runtime for SetLoadHooks and
SetReconcileDeletedKeys: add a synchronized started state, mark it when
StartWatchLoop begins, and reject or clearly report calls made afterward before
mutating preLoadFn, postLoadFn, reconcileDeletedKeys, or loadedKeys. Ensure all
reads and writes of the started state and these hook-related fields use the same
synchronization to prevent races.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pkg/utils/etcdutil/etcdutil.go`:
- Around line 378-381: Enforce the pre-StartWatchLoop contract at runtime for
SetLoadHooks and SetReconcileDeletedKeys: add a synchronized started state, mark
it when StartWatchLoop begins, and reject or clearly report calls made afterward
before mutating preLoadFn, postLoadFn, reconcileDeletedKeys, or loadedKeys.
Ensure all reads and writes of the started state and these hook-related fields
use the same synchronization to prevent races.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: e0d5e992-01dd-4b9c-b0c4-fb2975f4e375

📥 Commits

Reviewing files that changed from the base of the PR and between 03ed7ea and 05fde59.

📒 Files selected for processing (10)

• pkg/mcs/resourcemanager/server/keyspace_manager.go
• pkg/mcs/resourcemanager/server/manager.go
• pkg/mcs/resourcemanager/server/manager_test.go
• pkg/mcs/resourcemanager/server/metadata_watcher.go
• pkg/mcs/resourcemanager/server/metadata_watcher_test.go
• pkg/mcs/resourcemanager/server/resource_group.go
• pkg/mcs/resourcemanager/server/service_limit.go
• pkg/mcs/scheduling/server/rule/watcher_test.go
• pkg/utils/etcdutil/etcdutil.go
• pkg/utils/etcdutil/etcdutil_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• pkg/mcs/scheduling/server/rule/watcher_test.go

[ti-chi-bot]

@rleungx: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-error-log-review	b837a0c	link	true	/test pull-error-log-review

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.