Follower read target selection should consult PD for store recovery, not only TiKV's self-reported slow score

[AndreMouche]

Background

When a TiKV store hits a transient problem (e.g. a faulty/jittery disk), PD marks it as a slow store and evicts its leaders. Follower stale-read traffic, however, can still be routed to that store while it is recovering.

Current Behavior

When client-go selects the target replica for a follower read, it relies on the slow score that the TiKV store reports about itself to decide whether the store has recovered and is eligible to serve reads again.

The problem is that a recovered self-reported slow score does not guarantee the store is actually ready to take traffic:

• The signal is local to the store and can be noisy/flapping
• It is not coordinated with PD's view of the store's recovery / leader-eviction state.

As a result, follower reads can be sent back to a not-yet-recovered store too early, hotspotting it.

Expected Behavior

Before routing a follower (stale) read to a store that was recently slow/recovering, client-go should consult PD to confirm the store has fully recovered (e.g. that PD no longer considers it a slow store / has stopped evicting its leaders), rather than relying solely on the store's self-reported slow score.

Proposed Direction

• Treat PD's store recovery state as the authoritative signal for "is this store ready to serve reads again," and gate follower-read target selection on it.
• Keep the TiKV self-reported slow score as a fast local hint, but require agreement with PD's view before sending traffic back to a recently-slow store.
• Consider a hysteresis / cool-down so a flapping slow score cannot repeatedly re-add a store as a follower-read target.

Related

• Related PD behavior: Recovery time is not expected in slow store scheduler pd#9384 — Recovery time is not expected in slow store scheduler (Recovery time is not expected in slow store scheduler pd#9384) (leaders transferred back prematurely when recovery time < jitter duration).

[LykxSassinator]

For addressing this issue, we have made a proposal as the following shows:

Design summary

The key point is to treat PD as the source of truth for recovery-time.

Today, client-go only uses TiKV-side health feedback (slow_score) to decide whether a replica can serve stale-read traffic. This is insufficient for the slow-store recovery case, because PD may still keep the
store in evicted_as_slow_store = true during the recovery window, even after TiKV-side slow score has already returned to normal.

So the design should be:

• client-go keeps two independent signals for each store:
• TiKV-side health status
• PD-side slow-store eviction status
• stale-read replica routing must allow a store only when:
• TiKV-side health is normal, and
• PD-side evicted_as_slow_store is false
• client-go must not reimplement PD's recovery-time logic locally.
• PD only needs to expose its current decision result, and client-go only consumes that result.

Refresh / timeliness strategy

To avoid relying only on periodic refresh:

• periodic PD store-state refresh remains the background fallback
• when a store transitions from slow to healthy on the TiKV side, client-go should trigger a targeted PD revalidation
• before that revalidation completes, client-go should keep the store ineligible for stale-read conservatively

This avoids the two timing problems discussed earlier:

• PD has already recovered the store, but client-go has not refreshed yet
• local cached PD state is stale and incorrectly says the store is still safe to use