Bug Report
In PD microservices mode the pd (API) server forwards every client TSO request to the pd_tso primary, resolving the primary's address from an in-memory cache (servicePrimaryMap[tso]) maintained solely by the tsoPrimaryWatcher (an etcdutil.LoopWatcher). The read path, GetServicePrimaryAddr, only reads that cache; the self-heal ForceLoad() is triggered only on IsLeaderChanged stream errors, never when the cached address is simply absent.
If the watcher ever misses the PUT of a new primary — e.g. an etcd watch gap during an etcd-leader stall, where the watch delivers the DELETE of a primary key but not the subsequent PUT — servicePrimaryMap[tso] is left empty and never repopulated on the request path. getGlobalTSO and handleTSOForwarding then return ErrNotFoundTSOAddr (gRPC NotFound "not found tso address") to every client indefinitely, until the PD process is restarted (which re-runs initFromEtcd → load).
What did you do?
PD node's servicePrimaryMap[tso] was emptied by a delete with no following put and stayed empty. TiKV TSO workers got persistent NotFound "not found tso address", could not update max_ts (MaxTimestampNotSynced), and the SQL layer fell into a regionMiss/replica-selector retry storm at ~50% throughput. Restarting that single PD node cleared it instantly.
What did you expect to see?
PD should be able to find the right TSO after re-election
What did you see instead?
PD stuck with no TSO address even though TSO was elected.
What version of PD are you using (pd-server -V)?
8.5.5
Bug Report
In PD microservices mode the pd (API) server forwards every client TSO request to the pd_tso primary, resolving the primary's address from an in-memory cache (servicePrimaryMap[tso]) maintained solely by the tsoPrimaryWatcher (an etcdutil.LoopWatcher). The read path, GetServicePrimaryAddr, only reads that cache; the self-heal ForceLoad() is triggered only on IsLeaderChanged stream errors, never when the cached address is simply absent.
If the watcher ever misses the PUT of a new primary — e.g. an etcd watch gap during an etcd-leader stall, where the watch delivers the DELETE of a primary key but not the subsequent PUT — servicePrimaryMap[tso] is left empty and never repopulated on the request path. getGlobalTSO and handleTSOForwarding then return ErrNotFoundTSOAddr (gRPC NotFound "not found tso address") to every client indefinitely, until the PD process is restarted (which re-runs initFromEtcd → load).
What did you do?
PD node's servicePrimaryMap[tso] was emptied by a delete with no following put and stayed empty. TiKV TSO workers got persistent NotFound "not found tso address", could not update max_ts (MaxTimestampNotSynced), and the SQL layer fell into a regionMiss/replica-selector retry storm at ~50% throughput. Restarting that single PD node cleared it instantly.
What did you expect to see?
PD should be able to find the right TSO after re-election
What did you see instead?
PD stuck with no TSO address even though TSO was elected.
What version of PD are you using (
pd-server -V)?8.5.5