Enhancement Task
Related to #3577.
Background
We observed a production case on a v8.5.6-based cluster where different application-data compressibility caused significant physical disk skew between TiKV Stores.
TiKV engineering double-checked the behavior. With identical RocksDB compression settings, the compression-ratio difference is caused by the compressibility of the written data rather than a TiKV compression defect. The affected Store accumulated a substantially larger physical default CF L6 SST footprint.
At the same observation point:
|
Store A |
Store B |
| Physical Store used |
~2.4 TB |
~1.8 TB |
| Smoothed available space (A) |
~726 GiB |
~1,317 GiB |
| Logical Region size (R) |
~6.95M MiB |
~7.75M MiB |
| Region score |
~8.90M |
~9.00M |
Although Store A had substantially higher physical disk usage, its smaller logical Region size offset the higher disk-space multiplier. The two Region scores therefore remained close and PD did not generate strong outbound balancing pressure.
Current Region score behavior
For the normal-space branch of Region score v2:
F = max(50, C * (1 - low-space-ratio))
score = [1 + 256 * (ln(C) - ln(A - F + 1))
/ (C - A + F - 1)] * R
Here, C is Store capacity, A is smoothed available space, and R is logical Region size.
The disk has a nominal capacity of 3.2 TB, reported to PD as approximately 2,980 GiB. With the default low-space-ratio=0.8:
F ~= 2,980 GiB * (1 - 0.8) ~= 596 GiB
PD applies gradual disk-space pressure above this boundary, but the much stronger low-space penalty is used only when smoothed available space approaches or falls below approximately 596 GiB.
The observed scores can be approximated as:
Store A: 6.95M * 1.281 ~= 8.90M
Store B: 7.75M * 1.161 ~= 9.00M
This is mathematically consistent with the current formula, but it can delay correction of substantial physical disk skew caused by heterogeneous data compressibility.
Expected improvement
PD should apply earlier and smoother outbound balancing pressure when physical available-space divergence becomes large, before the affected Store reaches the emergency low-space boundary.
This does not necessarily mean scheduling directly by physical SST bytes, because SST files can span multiple Regions and PD does not know exactly how many physical bytes moving a Region will release. Possible directions include:
- adjusting the normal-space multiplier curve so that large available-space divergence has a stronger effect;
- incorporating a smoothed and capped Store-level physical amplification estimate; or
- adding another safeguard before the hard low-space branch.
The goal is to prevent a Store with significantly higher physical disk usage from retaining a Region score close to Stores with much more available space solely because logical Region size offsets the disk-space multiplier.
Version
Observed with PD Region score v2 behavior in a v8.5.6-based deployment.
Enhancement Task
Related to #3577.
Background
We observed a production case on a v8.5.6-based cluster where different application-data compressibility caused significant physical disk skew between TiKV Stores.
TiKV engineering double-checked the behavior. With identical RocksDB compression settings, the compression-ratio difference is caused by the compressibility of the written data rather than a TiKV compression defect. The affected Store accumulated a substantially larger physical default CF L6 SST footprint.
At the same observation point:
Although Store A had substantially higher physical disk usage, its smaller logical Region size offset the higher disk-space multiplier. The two Region scores therefore remained close and PD did not generate strong outbound balancing pressure.
Current Region score behavior
For the normal-space branch of Region score v2:
Here,
Cis Store capacity,Ais smoothed available space, andRis logical Region size.The disk has a nominal capacity of 3.2 TB, reported to PD as approximately 2,980 GiB. With the default
low-space-ratio=0.8:PD applies gradual disk-space pressure above this boundary, but the much stronger low-space penalty is used only when smoothed available space approaches or falls below approximately 596 GiB.
The observed scores can be approximated as:
This is mathematically consistent with the current formula, but it can delay correction of substantial physical disk skew caused by heterogeneous data compressibility.
Expected improvement
PD should apply earlier and smoother outbound balancing pressure when physical available-space divergence becomes large, before the affected Store reaches the emergency low-space boundary.
This does not necessarily mean scheduling directly by physical SST bytes, because SST files can span multiple Regions and PD does not know exactly how many physical bytes moving a Region will release. Possible directions include:
The goal is to prevent a Store with significantly higher physical disk usage from retaining a Region score close to Stores with much more available space solely because logical Region size offsets the disk-space multiplier.
Version
Observed with PD Region score v2 behavior in a v8.5.6-based deployment.