Summary
MPS readiness currently signals that some config has been applied, but not that the expected config has been applied. On a config change this leaves a window where the device plugin can advertise resources for the new config while the MPS control daemon is still enforcing the old one.
Follow-up to #1946, which gates the startup race (control pipe healthy before the first config is applied). This issue tracks the remaining config-change race.
The race
The device plugin and the MPS control daemon are separate processes that both read the same config. On a config change A → B they reload independently, not atomically:
- Config changes A → B.
- The device plugin observes B and calls
checkDaemonReady().
- The MPS daemon has not reprocessed yet:
.ready is still the marker from A, and the control pipe is healthy (serving A).
checkDaemonReady() returns true, so the plugin advertises resources per B while MPS is still enforcing A's per-device limits (pinned memory, active thread percentage).
The mismatch self-converges once the MPS daemon reprocesses the update, so it is not a correctness disaster, but there is a window where advertised capacity does not match what MPS actually enforces.
Proposed approach
Make readiness confirm the expected config, not just the presence of .ready:
- The MPS control daemon stamps an identity of the applied MPS config (e.g. a hash of the relevant sharing settings) into, or alongside, the
.ready file when it finishes applying a config.
- The plugin's
checkDaemonReady() compares that identity against the hash of its own current config and treats a mismatch as not-ready, so it waits (via the existing PollUntilContextTimeout loop) until MPS has applied the matching config.
This closes the window deterministically instead of relying on convergence.
Notes
Summary
MPS readiness currently signals that some config has been applied, but not that the expected config has been applied. On a config change this leaves a window where the device plugin can advertise resources for the new config while the MPS control daemon is still enforcing the old one.
Follow-up to #1946, which gates the startup race (control pipe healthy before the first config is applied). This issue tracks the remaining config-change race.
The race
The device plugin and the MPS control daemon are separate processes that both read the same config. On a config change A → B they reload independently, not atomically:
checkDaemonReady()..readyis still the marker from A, and the control pipe is healthy (serving A).checkDaemonReady()returns true, so the plugin advertises resources per B while MPS is still enforcing A's per-device limits (pinned memory, active thread percentage).The mismatch self-converges once the MPS daemon reprocesses the update, so it is not a correctness disaster, but there is a window where advertised capacity does not match what MPS actually enforces.
Proposed approach
Make readiness confirm the expected config, not just the presence of
.ready:.readyfile when it finishes applying a config.checkDaemonReady()compares that identity against the hash of its own current config and treats a mismatch as not-ready, so it waits (via the existingPollUntilContextTimeoutloop) until MPS has applied the matching config.This closes the window deterministically instead of relying on convergence.
Notes
.readygating introduced there).