Deflake "Throttling protects a replica above the soft COB limit"

[madolson]

The test freezes the replica, pipelines 30 x 1MB SETs, then asserts the replica is still connected. That assertion only holds while the throttler's soft-limit exemption is in force, and the exemption expires 120 seconds after the COB crosses the soft limit (src/throttle_repl.c:136-141). The throttler rate-limits ops per second and, with the replica frozen, drives that rate toward zero with no floor, so a write phase sized by a fixed op count has no time bound and outlasts the exemption on valgrind and Fedora TLS. This bounds the write phase by the COB reaching the state under test instead, then holds there for three seconds so the soft-limit second boundary is still crossed.

Fixes #30

Details

Problem

tests/integration/throttle-repl.tcl:143 sets client-output-buffer-limit "replica 1073741824 1048576 0". With soft_limit_seconds == 0, checkClientOutputBufferLimits() kills the replica as soon as one server.unixtime second ticks past the first over-limit observation (src/networking.c:6719-6729). The only thing that keeps it alive is src/networking.c:6738:

    if (soft && !hard && throttleRepl_isClientExemptFromCobLimits(c)) return 0;

and that exemption is bounded at src/throttle_repl.c:138:

    time_t elapsed = server.unixtime - c->obuf_soft_limit_reached_time;
    if (elapsed > 4 * STEADY_STATE_CONVERGENCE_SECS) return false;

STEADY_STATE_CONVERGENCE_SECS = 30 (src/throttle_repl.c:20), so 120 seconds. The replica is SIGSTOPped for the whole test, so the throttler can never converge and the exemption always expires eventually. The old write phase at tests/integration/throttle-repl.tcl:158-162 had no time bound, so whether the assertion held depended on machine speed.

Reproduction

The forcing knob is the op count of the write phase, not its byte count. throttle_adjustRate() limits ops per second (src/throttle.c:266-274), so raising the byte count changes nothing: 300 x 1MB completes in 3 seconds locally. Raising the op count for the same 30MB starves the phase.

Applied to the unmodified test at d6415e766:

-            set value_size [expr {1 * 1024 * 1024}]
-            set num_writes 30
+            set num_writes $::repro_ops
+            set value_size [expr {30 * 1024 * 1024 / $num_writes}]

$ REPRO_OPS=100000 ./runtest --single integration/throttle-repl
[ok]: Steady-state throttle happy case (1225 ms)
REPRO: write phase start 1789533866 100000 x 314B
REPRO: write phase end   1789533987
[err]: Throttling protects a replica above the soft COB limit in tests/integration/throttle-repl.tcl
replica was disconnected while above soft but below hard COB limit

The write phase took 121 seconds, one second past the cap, and the replica was dropped by design. A 125 second pause_process of the primary inserted at the same point reproduces the identical message, confirming wall-clock time in the window is the whole mechanism.

Testing

Before, 5 loops, unmodified test plus the REPRO_OPS knob:

$ REPRO_OPS=100000 ./runtest --single integration/throttle-repl --loops 5
[err]: Throttling protects a replica above the soft COB limit in tests/integration/throttle-repl.tcl
replica was disconnected while above soft but below hard COB limit
... x5 ...
Test Summary: 30 passed, 5 failed

After, 5 loops, same knob:

$ REPRO_OPS=100000 ./runtest --single integration/throttle-repl --loops 5
Test Summary: 35 passed, 0 failed
\o/ All tests passed without errors!

The test is still not vacuous. Adding $primary config set repl-throttling-enabled no immediately before the new dwell loop makes it fail, so the dwell really does depend on the exemption:

REPRO: write phase end   1789535034 above_soft 1
[err]: Throttling protects a replica above the soft COB limit in tests/integration/throttle-repl.tcl
replica was disconnected while above soft but below hard COB limit

clientsCron re-runs the limit check every cycle (src/server.c:1315), which is why a passive dwell is enough to catch the disconnect.

Rejected alternative

Restoring a large soft_limit_seconds (the pre-#4612 value was 10000) also makes the test pass, but it makes it vacuous: the base soft-limit kill can then never fire inside a test, so the test would pass whether or not the exemption exists.

Out of scope

While one replica is frozen the throttler drives the primary's write rate to ~0 for up to 120 seconds and then drops the replica anyway, because throttle_adjustRate() applies no floor on decrease (src/throttle.c:266-274). That is a design question for the product, not a test issue.

This was generated by AI but verified, with love, by a human.