Conversation
The three start_server blocks added by #3940 run with io-threads 4 and io-threads-always-active yes. In that mode idle IO threads busy-spin on an already-unlocked mutex (src/io_threads.c:423-425) with no syscall and no yield, and waitForClientIO spins the same way on the main thread (src/io_threads.c:122-137). Under valgrind every thread is serialized onto one core, so the spinners starve the main thread and the test body makes almost no progress. Measured locally with valgrind 3.19.0: the same 200 ZADDs plus 50 pipelined ZSCOREs takes 206 ms in a server without io-threads and 20531 ms with io-threads 4 io-threads-always-active yes, a 100x gap. In Daily CI that is enough to blow the 2400s per-test timeout and abort the rest of the tests/unit/type shard. tests/unit/io-threads.tcl:61 and :142 already carry valgrind:skip for the same reason. All three blocks need it in one change, because the timeout aborts the run before hash and set are reached, so skipping only zset just moves the stall. Signed-off-by: Madelyn Olson <matolson@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The three
start_serverblocks that #3940 added for nested prefetch run withio-threads 4 io-threads-always-active yes, and in that mode idle IO threads busy-spin on an already-unlocked mutex at src/io_threads.c:423-425 whilewaitForClientIOspins on the main thread at src/io_threads.c:122-137, both with no syscall and no yield. Valgrind serializes every thread onto one core, so the spinners starve the main thread and the test body barely progresses: the same 200ZADDs plus 50 pipelinedZSCOREs cost 206 ms in a server without io-threads and 20531 ms with io-threads under valgrind, a 100x gap that blows the 2400s per-test timeout in Daily CI and aborts the rest of thetests/unit/typeshard. This tags all three blocksvalgrind:skip, which tests/unit/io-threads.tcl:61 and :142 already do for the same reason.All three need it in one change, because the timeout aborts the run before hash and set are reached, so skipping only zset moves the stall rather than removing it. The busy-spin itself is intentional for
io-threads-always-activeand is only pathological under valgrind's serialized scheduler, so there is nothing to fix in src/.Fixes #48
Details
Reproduction
Local, valgrind 3.19.0, 32-core box, built with
make valgrind.The forcing knob is
--timeout. The stall is not a hang, so the default 2400s eventually completes locally (109632 ms for the zset test on this box); squeezing the per-test budget to 60s turns it into the exact CI failure. 60s is still ~290x the 206 ms the identical body costs without io-threads.Before the change, 15/15 failures (3 tests,
--loops 5):After, the blocks are skipped at tests/support/server.tcl:299 and the run finishes in 5s instead of timing out:
Native coverage is unchanged, 8 tests x 5 loops:
Isolating the cause to the config, not the body
A scratch suite ran the identical body (
del, 200ZADD, 50 pipelinedZSCORE) in two servers, one plain and one with the exact overrides from tests/unit/type/zset.tcl:3290, both under valgrind pinned to one core withtaskset -c 5:Same body, same valgrind, same core. The only variable is
io-threads 4 io-threads-always-active yes.Mechanism
src/io_threads.c:422-425, reached whenever the thread has no work and no pending responses:
IOThreadsAfterSleepunlocks all of them on any event in the always-active branch at src/io_threads.c:175-183, so both calls return immediately and thewhile (1)at src/io_threads.c:364 spins.waitForClientIOat src/io_threads.c:122-137 spins on the main thread with onlyatomic_thread_fencebetween iterations.Notes
==pid==block, noERROR SUMMARY, in the CI logs or locally. src/valgrind.sup is irrelevant.d6415e766rather thanagents/unstable, becauseagents/unstableis 66 commits behind and does not yet contain #3940, so the lines this patch touches do not exist there.test-failureissue exists for these test names; searched all 159 by name fragment.This was generated by AI but verified, with love, by a human.