Found while triaging a false "regression" on PR #10530. test-files/test_gap_cron_cronjob.ts waits up to 10 s for a
* * * * * * CronJob to tick twice and prints manual ticked at least twice: <bool> / auto ticked at least twice: <bool>. Node and Perry each run the program under their own deadline, so a slow moment on EITHER side (including the
Node oracle) flips one boolean and the harness reports an output mismatch — attributed to whatever PR happened to be
running.
Evidence
- CI: across the last 40
test.yml runs, 20 gap-suite jobs ran this test; it passed in 19 and failed in 1 (a PR whose
change provably cannot affect it).
- That PR's proof: compiling the fixture with
--trace llvm on the base commit and on the PR build produces
byte-identical IR (5,438 lines; the only difference is the embedded absolute compile-directory path). cron takes
the native-binding route (crates/perry-ext-cron), so the only TypeScript compiled is the test itself.
- Local reruns on both builds: 12/12 pass each, 24 concurrent instances identical, 5/5 pass pinned to one core against
three busy loops.
- Failure signature: both exit 0, the first line matches, a later line differs — i.e. a flipped boolean, not a crash.
Impact
A timing-dependent fixture in the ratchet gate makes the gap suite an unreliable signal: a green PR can be blocked and
a real regression can be dismissed as "probably the cron flake". It cost a full investigation cycle on #10530.
Suggested fix
Make the assertions independent of wall-clock progress, for example:
- drive the job deterministically (fire the callback via the API instead of waiting on a real schedule), or
- assert only facts that do not depend on how many ticks happened within a deadline (e.g. that a never-started job
never fired, that start()/stop() toggle a state flag), printing counts only as >= 1/0, or
- give both sides the same generous deadline AND make the printed value insensitive to the deadline being hit
(e.g. wait indefinitely for the first two ticks with a hard timeout that fails loudly rather than printing false).
The same review applies to other timer-driven gap fixtures (test_gap_backoff_options, node-cron variants).
Found while triaging a false "regression" on PR #10530.
test-files/test_gap_cron_cronjob.tswaits up to 10 s for a* * * * * *CronJob to tick twice and printsmanual ticked at least twice: <bool>/auto ticked at least twice: <bool>. Node and Perry each run the program under their own deadline, so a slow moment on EITHER side (including theNode oracle) flips one boolean and the harness reports an output mismatch — attributed to whatever PR happened to be
running.
Evidence
test.ymlruns, 20 gap-suite jobs ran this test; it passed in 19 and failed in 1 (a PR whosechange provably cannot affect it).
--trace llvmon the base commit and on the PR build producesbyte-identical IR (5,438 lines; the only difference is the embedded absolute compile-directory path).
crontakesthe native-binding route (
crates/perry-ext-cron), so the only TypeScript compiled is the test itself.three busy loops.
Impact
A timing-dependent fixture in the ratchet gate makes the gap suite an unreliable signal: a green PR can be blocked and
a real regression can be dismissed as "probably the cron flake". It cost a full investigation cycle on #10530.
Suggested fix
Make the assertions independent of wall-clock progress, for example:
never fired, that
start()/stop()toggle a state flag), printing counts only as>= 1/0, or(e.g. wait indefinitely for the first two ticks with a hard timeout that fails loudly rather than printing
false).The same review applies to other timer-driven gap fixtures (
test_gap_backoff_options, node-cron variants).