A macOS shard exceeds the 30-minute job cap when repartitioning moves too much into it -- the run is progressing, not hung

[fujibee]

A macOS shard stops producing output and the job ends non-zero with no failing test. The hang samples name what is actually running, and it is not what the log's last line suggests.

From the samples on macos-latest-4 (45 taken), the processes alive at the end:

bats (shard launch) -> bats-exec-suite -> bats-exec-file  20 tests/test_remote_setup_doc.bats
                                          bats-exec-test  test_remote_setup_doc

No node, no sleep, no sqlite3 — only bats itself and tests/test_remote_setup_doc.bats. The shared pipes are held by that bash group and nothing else.

Correction to the original description

This issue was first filed saying the shard stops "after the same test", naming Stage-1 sync engine protocol and security boundaries, because that was the last line in the log. That reading was wrong and so was the title.

bats --print-output-on-failure buffers a file's output until the file finishes. The last line printed therefore names the last completed file, not the one being executed. Execution was twenty files in; the log was showing something from earlier. Two people read that line as a position, and it is not one.

The pipe theory that followed from it is also refuted, by measurement: bats tests/test_remote_sync_engine.bats alone finishes green in 57 seconds under load 403, and running it together with test_codex_monitor.bats finishes green in 45 seconds with no leaked node process. Whatever this is, it is not that file and not a leaked child holding a pipe.

What the change on the branch actually does

tests/test_remote_setup_doc.bats is untouched by the branch this appeared on — the diff is empty for it. What changed is where it runs:

	shard
base	2 of 4
the branch	4 of 4

shard-tests.sh recomputes the partition from the tree on every run, so adding three .bats files repartitions everything. That also explains why the hung shard number moved between heads — 2/4, then 3/4, then 4/4. The shard number was never the variable; it was a consequence of repartitioning.

So the branch exposes this rather than causing it. Whether it is latent in the file itself or an interaction with whatever now sits beside it is the open question, and a standalone run of that file is being measured now.

Acceptance

• The hang is reproduced with a named cause: either the file hangs on its own, or a specific co-resident makes it hang.
• Whatever holds it is identified. Only bats processes were alive, so this is not a leaked child — something inside the test is blocked.
• A hung shard says so in its own output. Reconstructing "where it actually was" from a downloaded artifact, against a log line that names somewhere else, is how this issue was filed wrong the first time.

[fujibee]

The hang instrumentation captured it, and the file named in this issue is not where it stops.

From the hang-samples artifact of a cancelled macOS shard (30-minute timeout, twice in a row on the same commit):

bats-exec-test  tests/test_terminal_registry.bats
                "herdr hint place_right: gates the move, then executes new-tab and split right"  (test 40)
  fd 1,2,4 -> .../bats-run-XXXXXX/test/40.out   0 bytes

So the stuck test is in test_terminal_registry.bats, not test_remote_setup_doc.bats. The shard reaches ok 832 — every other file in its list finishes — while this one file is still running under parallel execution, which is why the symptom reads as "the shard hangs" rather than "a test fails".

The captured output file is zero bytes, so it stops at or before the first command that would print, not partway through the assertions.

The test installs a fake layout helper and then calls the arrange path twice:

_install_fake_herdr_layout
agmsg_terminal_load herdr
terminal_describe | grep -q ...
run terminal_arrange wA:p1 place_right wA:p2

Worth checking first, in this order:

1. whether anything on that path reads standard input — a helper that blocks on stdin never returns under a harness that does not redirect it, and produces exactly this signature (no output, no exit);
2. whether the fake helper is actually the one resolved, or whether a real binary is reached on a runner that does not have it;
3. whether this only reproduces under parallel execution, since the serial runs of the same file elsewhere in the matrix pass.

Practical consequence while this is open: the integration branch cannot get a green CI run — the aggregate bats check is a failure because this one shard is cancelled, and a rerun reproduces it on the same shard. Any merge gate that requires CI success on a head SHA is blocked by this, independent of what the change under test does.

[fujibee]

Retitled again, and the second edit restores something the first one removed.

My first edit replaced the file name with the one the hang sample actually shows. That part was measured and stands. But it also dropped "when repartitioning moves it" — and that clause carries the condition, which the sample cannot show on its own: the same file passes in other matrix legs, so what changes is which files share its shard.

Static analysis of the file in isolation has now been done thoroughly and found no deterministic hang mechanism: setup copies a script tree and sources a library (measured at 0.09s and 0.05s), every test isolates into its own mktemp -d, the registry lock is per-team inside that directory, and every terminal operation is served by an installed fake. That is a solid negative — about the file alone.

What has not been examined is the shard as a set. The co-resident files, in the order the sample records them:

test_terminal_registry        <- the one that stops
test_remote  test_install  test_instance_id  test_remote_sync  test_doctor
test_watch  test_self_identity  test_codex_bridge_launcher
test_migrate_team_store  test_self_name  test_hash  test_resurrect_panes
test_ci_sharding  test_bin_agmsg  test_antigravity_transport
test_perf_harness  test_setup  test_remote_curl_stderr_get
test_printed_command_paths  test_sync_cipher  test_cmdline_path_match
test_resume_seat_guard  test_bump_app_version  test_placement_collisions
test_remote_sync_reject_envelope  test_legacy_mirror  test_apply_fail_stderr
test_engine_inherited_fds

Directories are isolated; several other things are not — PATH, the process table, ports, anything real under the agent home, and file descriptors inherited from the parent. Four of these files start processes. Two of them (test_apply_fail_stderr, test_engine_inherited_fds) exist specifically because inherited descriptors have caused trouble before, which makes their presence in this shard worth a look rather than a coincidence.

The captured output file being zero bytes narrows it further: nothing had been printed when the sample was taken, so it stops in setup or before the harness starts the test body, not partway through assertions.

Reproduction has to preserve the condition: the same file set, in the same order, as one shard. Running the file by itself proves nothing, because that already passes.

Practical consequence, unchanged: while this is open the integration branch cannot produce a green run — the aggregate check fails because this one shard is cancelled, and a rerun reproduces it on the same shard.

[fujibee]

Third retitle, because the first two named a bystander. The mechanism is now identified and it is the same class as #575.

What is actually happening: something started by a test outlives it and keeps the harness's inherited descriptors open. bats cannot finish while those pipes have a writer — its formatter is still reading — so the shard sits at the timeout even though every test has already reported ok. The run reaching ok 832 and then stalling is exactly that shape: the suite is done, the process group is not.

Confirmed independently in the captured sample. Two pipes, on fds 143 and 146, are held by:

bash     100+ processes
Python   6 processes

They are inherited broadly down the tree, so any single survivor keeps the shard alive. This is why the symptom moves when sharding is repartitioned: it depends on whether a file that leaves something running lands in that shard, not on the file the sample happens to catch mid-wait.

The test I named in the previous two titles is not the cause. test_terminal_registry.bats test 40 appears in the sample as the test in flight when the snapshot was taken, with a zero-byte output file. I read a bystander as the culprit and put it in the title twice. Exhaustive static analysis of that file found no hang mechanism — that negative was correct and I was pointing it at the wrong question.

What this means for anyone reading the sample: the process the sampler catches is not evidence of a stuck test. The evidence is the descriptor table — who still holds the harness pipes — and the fact that the test log is complete. Those two together say "finished, cannot exit", which is a different failure from "stopped partway".

Practical consequence, unchanged and now attributable: while any file in the matrix can leave a process holding those descriptors, the integration branch cannot produce a green run, because the aggregate check fails on whichever shard the survivor lands in. Fixing this is a prerequisite for any merge gate that requires CI success on a head SHA.

[fujibee]

Fourth correction, and this one overturns the previous three including my own diagnosis. There is no hang. The shard is running normally the entire time and is cancelled at the cap while still working.

The artifact contains 46 samples at 30-second intervals. Reading them as a series rather than grepping the whole file:

07:35  test_terminal_registry
07:36 - 07:44  test_remote              (about nine minutes on one file)
07:45  test_install
07:46 - 07:49  test_remote_sync
07:51  test_doctor
07:53  test_watch
07:56  test_self_identity
07:57 - 07:58  test_codex_bridge_launcher   <- still starting new tests when cancelled

A different test is executing in nearly every sample, and new bats-exec-test processes appear right up to the final one, twenty-three minutes in. Nothing is waiting on anything. The descriptors held throughout belong to the harness itself — the sampler, the CI step shell, and the bats core processes — which is what an ordinary in-progress run looks like.

So the original title was right and I should not have changed it. "when repartitioning moves it" is the whole diagnosis: this shard's file set takes longer than the job allows, and which files land together is what decides whether one exceeds it. The condition was in the title from the start; I replaced it with a file name, twice, then with a mechanism, once, and all three were wrong.

Corrections to what I posted earlier in this thread:

• test_terminal_registry.bats test 40 is not the culprit. It is simply the test running when one snapshot was taken.
• The mock remote server is not the culprit either. It appears in six of forty-six samples, all between 07:37 and 07:44, and is entirely absent for the whole period I described as "the stall" — including every sample after 07:45. I found it by grepping the file for process names without checking when each appeared, which is the same mistake as the test above, made twice in one investigation.
• ok 832 followed by silence is not evidence of stopping. The formatter buffers, so the count reflects what had been flushed, not where execution had reached.

What still stands, separately from this issue: a helper that outlives its test and holds inherited descriptors is a real hazard, and closing them at startup is worth doing on its own terms. It is not what cancels this shard, and a change making that hygiene fix should not claim to close this.

The measurement this issue actually needs is per-file wall time on the macOS leg, and the balance the partitioner produces across shards — test_remote.bats alone accounting for roughly a third of the budget is the visible starting point.

[fujibee]

Timing data, which settles it and also shows why the green run that just happened is not evidence of a fix.

macOS shard wall time, two runs of the same suite:

                        1/4     2/4     3/4     4/4
run that was cancelled  16.4    12.3    19.6    30.2  (cancelled at the cap)
the next run            17.8    12.8    27.8    24.1  (all green)

The job cap is 30 minutes. Every macOS shard finishes somewhere between 12 and 28 minutes, so the suite normally runs at 40-95% of its budget, and whichever shard draws the heaviest slice that day goes over. Last time that was 4/4 at 30.2; this time 4/4 came in at 24.1 and 3/4 landed at 27.8 — two minutes of headroom. The next cancellation is more likely to be 3/4 than a repeat of 4/4.

That is the same thing the original title said: it moves with repartitioning. It is not a defect in any one test, and a run coming back green does not demonstrate that anything was repaired — the margin is thin enough that ordinary variance crosses it in both directions.

Consequences for how this gets closed:

• Treating a green run as proof would be wrong here. Any change wanting to claim this issue needs shard times as evidence — the distribution moving down, not one run fitting.
• The lever is the partition and the per-file cost, not a single culprit. test_remote.bats occupied roughly nine minutes of one shard on its own, which is where the balance question starts.
• Raising the cap would stop the cancellations and hide the trend; worth doing only alongside a measurement that keeps the distribution visible.

Correcting my own record in this thread once more: I posted three different culprits here — a test, then a mechanism, then a helper process — and none of them is why this shard is cancelled. Each came from reading a snapshot as though it described the whole run. The artifact is a time series of 46 samples and answers the question directly when read as one.

[fujibee]

The partitioner's own header contains the baseline, and comparing it against today says what changed.

shard-tests.sh documents a measurement taken when the sharding was designed:

whole suite on macOS            860 s   (14.3 min)
count-balanced quarters         125 s / 298 s / 366 s / 71 s

Two runs measured today, same four shards:

                        1/4      2/4      3/4      4/4     sum
cancelled run          16.4     12.3     19.6     30.2    78.5 min
next run               17.8     12.8     27.8     24.1    82.5 min

The heaviest quarter was documented at 366 s — six minutes. It is now between twenty and twenty-eight. Total work across the four shards is roughly 5.8x the figure the partition was tuned against. (The baseline is quoted as the whole suite on macOS; even allowing for differences in how it was taken, the gap is not a measurement subtlety.)

That reframes the issue. The partition is doing what it was built to do, with the weight it documents — @test count as a loose proxy, greedy longest-processing-time first, plus a pinned-apart list for files whose cost is mostly waiting. What has moved is the suite: it grew until every quarter sits at 40-95% of a 30-minute cap, and then whichever quarter draws the heaviest slice goes over. That is why it tracks repartitioning and why a rerun can come back green without anything being fixed.

The header also anticipated this. It says test count is a loose proxy, records that a checked-in table of per-file seconds would buy about 150 s more but "goes stale silently", and notes the floor is the slowest single file. Those trade-offs were priced against a suite that finished in fourteen minutes; at eighty-plus minutes of total work they need repricing, and so does the assumption that a stale weight table is the worse risk.

So the useful next measurements are not about any one test:

• per-file wall time on the macOS leg, emitted into the run's own artifact so it is available without correlating log timestamps by hand;
• the same numbers across several runs, so the comparison is a distribution rather than one sample — the margin here is two minutes, and single runs cross it in both directions;
• the largest single file's cost, since no partition beats it. test_remote.bats took about nine minutes of one shard, which on its own is one and a half times the documented heaviest quarter.

Raising the cap would stop the cancellations without touching any of this, and would hide the trend that produced them. If it is done to unblock the branch, it is worth pairing with the per-file emission so the growth stays visible.

[fujibee]

The answer was already written in tests.yml, next to the timeout it sizes, including the test for telling the two cases apart:

NOT "a timeout is a genuine hang" (#848): a cancellation here is a wall-clock cap being reached, and a shard that is still passing tests right up to the cap looks identical to one that is actually stuck. #857 hit this on macOS at 25m29s with all 335 of its tests ok and the last one landing at 24m53s — a shard that finished normally a heartbeat too late, not a hang. Distinguishing the two needs the job's own log (does the last ok land seconds before cancellation, or does output stop cold with time left on the cap), not this number.

That is the same confusion this thread went through, with the discriminator spelled out in advance. It also records what the cap was sized against:

measured 2026-08-19 (macOS)   shards 890 / 824 / 540 / 1230 s
heaviest                      1230 s, with 270 s of the 1500 s cap left (18%)

Today the heaviest shard is 27.8 minutes — 1668 s — against a 1800 s cap. The 18% margin is gone, and this is the third recorded occurrence of the same symptom (#848, #857, now this).

So the issue is not a defect to find. It is a budget that was sized correctly against a suite which has since grown past it, twice over:

partitioner header, at design time    whole suite 860 s, heaviest quarter 366 s
workflow header, 2026-08-19           heaviest shard 1230 s
today                                 heaviest shard 1668 s, four shards 4950 s total

The existing mechanism for this is already in the tree too: #847's pin in shard-tests.sh removes disproportionate files (722 s and 380 s at the time) from the ordinary weighted pool, on the reasoning that @test count is blind to files whose cost is waiting rather than counting. That pin list was populated from a measurement now thirteen months stale in suite-growth terms.

What this needs, in order:

1. Emit per-file wall time into the run's own artifact. Right now the discriminator the workflow describes — where the last ok lands relative to the cancellation — requires correlating log timestamps by hand, which is why three people in this thread reached for other explanations first.
2. Re-measure the pin list against current times, since it is what protects the balance and it was derived from numbers that have since roughly doubled.
3. Only then decide between re-pinning, re-weighting, more shards, or a larger cap. Raising the cap alone would clear the symptom and remove the signal that produced this thread.

For the record, corrections to my own posts here: I named three culprits (a test, an inherited-descriptor mechanism, a helper process) and none was the cause. Each came from reading a snapshot as if it described the whole run, and the workflow already said that reading a cancelled shard that way is exactly the trap.

[fujibee]

Root cause, and shard-tests.sh names it in advance as the thing to watch.

Measured @test weight per shard on a run that was cancelled:

1/4 = 281     2/4 = 302     3/4 = 832     4/4 = 831  (cancelled)

Two shards carry 2.9x what the other two do. That is not the balancer failing — it is the pin seeding working exactly as written, against a suite three times the size it was tuned on.

A pinned file is seeded at one shard's average share, total_tests / total, because its @test count is the number the pin exists to distrust. With the suite at 2246 tests over 4 shards that seed is 561. Both pinned shards therefore start 561 "full" and the ordinary weighted pass adds only 281 and 302 of real tests to them. Everything else — the whole remainder — lands on the two shards that hold no pin, at 832 and 831.

The file's own comment states this failure mode and its bound:

The failure mode to watch is over-reservation: a pinned file that is actually light leaves its shard underfilled and pushes the others up. That is bounded while the pins are few relative to total — at 2 pins in 4 shards the worst case is the remaining work over two shards, which the numbers above clear with room.

"The remaining work over two shards" is now 830 tests each, and it no longer clears with room. The numbers it cleared with were taken when the suite was about a third of its current size — the partitioner header records the whole suite at 860 s then, against four shards summing to 4950 s now.

So the seed is the lever, and it has a property worth naming: it scales with the suite, while the pinned file does not. A file pinned because it cost 722 s reserves a share that has tripled since, purely because other files were added around it. The reservation was chosen as "the honest reading of we cannot weigh this one", which was right when a share and the file's real cost were comparable; they no longer are.

Three directions, in increasing order of change:

1. Re-check whether those two files still need pinning at all. Their @test counts are unchanged, but per-test cost may have moved — if either is now ordinary, unpinning it returns a whole shard to the pool.
2. Bound the seed by the pinned file's measured cost rather than by the suite average, so it stops growing when the file does not. This reintroduces a measured number, which the header deliberately avoided as going stale silently — but a bound that is only ever an upper limit fails safe in the direction the current formula fails unsafe.
3. More shards, which dilutes "the remaining work over two shards" directly. Cheapest to try, does nothing about the scaling property.

Not a fix, but worth stating: this issue is now blocking merges concretely. A PR cleared for landing today carries 24 green checks and one cancelled macOS shard, and cannot merge because the aggregate check is a failure. Every gate that requires CI success on a head SHA is behind this.