Observation: join concurrency test failed once on a shard the PR did not touch (test_team.bats:111, #141)

[fujibee]

Recording a single observation so the next one can be counted. This is not a flakiness claim —
one failure is not evidence of anything, and calling a test flaky from a count has already cost us a
withdrawn issue this week. It is here so that whoever sees it a second time has a first.

PR #797, run 32877... , bats (macos-latest 1/4)
  not ok 14 join: concurrent joins to the same team do not lose registrations (#141)
  tests/test_team.bats:111

It is not caused by that PR, established three ways rather than assumed:

1. grep -rln "_wait_pidfile" tests/ → tests/test_watch.bats only. The PR widens that helper's
   window; nothing outside that file can see it.
2. .github/scripts/shard-tests.sh <i> 4 tests for i in 1..4 →
   test_team.bats is in shard 1, test_watch.bats is in shard 4. The changed file and the
   failing file are not in the same job.
3. The PR's diff is that one file.

It is also not currently visible on main. Checking every failing main run in the last ten
(gh run list --workflow tests.yml --branch main then --log-failed | grep "not ok "), the failures
present are the two #937 launcher tests and the pair-release test now correctly naming #983. This
test does not appear.

So: a concurrency test failed once, on a shard nothing in the PR touches, on a platform, and has not
been seen on main in the recent window. That is the whole of what is known.

What would make this actionable, for whoever hits it next:

• a second occurrence, with the run and shard, so there is a rate rather than an event
• whether it recurs on the same OS (this one was macOS) or crosses platforms
• the assertion that failed, not just the test name — test_team.bats:111 is a concurrency test, and
  which invariant it lost (a registration dropped? a lock not taken?) narrows it far more than the
  fact that it failed

[fujibee]

Second sighting — and it retires half of what this record said.

main, run 32944981464 (0d08381b), 08-26 07:53Z
  bats (ubuntu-latest 1/4)
  not ok 14 join: concurrent joins to the same team do not lose registrations (#141)

When this was filed the second half of the entry read: "It is also not currently visible on main",
based on every failing main run in the previous ten. That is no longer true. This one is on
main, with no pull request involved, on a different platform and a different shard from the first
sighting.

1st   PR #797,  bats (macos-latest 1/4)   — a PR touching only tests/test_watch.bats
2nd   main,     bats (ubuntu-latest 1/4)  — no PR

What changes. The first sighting needed three separate arguments to show it was not caused by
that PR — the helper it widened is used in one file, the changed file is in shard 4 while
test_team.bats is in shard 1, and the diff was that one file. None of that is needed any more:
a main occurrence establishes it directly. The careful separation in the original entry was correct
and is now redundant, which is the good outcome — it means nothing was mis-attributed on the way here.

What does not change. Two occurrences is a count, not a rate, and this is still not a flakiness
claim. What it now is: a concurrency test that fails on main, so it belongs to whoever owns
test_team.bats:111 rather than to any branch author.

Still the most useful next thing, unchanged from the original entry: the assertion that failed,
not the test name. join: concurrent joins to the same team do not lose registrations can fail
because a registration was dropped or because a lock was not taken, and those are different defects.
The log gives the test name and nothing else.

One note on the run this came from: it is a post-fix main run being counted for #984, and neither
of that issue's launcher tests appears in it. The two are unrelated failures that happened to share a
run.

[fujibee]

Second and third observations, 2.3 hours apart on 2026-09-13:

• run 34733371402 (a PR branch, 02:34Z): join: concurrent joins to the same team do not lose registrations (#141) failed the same way, on a shard whose files the PR did not touch.
• run 34738909849 (push to main at 4851c63, macOS 5/5): same test, same failure line -- for i in "${pids[@]}"; do wait "$i"; done at tests/test_team.bats:127. One of the twelve concurrent join.sh calls exited non-zero; its stdout/stderr go to /dev/null, so the log does not say which one or why.

A sweep of the 25 most recent main runs (2026-08-26 to 2026-09-13, covering both the 4-shard and 5-shard layouts) found no other occurrence; the two above are the only ones in the 60 most recent runs across all branches. The commit that was merged in the main run (#889) touches only scripts/drivers/types/codex/codex-bridge.js and its test file.

That is the "second time" this issue said it was waiting for. The next useful step is probably to stop discarding the joiners' output in the test so the failing join.sh names its reason.

[fujibee]

One more from the same day: the identical failure appeared on the macOS 4/5 shard of #1157 (a PR that changes the record writer/reader path, not join.sh). That makes three unrelated changes and one main push under which this test has now failed the same way, which settles that it is the test, not any of those changes.