Bridge run-state uses bare PIDs on an NFS-shared $HOME, so two login nodes collide

[fujibee]

Child of #906. The ring the reporter said they could not find in the tracker.

What the code does

instance-id.sh:148, _agmsg_pid_alive, is entirely local — kill -0 on POSIX, tasklist on Windows. It has no concept of which host a PID belongs to.

The launcher's run files carry no host in their names either:

codex-bridge.<team>.<name>.pid
codex-bridge.<team>.<name>.appserver
codex-bridge.<team>.<name>.thread

With $HOME on NFS and two login nodes mounting it, node B reads node A's pidfile and judges that PID in its own PID space, which answers a different question than the one being asked. The reporter measured the pid and appserver files being rewritten from the other node.

The same absence lets two hosts resume the same codex thread, so one rollout takes appends from both.

Directions, not a decision

Put the hostname — and, if a reused hostname is a concern, the boot id — into the run file names or into the owner line inside them, and refuse to draw conclusions from a PID written by a different host.

This is the cross-host half of #485.

Related: #906 (the report), #485, #935 (ring 1).

[fujibee]

Not planned for the current release, and the shape of any eventual fix is narrower than this issue proposes. Recording the reasoning so the next read does not start over.

What the parent report actually says

The cross-host collision is item 4 of four in #906, and its reporter labels it "(aggravator)". The three rings that took their account down twice in one day were the other three:

• the bridge surviving an already has an active writer failure — fix(codex-bridge): exit when another writer owns the thread (#906 link 1) #935
• the launcher respawning without an orphan scan — fix(codex-bridge): make the failure cap reach and bound the re-arm rate (#906 link 3) #941
• the re-arm loop with no backoff — fix(codex-bridge-launcher): reap a same-(project,role) orphan before spawning (#906 link 2) #943

All three are merged with their controls green, re-verified against main after #978 landed. So the failure mode that cost that user their session is closed. What remains here is that during such an incident the state on disk cannot be trusted — the reporter watched pid and appserver files being rewritten from the other login node while it was happening.

That is a real observation, and it is not the thing that broke them.

Why the proposed fix is out of proportion

Putting the hostname into the run file names means renaming every file in the families that carry it. Measured on one developer machine: 64 role-session.* files alone, 26 of which have no surviving spawn.*. The same families are the ones #1023 would rename for a different reason — an injective key derivation. Renaming them twice, for two reasons, in two releases, is the expensive way to arrive at the same place.

It also touches the widest possible surface for a condition that requires a specific setup: one $HOME on NFS, mounted by two login nodes, with agmsg running natively on both. Members that join over the remote transport are not exposed — they carry no local pid or placement at all, so there is no foreign PID for a local check to misread.

What may be done instead

Refuse to conclude, rather than rename. A PID written by another host is not a PID this host can judge, and the honest answer is undetermined, not a verdict. That means:

• the owner line carries the host that wrote it; file names are untouched
• a liveness check whose owner line names a different host returns undetermined with that reason, and no caller treats it as dead
• absence of a host field in an older record is also undetermined, not "local"

This costs no migration, and it makes the failure visible instead of silently wrong — which is the property that was missing. registry-lock.sh already records a host line, so the convention exists in the tree.

Deliberately not claimed here: that undetermined-everywhere is free. It will make some paths refuse where they previously guessed right most of the time, and each of those paths needs its own look at whether refusing is better than guessing. That review has not been done.

Related: #906 (the report, ring 4), #485, #1023 (the other reason to rename these files).

[fujibee]

One more thing that bears on whether this is worth building at all: most of what this issue describes stops existing under the daemon design in #1105.

Read from the RFC rather than assumed — the sync section is explicit that the per-team process tracked by a pidfile goes away:

No separate processes. Nothing to orphan, nothing to lose track of, no pidfile as the sole record of what is running. What is syncing is whatever the daemon says is syncing.

The mechanism this issue is about — bare PIDs written to files that another host can read and misjudge — is that arrangement. Remove the files and there is no foreign PID to misread. The RFC reaches the same conclusion from a different direction, citing #1103 (two engines for one team, the pidfile naming only the later one) as its worked example.

What does not go away, by the RFC's own words:

Worth being precise about what the lock is: it is local. It keeps two sessions on the same machine from both answering as bob; it is not an authority claim about who may act as bob anywhere.

So the cross-host authority question survives the daemon and is out of its declared scope. That is a smaller and different question than the one this issue asks, and it is not answered by adding a hostname to a filename either.

Splitting it that way:

• pid / appserver / thread state read across hosts — dissolved by docs: add the agmsgd RFC for comment #1105
• one team's engine discoverable regardless of which host recorded it — dissolved by docs: add the agmsgd RFC for comment #1105 (Two sync engines can run for one team, and the pidfile only records the second — the first survives stop/restart and is invisible to status #1103)
• "may this host act as this member" — untouched by docs: add the agmsgd RFC for comment #1105, and untouched by this issue's proposal

Which leaves the interim. Until the daemon lands, the fail-closed shape in the previous comment is the whole of what is worth doing here, and it is worth doing only if the interim is long. That is a scheduling question, not a design one.

Not claimed: that #1105 will land, or when. It is open for comment and has not been accepted.