Recover from poisoned org DB client mutex

[posthog]

Problem

• Attribution and metrics uploads silently stop for a machine until the daemon restarts, and the only signal is a panic line on stderr.
• run() in src/api/org_db.rs locked the cached Postgres client with .lock().expect("org client mutex poisoned"). If the wrapped op panics mid-query, Rust marks the mutex poisoned for the life of the process.
• The client lives in a process-global CONNECTIONS map inside a long-lived daemon, so there is no recovery: every later call panics on the same line.
• One poisoned mutex takes down all five org DB write paths at once — metrics, notes, CAS, file change counts, and commit authorship summaries.
• The original panic that poisoned the mutex was swallowed by the telemetry worker's join handle, which reported only a generic "telemetry flush task panicked".

Changes

• Recover instead of panic. org_db.rs was the lone outlier; the rest of the repo already recovers with unwrap_or_else(|poisoned| poisoned.into_inner()) (about twenty sites in src/streams/db.rs and src/daemon.rs). Apply the same to all five lock sites.
• Treat a poisoned client mutex like a stale connection. A mid-query panic can leave the Postgres protocol state desynchronized, so reusing that socket would just fail differently. run() now drops the entry from CONNECTIONS and redials — the same recovery it already runs for a dropped connection.
• Report the hidden panic. Wrap the telemetry flush closure in catch_unwind so a panic logs its own message instead of the join handle's generic text.

Notes

• No Rust toolchain is available in this environment, so task build, task test, task lint, and task fmt were not run here. The change reuses a pattern already established across the repo; please let CI run the suite.
• Added a unit test for the new panic_message helper.

Why

The daemon caches the org DB client for its whole life, so a single mid-query panic poisons the mutex permanently and quietly stops the attribution data this product exists to produce. The fix makes the failure recoverable rather than self-sustaining.

---

Created with PostHog Desktop from this inbox report.