Skip to content

fix(mail): soften transient liveness bounces#940

Open
BatalloLu wants to merge 1 commit into
Lingtai-AI:mainfrom
BatalloLu:repairman/email-cpr-liveness-20260715
Open

fix(mail): soften transient liveness bounces#940
BatalloLu wants to merge 1 commit into
Lingtai-AI:mainfrom
BatalloLu:repairman/email-cpr-liveness-20260715

Conversation

@BatalloLu

Copy link
Copy Markdown

Summary

  • Retry the existing heartbeat/presence liveness predicate briefly before refusing filesystem email delivery, so transient heartbeat lag does not immediately produce a false bounce.
  • Change the refusal text from an over-assertive "not running" to "not currently deliverable / liveness uncertain" with the heartbeat source named.
  • Add tests for stale heartbeat, missing heartbeat, and transient stale heartbeat that becomes fresh during the mail-delivery grace window.

Root cause

The observed email bounce versus later CPR result is best explained by a transient heartbeat observation, not two wholly different predicates.

Both paths use the same Core presence predicate for non-human agents: the target must have an agent manifest and a fresh heartbeat under the default two-second threshold.

The previous email path sampled that predicate once in PosixFilesystemMailAdapter.send(). If the one sample failed, the mailman reported "Agent at ... is not running" and emitted an email.bounce. CPR sampled the same predicate later; when heartbeat was fresh by then, system.cpr correctly returned "already running".

Fix

  • Keep the same Core presence predicate as the source of truth.
  • In the mailman delivery path only, wait up to 1.5s with a 0.2s retry interval before refusing delivery.
  • When refusal remains, report deliverability uncertainty and heartbeat freshness rather than declaring the process not running.

Evidence included in investigation

  • Manager timeline: first email(abs) bounce, CPR shortly after said already running, retry email succeeded.
  • Xiaohui analyst/chief evidence: bounce/CPR heartbeat timeout/later already-running patterns.
  • Worker-still-running logs show a separate health layer: an agent can be heartbeat-alive while the LLM interface is unsafe/unresponsive. This PR does not claim to solve that layer.

Validation

  • python3 -m py_compile src/lingtai/adapters/posix/mail.py
  • git diff --check HEAD^..HEAD
  • uv run --with pytest python -m pytest tests/test_filesystem_mail.py tests/test_mail_transport.py tests/test_agent_presence.py tests/test_karma.py -q

Result: 91 passed in 14.78s.

Out of scope

  • Does not solve worker_still_running or "alive but unresponsive" states.
  • Does not add a unified runtime health helper. That should be a follow-up design if needed, separating heartbeat liveness, mail deliverability, worker safety, and last-turn responsiveness.
  • No hotpatch, release, deploy, installed runtime/venv edit, or Xiaohui/Wenda runtime/mailbox/SCU/P0 mutation was performed.

Retry the same heartbeat liveness predicate briefly before refusing filesystem mail delivery, and report heartbeat-based deliverability uncertainty instead of declaring the agent not running. Add tests for no heartbeat, stale heartbeat, and transient stale heartbeat recovery.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant