Runtime health: alive heartbeat can hide stuck worker and broken Telegram outbound readiness

[BatalloLu]

What's wrong

An agent can enter a bad state where the process heartbeat and inbound Telegram listener look alive, but the current LLM/tool turn is stuck and user Telegram messages are deferred. During recovery, lifecycle tools can disagree with observed liveness, and Telegram MCP can be partially initialized: inbound/listener works but outbound telegram.reply fails with Telegram manager not initialized.

This makes the agent appear online while it cannot actually serve the user, and normal CPR/interrupt/suspend signals do not reliably recover or classify the state.

Where

• Component: LingTai kernel runtime / lifecycle health / worker recovery / MCP readiness
• Likely areas:
  • system.cpr / system.interrupt / system.suspend liveness and recovery behavior
  • LLM worker timeout/AED recovery path
  • notification deferral while an agent is active
  • Telegram MCP startup/readiness handling for outbound tools
• Repo: Lingtai-AI/lingtai-kernel

Reproduction

This is from a real incident on 2026-07-15. Local paths, Telegram chat ids, and project-specific task content are omitted/redacted, but the exact runtime error strings are preserved.

Observed chain:

1. The target agent received a user Telegram message and began responding.
2. Runtime logs then repeatedly showed:

LLM API not responding

3. After the watchdog/grace period, logs showed:

LLM worker thread still running after 300s + 5s grace — interface state may be inconsistent. Refusing AED retry.

4. From the user's perspective, the bot looked like it stopped replying. The agent could still appear alive at the process/heartbeat level, so this was not a simple process-dead condition.

5. system.cpr against the agent returned:

CPR launch did not produce a fresh heartbeat within 10s (...)

A follow-up host check shortly after showed the process was alive and heartbeat/status were fresh. The agent had woken on recovery/internal notifications rather than directly handling the user Telegram message.

6. The fresh process then entered an internal tool call and did not complete it promptly:

email.check

While the agent was active in that tool call, Telegram handling remained deferred.

7. system.interrupt returned success (status: interrupted) but did not clear the active stuck state. A follow-up check still showed fresh heartbeat/status and active state with no tool completion.

8. system.suspend returned an opposite liveness classification:

Agent at <agent-dir> is not running — already suspended?

At that same time, a host check showed:

• .agent.heartbeat fresh (<1s)
• .status.json fresh and state=active
• the agent process still running

So the lifecycle tool's “not running” result contradicted process/heartbeat evidence.

9. After a manual hard restart, inbound/listener and LLM work resumed, but the first outbound Telegram attempt failed. The agent attempted telegram.reply; the tool returned:

Telegram manager not initialized — server boot failed. Check stderr for the underlying exception (most often missing LINGTAI_TELEGRAM_CONFIG or invalid bot token).

The startup log around that same restart had shown the Telegram listener initially running, followed by an eager-start failure:

[Errno 54] Connection reset by peer

So the agent could receive/process work, but outbound Telegram tool readiness was broken.

10. A second clean restart finally restored all three layers:

• fresh heartbeat/status
• Telegram listener and mcp_inbox poller running
• MCP telegram ... loaded 1 tools
• telegram.accounts succeeded
• later telegram.reply returned status: sent

What you expected

• Heartbeat/liveness should not be treated as sufficient service readiness.
• A stuck LLM worker after the 300s + grace window should either be reliably terminated/recovered or put the agent into an explicit unhealthy/STUCK state that lifecycle tools can act on.
• interrupt, suspend, and cpr should not report contradictory states for the same fresh process/heartbeat condition.
• Telegram MCP readiness should distinguish inbound listener startup from outbound tool availability; a partial startup failure should be surfaced and recoverable.

What actually happened

• The agent looked alive at heartbeat/listener level while the active turn was stuck and user Telegram was deferred.
• interrupt reported success but did not restore responsiveness.
• suspend reported “not running” while host evidence showed a live active process with fresh heartbeat.
• A restart could leave Telegram in a partial state: listener/inbound present, but outbound telegram.reply failed because the Telegram manager was not initialized.
• Full recovery required manual PID-tree termination plus a second clean CPR/restart, followed by explicit verification that outbound Telegram tool calls succeeded.

Severity

major

This can make a user-facing bot stop responding while still appearing alive, and the workaround requires expert manual diagnosis/restart. It is not just a cosmetic status problem: it blocks the user-facing communication path and can mislead orchestration/lifecycle tools.

Suggested fix

Consider splitting agent health/readiness into separate dimensions and making lifecycle tools report/act on them explicitly:

1. Process liveness: heartbeat / PID / process exists.
2. Turn progress: no LLM worker stuck beyond watchdog/grace; active tool calls have bounded progress or explicit timeout state.
3. Service readiness: notifications are not indefinitely deferred behind an abandoned active turn.
4. MCP inbound readiness: listener/poller is running.
5. MCP outbound readiness: tool manager initialized and telegram.send/reply/accounts can be called.

Possible implementation directions:

• When LLM worker thread still running after 300s + 5s grace occurs, mark the agent as a distinct unhealthy/STUCK/unsafe state rather than leaving it looking normally alive.
• Make system.interrupt, system.suspend, and system.cpr consult more than the heartbeat predicate and return richer diagnostics when process liveness and service readiness disagree.
• Add a recovery path that can safely terminate/restart an abandoned worker/process group, or clearly instruct the operator that a hard restart is required.
• Add MCP readiness checks so startup failures like [Errno 54] Connection reset by peer do not leave an agent able to receive/process but unable to send.
• Add regression tests for: fresh heartbeat + stuck active worker; interrupt success but no progress; suspend false “not running” while heartbeat fresh; Telegram listener started but outbound manager not initialized.

Related context: PR #940 addresses a narrower transient liveness false-bounce/message wording issue. This report is broader and should not be treated as fixed by that PR alone.

[huangzesen]

Closing this as not planned after reviewing the incident against current main.

Several narrower partial fixes and diagnostics have already shipped:

• fix(cli): exit poisoned worker processes after teardown #899 makes the WorkerStillRunning path fail closed: it poisons the unsafe chat interface, preserves a bounded/redacted unfinished-turn artifact, publishes a recovery notification, requests refresh, and exits a poisoned CLI process after teardown so the watcher can relaunch it.
• fix(#164): budget recursive file tools and add ACTIVE-without-progress watchdog #165 adds an ACTIVE-without-progress watchdog and status telemetry. This is diagnostic rather than automatic cancellation/recovery.
• Add MCP inbox latency diagnostics #288 retries a stale/closed MCP transport once and surfaces a non-blank transport error.
• docs(mcp): add Telegram readiness checklist #956 documents a Telegram readiness check that separates registry/tool/account checks, listener ownership, and a controlled inbound/reply proof.

These changes materially improve the original incident, but they do not create the broad unified cross-layer health record proposed here. Lifecycle actions still primarily classify the target through heartbeat/presence, and a Telegram eager-start failure can still leave the MCP server present while outbound calls report that the manager is not initialized.

We are not planning to implement that umbrella health-model contract in this issue. The remaining narrower ownership is already tracked in #794 for durable task/worker liveness after timeout or interruption, and #711 for Telegram startup/readiness and in-session recovery. Closing this as not planned therefore does not claim that every incident path is fully resolved; it records that the broad combined issue will not be pursued as one unit.