Long-running task liveness is ambiguous after timeouts; need durable heartbeat/lease/terminal-state records

[BatalloLu]

What's wrong

Long-running LingTai tasks can outlive the safe duration of a single LLM worker turn or a synchronous tool call. When a timeout, WorkerStillRunningError, tool timeout, or process interruption happens, the operator/project lead often cannot determine whether the task is:

• still running normally but slow,
• hung,
• partially completed with side effects already written,
• process-dead/orphaned, or
• completed but not reported.

This is not just a TUI sluggishness problem and not just one WorkerStillRunning bug. The broader reliability gap is that long-running tasks do not have a durable, queryable liveness/lease/terminal-state record independent of the current process/turn.

Where

• Component: LingTai kernel runtime / turn recovery / long-running task lifecycle / observability
• Related surfaces: daemon/async work, synchronous tool timeouts, TUI/portal status display, project-level task-watch workarounds
• Target repo: Lingtai-AI/lingtai-kernel because the durable task lifecycle and timeout/recovery semantics should be kernel-owned; UI/portal can later surface the state.

Reproduction / Evidence

This report is based on a 2026-07-07 operator discussion plus local manager-agent evidence.

1. Local WorkerStillRunning artifacts show recurrent timeout/interruption

One manager agent has 52 local history/unfinished_turns/worker_still_running_*.json artifacts:

• 2026-06-27: 1
• 2026-07-02: 3
• 2026-07-03: 4
• 2026-07-04: 3
• 2026-07-05: 10
• 2026-07-06: 30
• 2026-07-07: 1 (worker_still_running_20260707T021436Z_b7d8b2.json)

Shared error string:

LLM worker still running after 300s + 5s grace; ChatInterface is unsafe for AED retry

The artifacts have privacy flags like:

raw_chat_history_included=false
raw_tool_args_included=false
raw_tool_results_included=false

So they prove the worker timeout/interruption happened, but they do not by themselves preserve enough task-level state to answer “what should I do with the long task now?” If the task state was not separately written to pad/knowledge/progress files, continuity can become ambiguous.

2. Existing issues cover adjacent lower-level failures but not this task-lifecycle invariant

I searched for related GitHub issues using terms including WorkerStillRunning, "worker still running", unfinished_turns, "long task" timeout, heartbeat task timeout, stale orphaned task, "task lifecycle", and "long-running task" across Lingtai-AI/lingtai-kernel and Lingtai-AI/lingtai until GitHub search rate limiting stopped later queries.

Related issues found:

• Interrupted turns lose committed tool results: kernel-executed sleep/refresh orphaned at park triggers false "MAY OR MAY NOT have happened" alarms; hung LLM workers poison the interface and drop turns #718 — interrupted turns lose committed/in-flight turn state; hung LLM workers poison the interface and drop turns.
• llm_utils watchdog misclassifies a worker-raised TimeoutError as "future not done", causing a WorkerStillRunningError false positive (and a hot log-spam loop) #738 — watchdog can misclassify a worker-raised TimeoutError as “future not done,” causing a false-positive WorkerStillRunningError.
• Unbounded growth: events.jsonl, chat_history_archive.jsonl, and worker_hang artifacts lack rotation/resolution #622 — events.jsonl, chat archives, and worker hang artifacts lack rotation/resolution.
• WorkerStillRunningError silently puts agent asleep and can lose in-flight task state #238 (closed) — WorkerStillRunningError silently puts agent asleep and can lose in-flight task state.
• tc_wake still hits unsafe ChatInterface after 300s LLM worker timeout #195 (closed) — tc_wake still hits unsafe ChatInterface after a 300s LLM worker timeout.
• WeChat inbound replay after worker hang creates duplicate unread messages lingtai#434 — WeChat inbound replay after worker hang creates duplicate unread messages.

These are important and related, but none appears to track the broader product/runtime invariant: a long-running task should remain machine-diagnosable after timeout/process interruption via a durable heartbeat/lease/terminal-state record.

3. Practical failure mode

If a genuinely long task is placed inside one synchronous LLM turn or one synchronous tool call and exceeds that layer’s timeout, the current task may be interrupted before completion. Worse, the task may have already performed partial side effects (file writes, batch changes, downloads, external calls), so retrying blindly can duplicate or corrupt work.

Project-level workarounds exist today — progress files, manifests, task-watch loops, mtime checks, daemon checks — but these are ad hoc and depend on each project/agent remembering to build its own liveness protocol.

What you expected

For any task that may exceed ordinary turn/tool timeouts, the system should have a durable task record independent of the current process/turn, with fields or equivalent semantics such as:

• task id / source event / owner agent,
• human-visible goal,
• current phase,
• last heartbeat timestamp,
• last progress artifact/path,
• expected next checkpoint or lease deadline,
• terminal state: done, failed, cancelled, timed_out, stale, orphaned, etc.,
• recovery instructions or next safe action.

Timeout should behave like a liveness/lease event, not as an implicit “task failed,” “task cancelled,” or “task disappeared.” When the timeout happens, the kernel should mark the task stale/orphaned (or otherwise non-terminal-but-needs-attention), preserve safe anchors, and notify the supervising agent/operator.

What actually happened

Today, related timeout paths can leave lower-level artifacts or runtime errors, but there is no unified durable task lifecycle visible to the operator/project lead. The human or manager must infer state by manually checking:

• unfinished-turn artifacts,
• event logs,
• progress files if they exist,
• output mtimes,
• daemon/job state,
• emails/notifications,
• project-specific manifests.

This makes long-running autonomy hard to trust: “wait,” “resume,” “retry,” “abort,” and “check artifact first” are not mechanically distinguishable from the runtime state alone.

Severity

major

Workarounds exist, but they are manual and project-specific. The issue directly affects trust in LingTai long-running autonomy: if operators cannot tell whether a delegated long task is still alive or safely terminal, they must revert to manual supervision.

Suggested fix

Possible phased approach:

1. Define a durable task lifecycle/lease record.

   • Store per long task, not per transient LLM turn only.
   • Include owner, source event, goal, phase, last heartbeat, progress artifact, lease deadline, and terminal/stale/orphaned state.

2. Integrate timeout/recovery paths.

   • On WorkerStillRunningError, sync tool timeout, daemon watchdog timeout, refresh/interrupt during active work, etc., preserve safe task anchors and mark the task stale/orphaned rather than leaving only metadata-only unfinished-turn evidence.
   • Do not treat timeout as implicit success/failure/cancel without a terminal record.

3. Expose status to agents and UI/portal.

   • Agents should be able to query “what long tasks are active/stale/orphaned?”
   • TUI/portal should show last heartbeat/progress and stale/orphaned status, not just whether an agent appears alive/asleep.

4. Keep lower-level fixes separate but linked.

   • Interrupted turns lose committed tool results: kernel-executed sleep/refresh orphaned at park triggers false "MAY OR MAY NOT have happened" alarms; hung LLM workers poison the interface and drop turns #718 and llm_utils watchdog misclassifies a worker-raised TimeoutError as "future not done", causing a WorkerStillRunningError false positive (and a hot log-spam loop) #738 remain important for turn recovery / worker watchdog correctness.
   • This issue is the higher-level invariant: even when those lower-level failures happen, long-task state should remain durable and diagnosable.

Notes

This issue intentionally does not prescribe the exact storage format or UI. The minimum desired property is: after a long-task timeout or process interruption, both the human and supervising agents can answer “is it still alive, safely terminal, stale, or orphaned, and what is the next safe recovery action?” without guessing from scattered logs.