Centralize the heartbeat-liveness threshold instead of scattering 2.0 / 3.0 / 10.0 across call sites

[huangzesen]

Summary

The kernel writes an agent heartbeat every 1.0 s (_heartbeat_loop in src/lingtai_kernel/base_agent/lifecycle.py), but the freshness window that defines "alive" is duplicated across three modules with three different values: is_alive defaults to 2.0 s (src/lingtai_kernel/handshake.py:43), the CPR relaunch poll passes 3.0 s (src/lingtai/agent.py:799), and retention uses its own 10.0 s constant (src/lingtai_kernel/maintenance/retention.py:28). A 2.0 s window is one missed tick away from declaring a healthy agent dead, and any future change to the tick cadence must be chased through three modules. Define one kernel-fixed liveness constant (in src/lingtai_kernel/config.py, derived from the tick interval with documented headroom) and have handshake, karma, CPR, mail delivery, avatar spawn, and retention all consume it.

Narrative

Every running agent has a daemon thread that stamps time.time() into .agent.heartbeat once per second. The loop bottom is a literal sleep:

# src/lingtai_kernel/base_agent/lifecycle.py:611
        time.sleep(1.0)

(the shutdown-gated early path at lifecycle.py:378 repeats the same literal). Everything else in the system decides whether an agent is alive by asking how stale that file is — and each consumer invented its own answer.

The canonical reader is handshake.is_alive:

# src/lingtai_kernel/handshake.py:43
def is_alive(path: str | Path, threshold: float = 2.0) -> bool:

That 2.0 s default is consumed implicitly by every caller that doesn't pass a threshold: filesystem mail delivery refuses to deliver to a "dead" agent (src/lingtai_kernel/services/mail.py:161), all the karma lifecycle gates — lull, suspend, cpr, nirvana — check it repeatedly (src/lingtai_kernel/intrinsics/system/karma.py:102,117,131,153,173,191,200,204), and avatar spawn dedup uses it to refuse duplicate spawns (src/lingtai/core/avatar/__init__.py:252).

The problem: with a 1.0 s tick cadence, a 2.0 s freshness window means a single delayed tick flips an agent to "dead". The heartbeat thread shares the GIL and the disk with everything else in the process — a long C-level call, a blocking network fetch that starves the interpreter, a slow fsync, or plain scheduler jitter under load can easily push one tick past the deadline. When that happens, the failure modes are not cosmetic: mail to the agent bounces with "Agent at {address} is not running", karma lull/suspend refuse with "already asleep?", and a peer can even start CPR on an agent that is perfectly healthy. The window heals one second later, so these are classic flap bugs — intermittent, load-dependent, and miserable to reproduce.

Meanwhile lingtai's CPR path deliberately widened its own copy of the number. After relaunching a dead agent it polls for a fresh heartbeat with a different threshold:

# src/lingtai/agent.py:799
            if is_alive(target, threshold=3.0):

This 2.0-vs-3.0 split creates a real correctness gap, not just duplication: karma's _cpr gate (karma.py:191) refuses CPR only if is_alive(resolved) at the 2.0 s default. A heartbeat aged between 2.0 s and 3.0 s therefore passes the gate as "dead" — and then immediately satisfies the 3.0 s poll on the next line of the relaunch loop, before the new process has written anything. CPR logs cpr_alive and returns True on the strength of the corpse's stale heartbeat. If the relaunch actually failed, the caller is told the agent is back when it is not.

Retention independently invented a third value:

# src/lingtai_kernel/maintenance/retention.py:28
DEFAULT_LIVE_HEARTBEAT_SECONDS = 10.0

used via RetentionOptions.live_heartbeat_seconds (retention.py:47) in _fresh_agent_heartbeat (retention.py:720) and the daemon-run .heartbeat mtime check (retention.py:369). Being conservative here is correct — retention protects files from being reported as deletable, so it should over-estimate liveness — but nothing in the code records that 10.0 is "the liveness window plus safety margin"; it is just another magic number that would silently become wrong if the tick cadence changed.

How the code got here is easy to reconstruct: is_alive picked "2× the tick" as an obvious default, CPR discovered that was too tight for a relaunch race and locally patched it to 3.0, and retention (a later, report-only subsystem) chose a deliberately fat window without a shared constant to derive from. Each choice was locally reasonable; collectively they encode the 1.0 s cadence in four places with three different multipliers and one latent CPR bug.

Better looks like the pattern this repo already uses for molt thresholds: src/lingtai_kernel/config.py holds kernel-fixed, documented constants (CONTEXT_PRESSURE_*, IDLE_SLEEP_TIMEOUT_SECONDS) that everything imports. Heartbeat cadence and liveness deserve the same treatment: one HEARTBEAT_TICK_SECONDS, one HEARTBEAT_LIVENESS_SECONDS derived from it with explicit headroom, and one retention window derived from the liveness value.

Current behavior

• src/lingtai_kernel/base_agent/lifecycle.py — _heartbeat_loop writes .agent.heartbeat via _write_heartbeat_tick (:645) and sleeps a hard-coded 1.0 at :378 (shutdown-gated path) and :611 (main loop).
• src/lingtai_kernel/handshake.py:43 — is_alive(path, threshold: float = 2.0); returns time.time() - ts < threshold.
• Implicit-default callers of is_alive (all get 2.0 s): src/lingtai_kernel/services/mail.py:161, src/lingtai_kernel/intrinsics/system/karma.py (eight call sites), src/lingtai/core/avatar/__init__.py:252.
• src/lingtai/agent.py:799 — CPR relaunch poll passes threshold=3.0 inside a 10 s deadline loop; combined with the karma gate at the 2.0 s default this allows a stale heartbeat aged 2.0–3.0 s to fake a successful CPR.
• src/lingtai_kernel/maintenance/retention.py:28 — DEFAULT_LIVE_HEARTBEAT_SECONDS = 10.0, feeding RetentionOptions (:47), _fresh_agent_heartbeat (:720), and the daemon .heartbeat mtime check (:369).
• src/lingtai_kernel/config.py — no heartbeat/liveness constant exists today; the file already hosts the analogous kernel-fixed constants (CONTEXT_PRESSURE_*, IDLE_SLEEP_TIMEOUT_SECONDS), confirming this is the right home.

Detailed implementation plan

1. Add the constants to src/lingtai_kernel/config.py (module level, next to IDLE_SLEEP_TIMEOUT_SECONDS), with a comment block in the existing style explaining the derivation and why the values are kernel-fixed rather than agent-configurable (an agent must not be able to widen its own liveness window, and the numbers are read cross-process by peers that do not share the writer's config):

   # Heartbeat cadence and liveness (kernel-fixed, cross-process contract).
   # _heartbeat_loop stamps .agent.heartbeat every HEARTBEAT_TICK_SECONDS;
   # a reader considers the agent alive if the stamp is younger than
   # HEARTBEAT_LIVENESS_SECONDS. Headroom is deliberate: the writer thread
   # shares the GIL/disk with agent work, so a single tick can be delayed
   # by seconds under load. 5x the tick tolerates four consecutive missed
   # ticks before flapping to "dead".
   HEARTBEAT_TICK_SECONDS = 1.0
   HEARTBEAT_LIVENESS_SECONDS = 5 * HEARTBEAT_TICK_SECONDS
   # Retention is report-only and must over-estimate liveness so it never
   # classifies a live agent's files as stale; keep 2x extra margin.
   RETENTION_LIVE_HEARTBEAT_SECONDS = 2 * HEARTBEAT_LIVENESS_SECONDS  # 10.0, unchanged

2. src/lingtai_kernel/base_agent/lifecycle.py — import HEARTBEAT_TICK_SECONDS from ..config and replace the two time.sleep(1.0) literals in _heartbeat_loop (:378, :611). Update the docstring "Beat every 1 second" to reference the constant.

3. src/lingtai_kernel/handshake.py — make the default sentinel-based so explicit callers keep working and the kernel constant is the single source of truth:

   from .config import HEARTBEAT_LIVENESS_SECONDS
   
   def is_alive(path: str | Path, threshold: float | None = None) -> bool:
       if threshold is None:
           threshold = HEARTBEAT_LIVENESS_SECONDS
       ...

   Keeping the threshold parameter preserves back-compat for tests and any out-of-tree callers that pass an explicit value.

4. src/lingtai/agent.py:799 — delete the explicit threshold=3.0 so the CPR poll uses the shared default. This closes the gate/poll gap: karma's _cpr gate (karma.py:191) and the relaunch poll now use the same window, so any heartbeat fresh enough to satisfy the poll is by construction newer than the gate check and must have been written by the new process. Verify the surrounding deadline = time.time() + 10.0 still exceeds the liveness window (it does at 5.0 s); if HEARTBEAT_LIVENESS_SECONDS is ever raised further, this deadline should be derived too (e.g. max(10.0, 2 * HEARTBEAT_LIVENESS_SECONDS)).

5. src/lingtai_kernel/maintenance/retention.py — replace the literal with a re-export so existing importers and RetentionOptions defaults are unchanged:

   from ..config import RETENTION_LIVE_HEARTBEAT_SECONDS
   
   # Back-compat name; value now derived from the kernel heartbeat contract.
   DEFAULT_LIVE_HEARTBEAT_SECONDS = RETENTION_LIVE_HEARTBEAT_SECONDS

   RetentionOptions.live_heartbeat_seconds stays overridable per scan — only its default changes provenance, not value.

6. No changes needed in services/mail.py, intrinsics/system/karma.py, or core/avatar/__init__.py — they call is_alive without a threshold and inherit the new default automatically. Grep for any other explicit numeric threshold= arguments to is_alive and remove them unless a site has a documented reason to deviate (none found today besides agent.py:799).

7. Docs — update the relevant ANATOMY.md entries (e.g. src/lingtai/core/avatar/ANATOMY.md mentions the liveness check) and any docstrings that state "2 seconds" or "1 second" as literals, pointing at the config constants instead.

Migration/back-compat: no schema, file-format, or on-disk changes. .agent.heartbeat content and cadence are unchanged. The only behavioral change is the wider liveness window (2.0/3.0 → 5.0 s), which is the point of the fix; retention's effective value stays 10.0.

Risks and alternatives

Risks of widening 2.0 → 5.0:

• A freshly-dead agent reads as alive for up to ~5 s instead of ~2 s. Consequences: mail is accepted into a dead agent's inbox (it persists on disk and is read on relaunch — benign); karma lull/suspend may write a .sleep/.suspend signal file to a dead agent, which the signal loop will consume on next start (worth a note in the karma docstrings, but the same race already exists in the 0–2 s window today); cpr refuses with "already running" for a few extra seconds — a retry succeeds.
• Mixed-version fleets: an old kernel's 2.0 s reader may declare an agent dead that a new reader calls alive. Harmless in practice (the writer cadence is unchanged), but release notes should mention it.
• Duplicate-launch protection (TUI lock + heartbeat) gets slightly more conservative, never less — a stale heartbeat blocks a relaunch marginally longer.

Alternatives considered:

• Make the threshold agent-configurable (init.json / AgentConfig). Rejected: liveness is a cross-process contract read by peers, mail, and maintenance tooling that do not share the writer's config; and this repo already treats safety thresholds (molt) as kernel-fixed precisely so agents cannot tune them. A config field would also let a wedged agent widen its own window.
• Write the tick interval into the heartbeat file and let readers derive the threshold. Most flexible, but changes the file format, touches every parser (handshake.py, retention.py:720), and buys nothing until someone actually wants per-agent cadence. Can be layered on later; the constant is a prerequisite either way.
• Only fix the 2.0 default and leave retention alone. Retention's 10.0 would drift silently if the cadence ever changes; deriving it (2 * liveness) keeps today's value while recording the relationship. This is nearly free, so the full plan wins.

Testing plan

• tests/test_handshake.py::test_is_alive_default_uses_kernel_constant — write a heartbeat aged HEARTBEAT_LIVENESS_SECONDS - 0.5 and assert is_alive(dir) is True; age it to HEARTBEAT_LIVENESS_SECONDS + 0.5 and assert False. Keep the existing explicit-threshold test (test_is_alive_custom_threshold) unchanged to prove back-compat.
• tests/test_handshake.py::test_liveness_headroom_invariant — assert HEARTBEAT_LIVENESS_SECONDS >= 3 * HEARTBEAT_TICK_SECONDS, guarding against a future edit reintroducing a one-missed-tick window.
• tests/test_heartbeat.py::test_heartbeat_loop_uses_tick_constant — monkeypatch time.sleep in _heartbeat_loop to capture the sleep argument and assert it equals HEARTBEAT_TICK_SECONDS (both the main-loop and shutdown-gated paths).
• tests/test_karma.py::test_cpr_gate_and_poll_share_threshold — regression for the 2.0/3.0 gap: create a target whose heartbeat is aged just past HEARTBEAT_LIVENESS_SECONDS, stub _cpr_agent so the relaunch never writes a heartbeat, and assert the CPR flow does NOT report success from the stale stamp (it must hit the deadline/exit-code path instead).
• tests/test_retention_report.py::test_live_window_derived_from_kernel_constant — assert retention.DEFAULT_LIVE_HEARTBEAT_SECONDS == config.RETENTION_LIVE_HEARTBEAT_SECONDS and that it is >= HEARTBEAT_LIVENESS_SECONDS; existing protected-agent tests should pass unchanged since the value stays 10.0.
• Sweep test (any suitable module, e.g. tests/test_handshake.py) — assert is_alive is never called with a numeric literal threshold in src/ outside documented exceptions, or at minimum assert inspect.signature(is_alive).parameters["threshold"].default is None.

Generated by a Claude Fable 5 deep review of the lingtai-kernel codebase.