fix(agent): sweep orphaned agent runs on startup

[yh928]

Summary

The core is a single in-process runtime. When it exits — a crash, a restart, or a deploy — every agent run still in a non-terminal state (Pending / Running / Interrupted) in the durable status store is orphaned: the task that owned it is gone, no executor remains to advance it, and no cancel path can reach it. Yet the run stays in the openhuman.agent_runs_active listing forever. A long-lived instance was observed carrying 50 such zombies.

This adds a startup sweep that reconciles the persistent status store against that reality.

What it does

tinyagents::reaper::reap_orphaned_runs(workspace) lists the still-active runs in the persistent FileStatusStore and moves each to the terminal Cancelled state, recording a stable, grep-friendly error (run orphaned: core restarted while the run was in flight).

It is wired into start_boot_once_jobs, right after the legacy migrations — before readiness is published and regardless of ServiceSet — so the very first agent_runs_active read reflects reality rather than a graveyard of the previous process's in-flight work.

Design notes

• Cancelled, not Failed — a restart is not a run failure, and Failed would render every reaped run as a red error row in the UI. tinyagents exposes mark_completed / mark_failed / mark_interrupted but no mark_cancelled; HarnessRunStatus's fields are public, so the terminal state is set directly rather than taking a cross-repo dependency on a new helper (mirrors exactly the field writes mark_failed performs: terminal status + Done phase + error + ended_at/updated_at).
• Own module, so the read side stays read-only — the replay/status controllers (tinyagents::replay) are documented as strict readers over the status seam. The sweep is the one writer, so it lives in tinyagents::reaper rather than polluting that contract.
• Best-effort, never blocks boot — a failure to open/list the store logs and yields 0; a per-run persist failure is logged and the sweep continues.
• Idempotent — a clean listing reaps nothing; a second sweep over an already-swept store is a no-op (pinned by a test).

Tests

cargo test -p openhuman --lib openhuman::tinyagents::reaper — two tests:

• reap_cancels_every_active_run_and_spares_terminal_ones: seeds Pending/Running/Interrupted + one Completed run, asserts the three non-terminal runs become Cancelled with the reason and an end time, the Completed run is untouched, the active listing empties, and a re-sweep reaps 0.
• reap_on_empty_workspace_is_a_noop: a workspace that never hosted a run reaps nothing and does not error.

Summary by CodeRabbit

• Bug Fixes
  • Improved startup recovery by automatically identifying and cancelling unfinished agent runs from previous sessions.
  • Preserved completed runs while safely handling empty or unavailable workspaces without blocking startup.
  • Added consistent cancellation reasons and completion timestamps for recovered runs.
  • Ensured repeated startup checks are safe and do not alter runs that have already reached a completed state.

Closes #5298

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The runtime now reconciles durable non-terminal agent runs during CoreBuilder::build. The reaper marks orphaned runs as Cancelled, records a stable reason and timestamps, tolerates store failures, and includes unit and integration coverage.

Changes

Orphaned run reaping

Layer / File(s)	Summary
Reaper implementation and validation src/openhuman/agent/tinyagents/reaper.rs	The reaper lists active runs, marks them Cancelled and Done, records the orphan reason and timestamps, and validates reconciliation, preservation, idempotency, and empty workspaces.
Boot-time reaper integration src/openhuman/agent/tinyagents/mod.rs, src/core/runtime/builder.rs, src/core/runtime/services.rs	The reaper is registered and invoked during CoreBuilder::build. Comments document that reconciliation applies to build-only runtimes and served runtimes. Integration coverage validates the boot path.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CoreBuilder
  participant reap_orphaned_runs
  participant DurableStatusStore
  CoreBuilder->>reap_orphaned_runs: reconcile configured workspace
  reap_orphaned_runs->>DurableStatusStore: list active runs
  DurableStatusStore-->>reap_orphaned_runs: pending, running, or interrupted runs
  reap_orphaned_runs->>DurableStatusStore: save cancelled terminal status
  reap_orphaned_runs-->>CoreBuilder: return reaped count

Loading

Possibly related PRs

• tinyhumansai/openhuman#4791: Extends the CoreBuilder::build startup path and registers the tinyagents reaper.
• tinyhumansai/openhuman#5074: Handles durable agent-run lifecycle reconciliation during startup.
• tinyhumansai/openhuman#5135: Adds startup reconciliation for orphaned agent or flow runs.

Suggested labels: rust-core, agent, bug

Poem

I sweep the runs at break of day,
Mark stale work Cancelled away.
Timestamps rest in tidy rows,
While startup calmly onward goes.
The active list is clean. 🐇

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the startup sweep for orphaned agent runs, which is the primary change.
Linked Issues check	✅ Passed	The changes implement startup reconciliation, best-effort failure handling, idempotency, terminal-state preservation, runtime wiring, and regression tests for issue #5298.
Out of Scope Changes check	✅ Passed	All changes support orphaned-run reconciliation, its startup integration, documentation, module wiring, or related regression coverage.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.

Warning

Your free Security trial is over. An organization admin can upgrade to Advanced for continuous pull request security review or dismiss this notice.

---

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

This PR fixes the "zombie run" problem where agent runs stuck in Pending, Running, or Interrupted states from a previous process were never cleaned up. The fix adds a startup sweep (reap_orphaned_runs) that transitions all still-active runs to Cancelled with a stable, grep-friendly error before the runtime becomes queryable.

• Placement in build() (not serve()): The sweep runs in CoreBuilder::build() before CoreRuntime is returned, so build-only embedders that never call serve() — and whose first dispatchable RPC is agent_runs_active — still see a clean listing.
• Best-effort design: Any per-run or store-level failure is logged and the sweep continues; a store that can't be opened yields 0 rather than blocking startup.
• Well-tested: Three tests cover the primary scenario (Pending/Running/Interrupted → Cancelled, Completed untouched, idempotent re-sweep), the empty-workspace no-op path, and an integration test that pins the sweep to the build() path with a real CoreBuilder.

Confidence Score: 5/5

Safe to merge; the sweep is best-effort, runs before the runtime is returned to any caller, and is fully covered by tests including a wiring integration test.

The change is narrowly scoped: a new read-then-write sweep over the durable status store at startup, gated on a workspace config being present. Errors at every layer (store open, list, per-run persist) are caught and logged rather than propagated, so a broken store cannot block boot. The list_active() filter correctly excludes already-terminal runs, verified by the test that seeds a Completed run and asserts it survives the sweep unchanged. Placement in build() rather than serve() is intentional and pinned by an integration test. No existing code paths are altered.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
src/openhuman/agent/tinyagents/reaper.rs	New module implementing the startup orphan sweep; well-structured with best-effort error handling, idempotent logic, and three targeted tests covering all non-terminal states, the empty-workspace path, and the build-path wiring.
src/core/runtime/builder.rs	Correctly places the sweep in build() after CoreContext::init() and before CoreRuntime is returned; gated on config.as_ref() so workspaceless builds skip it; return value intentionally discarded since logging is handled inside reap_orphaned_runs.
src/core/runtime/services.rs	Adds only an explanatory comment in start_boot_once_jobs clarifying why the sweep is absent here; no logic changes.
src/openhuman/agent/tinyagents/mod.rs	One-line module declaration exposing the reaper at pub(crate) visibility, consistent with adjacent modules.

Sequence Diagram

sequenceDiagram
    participant E as Embedder / serve()
    participant CB as CoreBuilder::build()
    participant CI as CoreContext::init()
    participant R as reaper::reap_orphaned_runs()
    participant S as FileStatusStore

    E->>CB: build().await
    CB->>CI: CoreContext::init(...)
    CI-->>CB: (ctx, token, config)
    CB->>R: "reap_orphaned_runs(&cfg.workspace_dir).await"
    R->>S: list_active()
    S-->>R: [Pending, Running, Interrupted runs]
    loop For each orphaned run
        R->>R: mark_orphaned(status) → Cancelled + Done + error + ended_at
        R->>S: put_status(status)
        S-->>R: Ok(())
    end
    R->>R: log::info! reaped N orphaned run(s)
    R-->>CB: usize (ignored)
    CB-->>E: Ok(CoreRuntime)
    Note over E,S: First agent_runs_active RPC now sees a clean listing

Loading

_{Reviews (4): Last reviewed commit: "fix(agent): sweep on the build path, and..." | Re-trigger Greptile}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/core/runtime/services.rs`:
- Around line 335-345: Add a boot-path test covering start_boot_once_jobs with
optional services disabled: seed an active agent run, invoke the boot sequence,
and assert the run is cancelled. Reuse the existing reaper test setup and verify
the unconditional startup sweep rather than calling reap_orphaned_runs directly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 7e5f27d7-e1a0-4635-bc18-a1decd8ebe74

📥 Commits

Reviewing files that changed from the base of the PR and between 8072f08 and 6fb468b.

📒 Files selected for processing (3)

• src/core/runtime/services.rs
• src/openhuman/tinyagents/mod.rs
• src/openhuman/tinyagents/reaper.rs

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6fb468b61b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[yh928]

Pushed 1ce6eb7a8 — all three findings applied.

Build-only runtimes (codex P2) — fixed, and it was the more serious of the three. You were right that start_boot_once_jobs only runs from serve(), so the documented build()-then-invoke() embedder never reached the sweep — and agent_runs_active is dispatchable the moment build() returns, so exactly that caller kept reading the previous process's graveyard. The sweep now runs in CoreBuilder::build(), which every runtime goes through (CLI, TUI, embedded, desktop), and is no longer duplicated in the boot-once jobs.

Boot-path test (CodeRabbit) — added, and it landed on the same path this fix moved to: a_build_only_runtime_is_swept_before_it_can_be_invoked seeds a Running run, builds with ServiceSet::none() + DomainSet::harness(), and asserts the run is Cancelled with the orphan reason. Two things the first draft got wrong, worth noting since they would have made it a false-positive test: OPENHUMAN_WORKSPACE is process-global (it now takes TEST_ENV_LOCK, as the approval/config tests do), and the resolved workspace is <OPENHUMAN_WORKSPACE>/workspace, not the env value — so the test also asserts the build resolved the directory it seeded, rather than passing by sweeping somewhere else.

Duplicate info log (greptile) — fixed. Dropped the caller's summary rather than demoting the module's, since the module line carries the count and the module context. The pre-sweep "N orphaned run(s) to reap" line is now debug, and the summary only fires when reaped > 0 — a clean boot, which is the normal case, now says nothing at info.

Tests: reaper 3 green.

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/openhuman/agent/tinyagents/reaper.rs`:
- Around line 39-100: Update reap_orphaned_runs and mark_orphaned to follow the
domain-log contract: generate one opaque sweep_id at sweep entry, use [domain]
or [rpc] for every sweep log, and emit a debug start event containing the
sweep_id. Include sweep_id in all subsequent events, and before put_status log
each transition with run_id, from_status, and to_status=Cancelled; preserve the
existing reaping behavior and counts.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f52d5134-9d78-420b-bbdb-708bf9e9dccf

📥 Commits

Reviewing files that changed from the base of the PR and between d75b0a4 and f69d42f.

📒 Files selected for processing (4)

• src/core/runtime/builder.rs
• src/core/runtime/services.rs
• src/openhuman/agent/tinyagents/mod.rs
• src/openhuman/agent/tinyagents/reaper.rs

🚧 Files skipped from review as they are similar to previous changes (2)

• src/core/runtime/builder.rs
• src/core/runtime/services.rs

[greptile-apps]

yh928 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

[greptile-apps]

yh928 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[greptile-apps]

yh928 has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

[M3gA-Mind]

Maintainer review pass (comment only — no approval, and I am not pushing to this branch).

Verdict: good change, but it will not compile on today's main. GitHub says MERGEABLE and every check is green, and both are misleading here: the checks last ran on 2026-08-05, and the green comes from a base four weeks and one repo-wide refactor old.

Blocking: the tinyagents crate path no longer exists

reaper.rs opens with

use tinyagents::harness::events::HarnessRunStatus;
use tinyagents::harness::ids::{ExecutionStatus, HarnessPhase};
use tinyagents::harness::observability::HarnessStatusStore;

vendor/tinyagents is now a virtual manifest, not a package — the vendored pin moved 175 commits (e0f321062 → a53888802, v2.1.2) and openhuman names the member crates directly. On current main, git grep "use tinyagents::harness::" -- src/ returns zero files. The merge is textually clean only because reaper.rs is a new file; the build breaks the moment it lands.

The rename is mechanical — tinyagents::harness:: → tinyagents_harness:::

use tinyagents_harness::events::HarnessRunStatus;
use tinyagents_harness::ids::{ExecutionStatus, HarnessPhase};
use tinyagents_harness::observability::HarnessStatusStore;

That matches how src/openhuman/agent/tinyagents/journal.rs:55-62 imports the same symbols today.

Everything else you depend on is still there — checked individually

What reaper.rs needs	Status on main
HarnessStatusStore::list_active	present — journal.rs:243
FileStatusStore	present — journal.rs, impl HarnessStatusStore at :207
open_session_stores + SessionStores.kv	present — session_import/ops.rs:53, struct at :42
start_boot_once_jobs	present — core/runtime/services.rs:354
CoreBuilder::build insertion point	present, but the line above it changed: CoreContext::init(...) is now CoreContext::init_with_config(...) (builder.rs:596). Your hunk inserts before Ok(CoreRuntime { (:605), so the placement still holds — worth an eye after rebasing.

On the design

No objection from me, and the reasoning in the PR body is unusually careful. Two things I want to record as read-and-agreed rather than have you re-argue them:

• Cancelled over Failed — right call. A restart is not a run failure and Failed would paint every reaped run red in the UI. Writing HarnessRunStatus's public fields directly instead of adding a cross-repo mark_cancelled is the proportionate choice.
• Moving the sweep from start_boot_once_jobs into CoreBuilder::build — this is the right answer to the codex thread about build-only embedders, and the comment you left behind in services.rs explaining why it is not there is the part that keeps it from being re-added later. Please keep that comment.

What to do

1. Rebase onto current main.
2. Fix the three use lines as above.
3. Re-check the builder.rs insertion point after the rebase.
4. Push — CI on the new head is the first result that will mean anything.

Nothing here is a change of design; it is all drift in the tree underneath you. Sorry for the four weeks.

[tinysweeper]

tinysweeper found nothing blocking. Approving.

             $0.0083 · 123,355 in / 1,255 out · 39,936 cached (32%) · openrouter/openai/text-embedding-3-small, deepseek/deepseek-v4-flash · 600 embedded
critique:    $0.0036 · 61,346 in  / 347 out   · 26,112 cached (43%) · deepseek/deepseek-v4-flash
security:    $0.0025 · 38,293 in  / 387 out   · 13,824 cached (36%) · deepseek/deepseek-v4-flash
tests:       $0.0015 · 15,802 in  / 449 out   · 0 cached (0%)       · deepseek/deepseek-v4-flash
description: $0.0007 · 7,914 in   / 72 out    · 0 cached (0%)       · deepseek/deepseek-v4-flash

[tinysweeper]

How this change flows

2 changed behaviours across 2 relationships. 2 surrounding behaviours are shown (60 graph nodes walked). 46 further behaviours left out to keep the diagram readable.

flowchart LR
  n0["CoreBuilder<br/>changed"]:::changed
  n1["start_bootstrap_jobs<br/>changed"]:::changed
  n2["bootstrap_job_plan"]:::impacted
  n3["ServiceSet"]:::impacted
  n0 -->|uses| n3
  n1 -->|calls| n2
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff

Loading

_{Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge.}

[yh928]

Rebased onto main (8e65c4008) and pushed — bde2633bb. Thanks for the pass; you were right that the green was misleading, and the specific thing you named is exactly what broke.

The crate rename, applied. tinyagents::harness:: → tinyagents_harness:: in reaper.rs's three imports plus the one in its tests, matching how journal.rs — which owns the status store this reaps through — names the same symbols today. Nothing else in the sweep needed changing: HarnessStatusStore::list_active, FileStatusStore, open_session_stores + SessionStores.kv, and the CoreBuilder::build hook are all still where your table said they were, and the merge itself was clean.

One thing your review predates. The layout gate grew a rule since (check-openhuman-rust-layout.mjs): an inline #[cfg(test)] mod tests is now rejected in favour of a descriptive sibling. The sweep's tests moved to reaper_tests.rs unchanged, wired the way extract_tool.rs does it.

Verified locally, since the checks on this PR were four weeks stale:

• cargo check --lib — clean
• cargo test --lib reaper — 3/3 pass (reap_cancels_every_active_run_and_spares_terminal_ones, reap_on_empty_workspace_is_a_noop, a_build_only_runtime_is_swept_before_it_can_be_invoked)
• cargo fmt --check, the layout gate, and the full pre-push suite (lint, tsc, cargo clippy -D warnings on both crates, token lints) — all clean

The reasoning for keeping the sweep in CoreBuilder::build rather than start_boot_once_jobs is unchanged and still documented at both sites: the boot-once jobs only run from serve(), so a build-only embedder would never be swept while openhuman.agent_runs_active is already dispatchable.

[yh928]

The Rust Core Coverage lane went red on this branch, and I want to correct the record before anyone spends time on it: it was my bug, not a flake. My first read was that it looked unrelated — the failing test is agent::triage::evaluator::tests::part_01_tests::cloud_5xx_falls_through_to_local_fallback, in a file this branch does not touch — so I checked instead of asserting it.

Reproduced, and it is deterministic. Against the coverage lane's own filter, cargo test --lib -- 'core::runtime' 'openhuman::agent':

	result
clean main	3/3 green (2350 passed)
this branch, before the fix	3/3 red
this branch, after the fix	3/3 green (2353 passed)

A different triage test failed on some runs, which is what first suggested a race — but "3/3 red vs 3/3 green on main" is not a flake, and the honest reading of a differing-test-each-time failure is a race my change made deterministic, not one that was already there.

The mechanism. a_build_only_runtime_is_swept_before_it_can_be_invoked — a test I added — calls CoreBuilder::build(), and CoreContext::init re-registers the native bus handlers as part of that. When it runs in parallel with a test holding a mock_agent_run_turn stub, the build reinstalls the real agent.run_turn handler over the stub. The triage test then reaches a real harness instead of its mock and hits unused_model_source() — a ScriptedModel::new(Vec::new()) that exists precisely because it should never be called. Hence response queue is exhausted.

The test already took TEST_ENV_LOCK, for OPENHUMAN_WORKSPACE. That is a different lock. core::bus_testing::BUS_HANDLER_LOCK is the one that covers bus registration, and its own documentation states the rule:

"Any test that touches global native-bus registration state should acquire this lock first."

Building a core is exactly that. It now takes both.

One hypothesis I chased and discarded, because it would have been a real defect: that reap_orphaned_runs was cancelling a run another test had in flight — the sweep does terminate every non-terminal run in the workspace, so it was worth ruling out rather than assuming. It isn't that. I disabled the sweep call in build() and the failures persisted (and gained one), which pointed at the test's core-build rather than the sweep's writes. The sweep only mutates HarnessRunStatus rows and never resolves a model; the failure was a model-resolution error.

Pushed. Sorry for the noise — the lane was right and I read it wrong the first time.