Skip to content

feat(observability): make the stranded-operation reaper visible while idle - #973

Draft
aparajon wants to merge 2 commits into
mainfrom
armand/reaper-idle-heartbeat
Draft

feat(observability): make the stranded-operation reaper visible while idle#973
aparajon wants to merge 2 commits into
mainfrom
armand/reaper-idle-heartbeat

Conversation

@aparajon

@aparajon aparajon commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Why this matters

An idle reaper pass is silent by design: it settles nothing, and an instance that loses the election has nothing to report at all. That leaves a healthy steady state indistinguishable from a reaper that stopped ticking — hours of INFO logs answer neither question, and telling the two apart is the first thing an operator needs during an incident.

What it does

  • States the reaper's cadence and batch size once at startup, so a later pass count can be read against a known interval.
  • Closes each window of passes with one heartbeat line tallying passes run, operations settled, passes that stood down for another instance, and passes that failed.
  • Separates the three pass outcomes in that tally, so an operator seeing no settlements can tell whether this instance did the work and found nothing, deferred to the election holder, or was failing outright.

Summarizing a window rather than narrating every pass keeps a healthy reaper to a few lines an hour — quiet enough to leave on, frequent enough that its absence is noticeable.

🤖 Generated with Claude Code

… idle

An idle reaper pass is silent by design: it settles nothing, and an
instance that loses the election has nothing to report. That leaves a
healthy steady state indistinguishable from a reaper that stopped
ticking, and hours of INFO logs answer neither question.

The loop now states its cadence and batch size once at startup, and
closes every window of passes with one heartbeat line tallying passes
run, operations settled, passes that stood down for another instance,
and passes that failed. An operator reading INFO logs can tell a reaper
with nothing to do from one that is not running.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 7, 2026 20:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds periodic “heartbeat” observability to the stranded-operation reaper so operators can distinguish a healthy idle reaper from a stalled or unelected one, without logging every pass.

Changes:

  • Introduces a heartbeat window that summarizes pass outcomes (ran / not elected / failed) and total operations settled.
  • Emits a one-time startup INFO log stating cadence, batch size, and heartbeat window size; upgrades stop logs to INFO.
  • Adds unit tests covering heartbeat window behavior and outcome tallies.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
pkg/api/reaper.go Adds heartbeat aggregation + startup/heartbeat logs; refactors reaper pass to return an outcome and settled count.
pkg/api/reaper_test.go Adds tests validating heartbeat summary emission, outcome tallies, and window reset behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/api/reaper.go
…pped

A pass that finished cleanly just as the context ended logged "error": null,
which reads as a fault with no cause. The shutdown branch now reports ctx.Err()
as the reason and carries the pass's own error, when it has one, under its own
key so the two causes stay distinguishable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants