Feature: Error Manager Service

[RyanTheRobothead]

Re-scoped 2026-05-23 as part of the → 1.0 roadmap review. See the pinned comment for rationale. The original broad scope has been split: the recovery primitives (rule engine, recovery workflows, multi-channel notifications, hanging-action detection) now ship as AbstractEventHandler plugins inside the Event Manager. This issue is now scoped to the lifecycle layer and agentic layer that build on top of those primitives.

Summary

Provide centralized error lifecycle tracking, pattern detection, and agentic troubleshooting on top of the AbstractEventHandler plugin system (introduced by #312 / PR #312). Whether this lands as a dedicated madsci.error_manager service or remains an Event Manager extension is an explicit Post-1.0 decision tracked by #325.

In scope

Error lifecycle layer

• Error records as first-class entities, distinct from the underlying events. Each record has its own state machine: new → acknowledged → investigating → resolved (or recurring if the same fingerprint reappears after resolution).
• Pattern detection across event history: recurring errors, correlated errors across nodes/managers, errors that consistently precede workflow failures.
• Dedicated operator UX: dashboard view for error triage (assign, comment, acknowledge, close), filtering by source/severity/state, audit trail of resolution actions.
• Retention semantics: error records persist longer than the events that generated them (audit trail for past incidents).

Agentic troubleshooting layer

• Configurable troubleshooter agents that can autonomously attempt to resolve known issues, scoped by an authorization boundary (e.g., "can restart nodes but cannot modify resources, can retry workflows but cannot skip steps").
• Guidance rulesets: user-defined rulesets describing known failure modes, diagnostic steps, and remediation procedures.
• Pluggable backends: rule-based (deterministic) and LLM-backed (agentic), with the option to use both in a tiered approach (deterministic first, escalate to agentic).
• Audit trail of all diagnostic steps taken and decisions made by troubleshooters.
• Human escalation when the agent exhausts its ruleset, encounters an unfamiliar error, or reaches its autonomy boundary.

Out of scope (covered elsewhere)

The recovery primitives that #247 originally bundled now have their own issues, all built on the EventHandler plugin system:

• Rule engine / deterministic condition→action handling: Error Recovery: Deterministic condition→action rule engine (as EventHandler plugin) #321
• Recovery workflow framework (retry, cooldown, circuit breakers): Error Recovery: Recovery workflow framework + retry/cooldown/circuit-breaker primitives #322
• Multi-channel notifications (Slack, webhook, PagerDuty): Error Recovery: Additional notification handlers (Slack/webhook) + criteria-rich dispatch #323
• Hanging-action detection + alerting: Error Recovery: Hanging-action detection + alerting integration #324
• Async/background dispatch (required prerequisite): Event Manager: Async/background EventHandler dispatch #338

Service-vs-plugin decision

Tracked by #325 (Post-1.0 decision card). Before re-opening that question, the recovery primitives above need to ship and operate in production long enough for the team to have hard data on plugin codebase size, Event Manager restart frequency caused by plugin issues, scaling-pressure mismatch, and operator UX demand.

If the answer is "promote to a service," the AbstractEventHandler contract pinned by #339 becomes the migration boundary — well-factored handlers move with minimal change.

Open questions

• Should error records own their own document storage, or extend the Event Manager's?
• What expression language for pattern matching (simple key-value, JSONPath, custom DSL)?
• Integration with external incident management tools (PagerDuty, Opsgenie, etc.) — likely overlaps with Error Recovery: Additional notification handlers (Slack/webhook) + criteria-rich dispatch #323's notification handlers; clarify ownership.
• LLM backends for agentic troubleshooters: local-first, Claude API, configurable?
• How should troubleshooter rulesets be versioned and shared across labs?

Roadmap metadata

• Phase: 1.1 — Near-term Post-1.0
• Project: Advanced Error Recovery
• Blocked by: Error Recovery: Re-decide Event-Manager-plugins vs dedicated Error Manager #325 (decision), and implicitly the recovery primitives reaching production

[RyanTheRobothead]

Roadmap update recommendation for #247: Feature: Error Manager Service

Roadmap phase: 1.1 — Near-term Post-1.0
Project: Advanced Error Recovery

Why this issue is being updated

RE-FRAME. The recovery primitives (rule engine, recovery workflows, notifications, hanging-action detection) ship as EventHandler plugins inside Event Manager through 1.0 — they do NOT require a dedicated service. The remaining value of #247 is specifically the LIFECYCLE LAYER and AGENTIC LAYER on top of those primitives: error-record lifecycle (new/ack/investigating/resolved/recurring), pattern detection across event history, dedicated dashboard view for error triage, agentic troubleshooters. Recommend updating the issue to scope it to those layers explicitly, and to flag that the 'spin out a service' decision is deferred to draft-err-promote-mvp Post-1.0.

Related roadmap work

Blocked by: draft-err-promote-mvp

Suggested action

Re-scope the issue body to align with the rationale above. Add the recommend-update label so the next roadmap review knows it's been triaged.