engine: stale-claim reconciliation for crashed workers (proposal, draft)

[chubes4]

engine: stale-claim reconciliation for crashed workers (proposal)

Opened as a DRAFT on purpose. The implementation is complete and gated, but the reviewer question this PR really asks is not "is this code correct" — it is "is a reconciler the right shape at all, or is it a daemon that exists to keep two copies of one fact agreeing?" See Is this the right layer? below, answered from what the code actually shows. My conclusion: necessary stopgap, with a concrete follow-up direction that could shrink it. Merge decision is a human call.

Closes #3478

Root cause

A worker that dies between completing step N and scheduling step N+1 leaves the job row status = 'processing' with no Action Scheduler action that will ever advance it, and nothing in the engine notices. status and "an AS action exists for this claim" are two copies of one fact — "this run is alive" — written by the same request; when the worker dies mid-request, only AS-side knowledge dies with it, and the row's copy persists forever. Verified on production (events.extrachill.com): jobs 1015119/1015122/1015124 had step_results for step 0, frozen run_lifecycle.updated_at, and zero rows in actionscheduler_actions despite being inside the retention window — and AS held no failed actions either, so the death left no trace on the scheduler side. The Redis outage that killed the workers was incidental; OOM, PHP fatal, timeout, or a deploy mid-flight all produce the same orphan.

What this PR does

A recurring engine task (stale_claim_reconciliation, five-minute cadence — the same shape as the high-churn retention_as_actions schedule: SystemTask handler + datamachine_recurring_schedules entry, dispatched through the existing datamachine/dispatch-system-task routine wake; no parallel scheduling path) runs a bounded reconciliation pass:

1. Detect. Candidates are processing jobs with no pending AS action and no fresh in-progress action for the job (exact decoded-args match, mirroring RecoverStuckJobsAbility::getActiveStepActionIds semantics: pending is live unconditionally, in-progress only inside the freshness window) whose run_metrics.last_activity_at (fallback created_at) is older than the threshold. Bounded batch of 50 per tick.
2. Resume where safe. The resume point is the first step in execution order with no step_results entry (never started → nothing applied → running it is the safe resume), scheduled as a unique datamachine_execute_step action, job returned to pending, attempt recorded.
3. Terminalize loudly where not. See the decision table; reasons land in the compound status (failed - worker_died_mid_step etc. — stored per the canonical base+job_status_reason convention) plus full context under engine_data.stale_claim_reconciliation.
4. wp datamachine jobs recover-stuck is untouched as the operator escape hatch; it is simply no longer the only thing between a crash and a permanent orphan.

Existing primitives reused (and what I deliberately did not build)

• Claim identity: the operation_generation / operation_claim_token AS args — no new claim mechanism.
• Live-claim checks: DirectJobEnqueuer::liveGenerationExecution() for direct jobs; the LIKE-scan + exact-args-decode pattern from RecoverStuckJobsAbility for flow jobs.
• Direct-job recovery: DirectOperationRecoveryPolicy::diagnose(), Jobs::commit_missing_direct_operation_requeue() (fenced requeue), Jobs::transition_missing_direct_operation() (fenced terminalize), getProcessingSystemTaskChildren() — the recurring task delegates; it adds no direct-job machinery of its own.
• Resume-point resolution: ExecutionPlan::from_flow_config() + engine_data.step_results with the same completed-step semantics as JobRetryPolicy::stepCompletedSuccessfully().
• Attempt bound: engine_data.retry.attempts — the same counter JobRetryPolicy draws from, so retries and crash-resumes share one bounded budget (default 3, filter datamachine_stale_claim_max_resume_attempts).
• Recurring task wiring: SystemTask + RecurringScheduleRegistry + Agents API Routines, exactly like the retention tasks.

Deliberately not built: a new claim/lease table, a heartbeat column, a job-state state machine rewrite, or a second recovery codepath for direct jobs. Also deliberately not touched: the AS retention windows (that is #3479's discussion), RecoverStuckJobsAbility (the operator pass stays authoritative for overrides, pathless children, and pending-AI deferrals), and both CLI files owned by the sibling PR.

Detection rule and threshold

status = 'processing' AND no pending/fresh-in-progress AS action for the job AND last_activity_at older than 15 minutes (filter datamachine_stale_claim_activity_threshold), candidates prefiltered by created_at < now - threshold so the query stays cheap.

Justification: an AS action is the only mechanism that advances a processing job, so an actionless processing row is already beyond the engine's recovery contract — the threshold exists only to stay clear of legitimate scheduling latency (queue backlog, replication lag, the seconds-long gap between one action completing and the next being inserted). Fifteen minutes is an order of magnitude beyond any of those while keeping crash-to-recovery latency inside a few ticks of the five-minute cadence.

Resume-vs-terminalize decision table

Evidence	Disposition
Live pending action (any hook)	skip — run is alive
Fresh in-progress action (inside threshold window)	skip
Batch parent row (source = 'batch')	skip — BatchScheduler/PathlessBatchRecovery own it
engine_data.job_status override present	skip — operator pass owns overrides
Activity inside threshold window	skip
Direct job, effects not begun	resume via fenced requeue (commit_missing_direct_operation_requeue; fence loss ⇒ skip — a newer claim owner advanced it)
Direct job, effects begun	terminalize scheduler_path_lost_after_effects (+ processing system-task children), reusing the operator pass's exact reason
Flow job, first incomplete step never started, attempts under bound	resume from it
Flow job, first incomplete step already started (recorded but not successful)	terminalize worker_died_mid_step — effects cannot be proven absent
Flow job, attempts at bound	terminalize stale_claim_resume_exhausted — no infinite loop
Flow job, step result is waiting	skip — a webhook gate owns the resume; scheduling past the gate would bypass it
Flow job, all steps complete, or plan unresolvable	terminalize stale_claim_unresumable — completion accounting cannot be safely synthesized

The generation-advanced case from the issue is enforced structurally: resumption only ever happens through a fenced commit on the current row's generation+token, so a stale claim can never resume the job — it loses the fence and skips.

Tests

tests/Unit/Core/StaleClaimReconcilerTest.php (WP_UnitTestCase + real Action Scheduler), covering the four required scenarios plus two:

1. processing job with a live pending action → NOT reconciled (skipped/live_action_exists, row untouched).
2. actionless stale job with step 0 completed → resumed: re-scheduled from step 1, row pending, retry.attempts = 1, resume action present.
3. attempts at bound → terminalized (stale_claim_resume_exhausted) instead of looping.
4. direct job with begun effects → terminalized (scheduler_path_lost_after_effects), not resumed.
5. mid-step death (resume step recorded but unsuccessful) → worker_died_mid_step.
6. recent activity → skipped.

No existing assertions were weakened or removed.

Gates

• homeboy review lint data-machine --placement local (per touched file; runs PHPCS (WordPress) + ESLint + PHPStan level 7): pass, no findings on inc/Core/StaleClaimReconciler.php, inc/Engine/AI/System/Tasks/StaleClaimReconciliationTask.php, inc/Engine/AI/System/SystemAgentServiceProvider.php, tests/Unit/Core/StaleClaimReconcilerTest.php.
• homeboy review test data-machine --placement local (full SQLite suite): pass — 1520 tests, 32 skipped, 0 failures (includes the six new tests; no existing test regressed).
• PR CI (homeboy-test.yml) additionally runs the full MySQL suite, which covers the transaction-fenced paths exercised by scenario 4.

Is this the right layer?

Asked because the deeper suspicion is fair: status = 'processing' is a second copy of a fact Action Scheduler already owns, and a reconciler is structurally a daemon whose job is keeping two copies of one fact agreeing. What the code shows:

1. Where is job status written and read? Transitions are centralized in Jobs.php (7 canonical methods: create_job, start_job, update_job_status, transition_job_status, complete_job, plus the fenced transition_missing_direct_operation / transition_recovery_owned_child variants), but those are called from 21 files / 65 call sites across inc/, and 24 files reference the processing state. Reads are heavier: 86 get_job()/get_job_metadata() call sites and ~160 ['status'] checks on job rows. Could reads derive liveness from AS instead? Only the live ones. Most read sites are terminal-state or reporting reads — status filters in get_jobs/count_old_jobs, JobsSummaryAbility/RunMetricsAbility/CLI tabulations — which AS cannot answer after its retention prunes history.

2. What does the status column give us that AS cannot? Three concrete things. (a) Terminal history with reasons: execute_step/resume_ai_step AS actions are pruned after 1 hour (retention_as_actions hook windows: 1/24 day), while job rows are kept 30 days — the column is the only durable record a job ever ran, how it ended, and why (failed - <reason> compounds, cancelled, plus statuses with no AS analogue at all: completed_no_items, agent_skipped). (b) Cross-request queries: retention deletes, summaries, dashboards, and dedup all query WHERE status IN (...) directly against the table; deriving that from AS args-JSON scans would be strictly worse. (c) Terminal accounting inputs: the operation_* / terminal_accounting_* columns that fence recovery live on the same row. So yes — the column earns its place. What it does not earn is the processing value specifically: that is the one state whose truth AS already owns.

3. Would AS-authoritative liveness eliminate the drift class, or move it? The codebase already ran this experiment for direct jobs: the operation_state / operation_generation / operation_claim_token / operation_action_id columns make AS authoritative for direct-job liveness, with DirectOperationRecoveryPolicy + fenced requeue/terminalize as the enforcement. Drift still happened — the fencing and recovery primitives exist because a duplicated fact drifted (enqueue crashed between schedule and receipt). What the fencing bought was not "no drift" but bounded, detectable, owner-verified drift. Making AS authoritative for flow-job liveness the same way would shrink this orphan class to the same bounded shape — it would not vanish, because every status write still happens in-request on a worker that can die mid-write. Fully eliminating the class would mean never persisting liveness at all: treat processing as a derived view (row is either pending-with-an-action or terminal), which is a real architectural change across those 24 files, the status machine, and terminal accounting — not a PR-sized change.

4. Recommendation. Necessary stopgap, right layer for now — with a follow-up worth filing. The reconciler is the flow-job counterpart of the direct-job recovery machinery that already exists; it reuses that machinery rather than parallel-building, it is bounded (50 candidates/tick, shared attempt bound), and it converts an invisible permanent orphan into a resume or a loud, reasoned terminal state within minutes. That logic — resume from last completed step vs terminalize loudly — is worth having under any architecture, because even a derived-liveness engine still needs to decide what to do with a dead run's partial effects. The durable fix direction is the one point 3 points at: extend the direct-job operation fencing to flow jobs so "is anything scheduled" becomes owner-verifiable per claim, shrinking the reconciler to a small safety net. I would rather land this stopgap behind the draft flag and file that follow-up than hold crash recovery hostage to the larger refactor. Happy to close this in favor of the architectural approach if that is the call — the decision table and tests transfer.

---

This PR was authored by an AI coding agent (Extra Chill Bot minion) and is pending human review.