architecture: job liveness is a second copy of a fact Action Scheduler already owns — the drift class exists because the state is duplicated

[chubes4]

The observation

A job row carries status = 'processing'. Action Scheduler independently carries the pending/in-progress action that will actually advance that job. These are two copies of one fact: is this run alive?

Every "stuck job" incident is that duplication drifting. When a worker dies between finishing a step and scheduling the next one, the row still says processing and Action Scheduler has nothing — and nothing reconciles them (#3478). The row is not wrong because of a bug in one write path; it is wrong because the fact has two homes and only one of them was updated.

grep -rn "status.*=.*'processing'" inc --include=*.php → ~25 write sites.

Why file this separately

#3478 proposes a reconciler: a recurring task that detects rows whose claim has no live Action Scheduler action and resumes or terminalizes them. That is a reasonable stopgap and probably worth shipping. But structurally a reconciler is a daemon whose purpose is to keep two copies of one fact agreeing, and that is usually the thing you build immediately before concluding that one of the copies should not exist.

Data Machine currently has ~670 PHP files / ~230k lines in inc/, with inc/Core/Database/Jobs/Jobs.php at 4,440 lines (#3461). Adding permanent machinery to make a duplication maintainable is a real cost, and it forecloses the cheaper question. Worth asking the question once, explicitly, before the reconciler becomes load-bearing.

The question

Split job state into two kinds and ask whether they belong in the same place:

• Liveness — processing, and "which step is next". Action Scheduler already owns this authoritatively: it has the action, the schedule, the claim, the attempt. A persisted mirror can only ever be stale.
• Terminal / business outcome — completed, failed, completed_no_items, agent_skipped. These have no Action Scheduler analogue, must outlive the 2-day AS retention window (liveness: no_scheduler_path is guaranteed by the 2-day Action Scheduler retention window, so 39 of 50 'stuck' jobs are pruning artifacts #3479), and are legitimately Data Machine's to own.

If liveness were derived from Action Scheduler rather than mirrored into the jobs table, the drift class disappears by construction — there is no second copy to go stale, and no reconciler needed for it. The terminal column stays exactly as it is.

What would need to be true

Not asserting this is tractable. Concretely, someone needs to establish:

1. Every read of status === 'processing' and what it actually needs — a cheap boolean, an ordering key, an index for a list query, or genuine cross-request durability.
2. Whether Action Scheduler can answer those reads at acceptable cost. Listing jobs by liveness currently indexes a local column; asking AS per row does not scale the same way. A derived-but-cached projection may be the honest middle, and if so, say so — a cache with a single writer is a different animal from two independent writers.
3. Whether operation_generation / operation_claim_token (already carried in datamachine_execute_step args) are sufficient to identify the owning action for a job without a mirrored column.
4. Whether the multi-step batch/child-job shapes (datamachine_pipeline_batch_chunk, waiting_children) survive a derived model, or genuinely need their own persisted coordination state.

The acceptable outcomes include "the column earns its place, keep it and keep the reconciler" — that is a real answer, and documenting why is worth as much as changing it. What is not acceptable is never asking, and accumulating reconcilers.

Related

• engine: a worker killed mid-step orphans its job forever — no stale-claim reconciliation, and recover-stuck only terminalizes #3478 — the reconciler (stopgap; its PR is asked to answer "is this the right layer?" from the code)
• liveness: no_scheduler_path is guaranteed by the 2-day Action Scheduler retention window, so 39 of 50 'stuck' jobs are pruning artifacts #3479 — no_scheduler_path is guaranteed by the 2-day AS retention window; the same duplication seen from the reporting side
• audit: six overlapping item-identity ledger tables (processed/tracked/batch items, run_metadata, post_identity, reservations) #3460 — six overlapping item-identity ledger tables: the same duplication disease one layer down
• refactor(post-1.0): split Jobs.php (4,440 lines, 58 public methods) by responsibility #3461 — Jobs.php at 4,440 lines / 58 public methods

Filed by Extra Chill Bot (AI agent) on behalf of @Chubes, prompted by production forensics on orphaned jobs and a review of whether the proposed fix was the right shape.