fix: batch parents stay processing after all children terminal — stale cache read + no durable recheck

[chubes4]

Closes #3518

Root cause (proven, not inferred)

maybeCompleteParent() made its children_complete decision on a cache-served parent engine snapshot. EngineData::retrieve() (inc/Core/EngineData.php:76-91) reads wp_cache_get('datamachine_engine_data') — persistent Redis on events — and its miss-fill is non-atomic against mutate():

1. Reader cache-misses → SELECT returns a pre-worklist_complete snapshot.
2. The final chunk worker's finishV2State() CAS commits worklist_complete=true and does wp_cache_set(fresh).
3. The reader's wp_cache_set(stale) lands last and poisons the entry.

A stranded parent's engine_data is never written again (only completion would rewrite it), so the poisoned entry is served forever. Every subsequent child callback read batch_pending=true and early-returned at the guard; the last child's callback — the one that should have completed the parent — read the same poisoned entry and stranded it.

Live evidence (events.extrachill.com, blog 7) matched every prediction:

• Parent 1213509: batch_scheduled=100, 100 children, 97 completed / 2 no_items / 1 failed / 0 open, worklist_complete=true in the DB (direct SQL), yet parent processing hours later → the decision-time read disagreed with the DB.
• All 100 children of 1213509 have terminal_accounting_state=4 (complete): every child's core_callbacks → onChildComplete did run, so the bug was inside the decision, not in hook delivery. (This also rules out the callback-before-status-write race: the child row is committed terminal before accounting stages run, Jobs.php:3198.)
• All 38 stuck parents since 09-17: batch_scheduled == child count == terminal children, worklist_complete=true — the early-return conditions were false in the DB for every one of them, so only a stale read explains the stranding. Hypothesis 2 (dedupe shrinking child count below batch_scheduled) is dead: parent 1212793 has batch_scheduled=19 = exactly its 19 children.

What changed

PipelineBatchScheduler (owning layer):

• maybeCompleteParent() split into inspectParent() + reconcileParent(). The decision now reads the parent snapshot via Jobs::retrieve_engine_data() — database-direct — never the object cache.
• batch_results persists through EngineData::mutate() CAS instead of a blind whole-snapshot datamachine_set_engine_data write (which could clobber concurrent mutations and re-poison the cache with the stale snapshot it was handed).
• New failure modes re-arm the idempotent chunk action (BatchScheduler::scheduleFinalizeRetry(), now public): worklist fence missing with zero active children (dead final-chunk worker), counts-query failure, and persist/complete failure. A fully scheduled, fully terminal parent can no longer end up with no future scheduler action — the invariant the docblock promises is now structural, and it also self-heals the dead-final-chunk-worker shape.
• reconcileParent() / parentCompletionDue() exposed for recovery tooling; the pipeline_batch core-callback bool contract is preserved (false only when a due completion could not be made durable, keeping the accounting stage retryable — existing tests test_parent_engine_write_failure_keeps_child_core_stage_retryable / test_parent_completion_failure_keeps_child_core_stage_retryable pass unchanged).

RecoverStuckJobsAbility:

• Pipeline batch parents route through reconcileParent() before the scheduler-ownership guard (pruned AS evidence otherwise shields them forever), instead of the timeout path that marked children-succeeded parents as failed. Dry run reports would_complete_batch_parent + predicted status; not_due parents fall through to the existing ownership/pathless/timeout handling unchanged. New batch_parents_completed counter surfaces in the ability result, message, log context, and CLI.

Verification

• 3 new PHPUnit tests, all fail on unfixed code, pass on fixed (verified via git stash):
  • stale-cache snapshot cannot strand a fully terminal parent (the engine: batch parents stay processing after all children are terminal — children_complete never lands (~6–22/day on events) #3518 reproduction),
  • unfenced worklist + terminal children re-arms a pending chunk action and the recheck completes the parent,
  • recover-stuck completes instead of timing out a fully terminal batch parent.
• Full tests/Unit suite (1955 tests) on a MySQL harness: zero new failures vs clean origin/main (junit diff — identical failure sets, which are pre-existing environment issues in this local harness).
• phpcs --standard=WordPress on all changed files: only pre-existing violation classes (camelCase method names matching file conventions, filename sniff); no new docblock or real issues from this change.

Notes / follow-ups

• Not addressed here (substrate, separate issue worth filing): EngineData::retrieve()'s miss-fill race exists for every consumer; filling with wp_cache_add instead of wp_cache_set would stop readers from poisoning a fresher writer's value. I kept this PR in the owning layer, but the substrate hardening would shrink the stale window platform-wide.
• recover-stuck already requeues v2 batches with an unfenced worklist (PathlessBatchRecovery); after this PR the fenced-but-terminal shape is also handled, so both stranding shapes reconcile.
• After deploy, watch wp --url=https://events.extrachill.com datamachine jobs liveness --limit=200 for 24h: no new no_scheduler_path batch parents should accumulate; existing 115 can be cleared with one recover-stuck --limit=100 pass (dry run first — it now reports would_complete_batch_parent).

Squash-merge note: conventional commits used; no CHANGELOG or version strings touched.