critical: FlowRoutines::boot() on init takes down the whole network — non-terminating as_unschedule_all_actions loop across concurrent workers

[chubes4]

Summary

Deploying Data Machine to extrachill.com took the entire 10-site multisite network down for ~45 minutes. Every site shares one PHP-FPM pool, and FlowRoutines::boot() saturated it from the Events site (blog 7, 718 flows).

This is not "slow under load" — concurrent workers enter a mutually non-terminating loop. Deactivating the plugin did not stop the already-running workers; only killing them did.

Symptoms

• Load average 64+ on an otherwise idle box; MariaDB pegged at ~88% CPU.
• 58–63 concurrent copies of a single query, permanently:

  SELECT a.action_id FROM c8c_7_actionscheduler_actions a
  LEFT JOIN c8c_7_actionscheduler_groups g ON g.group_id=a.group_id
  WHERE 1=1 AND g.slug='agents-api'
    AND a.hook='wp_agent_routine_run_scheduled'
    AND a.args='{"routine_id":"flow-93"}'
    AND a.status IN ('pending')
  ORDER BY a.scheduled_date_gmt ASC LIMIT 0, 1

• nginx logging upstream timed out ... while reading response header for every site in the network (extrachill.com, community, events, wire…).
• debug.log full of WordPress database error Query execution was interrupted with this stack:

  do_action('init') → FlowRoutines::boot → register_routine_logged
    → WP_Agent_Routine_Registry::register → do_action('wp_agent_routine_registered')
    → register-routine-bridge-sync.php:28
    → HashGatedRoutineBackend->register
    → WP_Agent_Routine_Action_Scheduler_Bridge->register
    → as_unschedule_all_actions → as_unschedule_action
    → ActionScheduler_DBStore->query_actions
  

Mechanism

1. FlowRoutines::boot() is hooked to init on every request (inc/Engine/Actions/Engine.php:209). On blog 7 that re-declares 718 flow routines per request, plus system schedules.
2. Each register() that misses the hash gate reaches WP_Agent_Routine_Action_Scheduler_Bridge::register(), whose first act is as_unschedule_all_actions( $hook, $args, $group ) "for idempotency".
3. Action Scheduler implements that as an unbounded loop:

   do {
       $unscheduled_action = as_unschedule_action( $hook, $args, $group );
   } while ( ! empty( $unscheduled_action ) );

   It terminates only when no matching pending action exists.
4. Meanwhile other concurrent workers are inserting pending actions for the same (hook, args, group) via as_schedule_recurring_action(). Worker A's loop keeps finding rows created by worker B and vice versa. No loop ever exits. 86 exact duplicate pending actions were created inside single seconds (10 copies of flow-141 at 17:30:09–17:30:10), confirming there is no lock or unique constraint on the check-then-insert.
5. Because the request never finishes, it never reaches HashGatedRoutineBackend::persist() at the end of boot(). The fingerprints are never written, so the next request repeats the entire pass. The gate can never converge once the box is loaded — it is a latch that only closes when the system is already healthy.
6. inc/setup/flow-schedules.php makes it worse on deploy: activation sets datamachine_flow_schedule_reconciliation_pending, and the init handler runs FlowRoutines::reconcile( true ), which calls HashGatedRoutineBackend::set_verification_mode( true ) — unconditional pass-through for all 718 routines, in web requests, on every request, until a pass fully succeeds. Under the load it causes, it never succeeds, so the marker is never deleted.

Observed directly: re-activating the plugin re-created the marker (marked_at matching the activation second) and the meltdown resumed within ~80 seconds.

Why it was unrecoverable without root

wp plugin deactivate data-machine --network did not stop it. Workers that had already loaded the plugin stayed inside the as_unschedule_all_actions loop indefinitely, still inserting and deleting rows and keeping each other alive. 61 PHP-FPM workers were stuck >20 minutes. There is no request_terminate_timeout configured, so FPM never reaped them. Recovery required terminating the workers.

Recovery performed

1. wp plugin deactivate data-machine --network + per-site deactivation.
2. Deleted datamachine_flow_schedule_reconciliation_pending from every site's options table.
3. Deleted 1,249 runaway wp_agent_routine_run_scheduled pending/in-process rows; deduped 86 duplicates; truncated actionscheduler_claims.
4. Terminated stuck PHP-FPM workers.

Data Machine is currently deactivated network-wide on production. It cannot be safely re-activated until this is fixed — activation alone re-arms the reconciliation marker.

What needs to change

1. boot() must not re-register every routine on every request. Declaring 718 routines per page load is the defect; the hash gate is a mitigation for a design that shouldn't exist. Register on schedule-change, activation, and an explicit reconcile — not on init.
2. The fingerprint must be committed before the expensive work, or incrementally per routine — not once at the end of a pass that cannot complete. A gate that only persists on full success cannot recover a degraded system.
3. Single-flight the whole pass. A site-level lock (transient/GET_LOCK) so exactly one process reconciles; everyone else no-ops immediately. This alone would have prevented the outage.
4. Never run reconcile inline in a web request. The datamachine_reconcile_marked_flow_schedules marker should dispatch one background/WP-CLI pass, with the marker cleared (or backed off) even on failure so it cannot spin forever.
5. Stop calling as_unschedule_all_actions() per routine per boot. Its unbounded do/while is unsafe under concurrency by construction. Compare against the existing schedule and only mutate on an actual diff.
6. Bound the blast radius: a routine count over some threshold should log once and refuse the inline path rather than attempt 718 registrations in a page request.

Environment

• data-machine 0.176.8 (bug present since dca1d22c2 / refactor(scheduling): converge flow and system-task scheduling on Agents API Routines #3475, deployed 13:01 UTC; the 17:15 deploy re-armed the marker and triggered the meltdown).
• agents-api vendored at vendor/wordpress/agents-api (2026-09-09).
• Events site: 718 flows, ~103k rows in actionscheduler_actions.
• WordPress multisite, 10 sites, one shared PHP-FPM pool.