Skip to content

[codex] add durable heartbeat scheduler - #11

Draft
divo12 wants to merge 4 commits into
codex/paired-ab-gatesfrom
codex/durable-heartbeat-scheduler
Draft

[codex] add durable heartbeat scheduler#11
divo12 wants to merge 4 commits into
codex/paired-ab-gatesfrom
codex/durable-heartbeat-scheduler

Conversation

@divo12

@divo12 divo12 commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Stack

Depends on #10. Review against codex/paired-ab-gates.

Problem

Manual Mine and Fit services do not create an always-running improvement loop. OFW needs a restart-safe local control plane that only reconciles and enqueues work on heartbeat, while typed workers execute bounded jobs under immutable policies.

Implementation

  • Adds a domain-specific SQLite scheduler using BEGIN IMMEDIATE; no general workflow engine or service-mode claim.
  • Persists typed jobs, required/optional dependency edges, attempts, lease ownership, budget reservations, results, revision circuit state, and the scheduler heartbeat lease.
  • Freezes the full automation policy in the database and the exact Fit policy digest into every Fit job.
  • Uses deterministic job IDs over typed immutable specs and dependency edges, making duplicate heartbeats idempotent.
  • Reconciles expired leases, charges their reserved budget fail-closed, retries within the frozen attempt cap, and rejects late completions by lease token.
  • Allows long jobs to renew live leases through a typed JobContext.
  • Reserves maximum job cost atomically with the lease and settles actual cost on completion/failure/cancellation.
  • Enforces the exact Fit predicate: successful Mine and benchmark export, matching Mine-result lineage and harness revision, optional memory terminal state, exact policy digest, one active Fit, cooldown, quiet hours, and a closed circuit.
  • Prevents overlapping Mine and Fit jobs for the same harness revision.
  • Opens a manual-resume circuit after three consecutive no-progress Fit results.
  • Materializes TRACE_SYNC → MINE → good/benchmark/memory exports → FIT without running heavy work in the heartbeat.
  • Excludes OFW control-plane evidence from the production improvement pipeline.
  • Provides typed worker dispatch, retry/terminal/optional failures, status, cancel, skip, resume, daemon, and public ofw.serve(...) surfaces.

TDD evidence

  • Duplicate enqueue and duplicate heartbeat produce one durable DAG.
  • Required failure stops its branch; failed optional memory does not block an otherwise valid Fit.
  • Wrong benchmark/Mine lineage and wrong Fit-policy digest block Fit.
  • Only one Mine and one Fit may be active per harness revision.
  • Scheduler restart recovers an expired running lease; its late completion cannot overwrite the retry.
  • Expired tokens fail even before reconciliation, and unknown worker-loss spend is charged at the reserved cap.
  • Cancellation and resume survive restart; reopening with another automation policy is rejected.
  • Quiet hours, hard daily budget reservation, Fit cooldown, no-progress circuit, and manual circuit resume are enforced.
  • Two SQLite claimers racing for the last budget slot produce exactly one lease.
  • A worker renews its lease and a typed handler commits one result exactly once.
  • Invalid or over-budget handler results terminate promptly without leaking their reservation.

Verification

  • pytest -q — 143 passed
  • Ruff and strict mypy across src and tests — clean
  • forbidden construct scan — no Any, raw dict, object, getattr, setattr, lambdas, or __import__
  • Bandit — zero findings
  • pip-audit — no known dependency vulnerabilities
  • source distribution and wheel build successfully

Deliberate scope

  • Local v0 serializes scheduler writes with SQLite; PostgreSQL/distributed leasing is deferred until local throughput requires it.
  • Heartbeat and workers are separate loops. Concrete Mine/Fit handlers and Git promotion are added by the next stacked layers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant