Harden recovery verification and submission runbook

CLAUDE.md

@@ -53,11 +53,15 @@ acquire `SourceLock` — they take only the request gate.
 
 ### Processing semantics (plan §8)
 
-At-least-once execution + idempotent writes + unique constraints =
-effectively-once persisted outcomes. Never claim or code against exactly-once
-execution.
+Repeated observation and job execution are expected. State only the two proved ingestion
+invariants: a duplicate GitHub event ID cannot create another `push_events` row, and that
+duplicate cannot register entity activity or reactivate a `skipped_budget` entity.
+Executions, ingestion runs, quarantine occurrence counts, budget debits, and logs may
+repeat or change. Recovery before commit is conditional on the event remaining in a later
+sliding-feed response. Never claim or code against exactly-once execution or universal
+idempotency of persisted state.
 
-### Idempotency invariants (plan §7)
+### Duplicate-event invariants (plan §7)
 
 - `push_events` inserts use `ON CONFLICT (github_event_id) DO NOTHING RETURNING id`.
 - Entity activity fields (`last_seen_at`, `latest_event_at`, reactivation) update
@@ -75,7 +79,7 @@ Fixture mode fails closed — never a live fallback.
 ## Database changes
 
 Schema changes go through migrations with intentional indexes, constraints where
-correctness depends on them, and tests for uniqueness/idempotency (plan §7, §12).
+correctness depends on them, and tests for uniqueness and replay behavior (plan §7, §12).
 
 ## Testing rules (plan §12)
 

app/controllers/api/push_events_controller.rb

@@ -22,7 +22,7 @@ def index
     # ambiguous — real GitHub event ids are numeric strings, so only one reading can be
     # right. §7 keeps the surrogate key out of this application's identity vocabulary
     # entirely (even the foreign keys target github_id), github_event_id is the unique index
-    # the whole idempotency story rests on, and it is the identifier §11 puts on every log
+    # that prevents a second event row, and it is the identifier §11 puts on every log
     # line — which is what makes "log line -> record" a URL a reviewer can type.
     #
     # find_by! rather than find_by plus an explicit render: ApplicationController maps

app/jobs/application_job.rb

@@ -12,7 +12,7 @@
 # uncoordinated Active Job ladder would re-poll a source whose backoff was just written and
 # spend the hourly allowance twice on the same failure. The 60-second recurring tick is the
 # retry — an escaped exception is a defect, so it fails the execution, job.failed says why,
-# and the next tick starts from committed state.
+# and a later successful scheduled tick starts from committed state.
 #
 # No discard_on ActiveJob::DeserializationError either: no job here takes a record argument.
 class ApplicationJob < ActiveJob::Base

app/jobs/reconcile_pending_enrichments_job.rb

@@ -3,10 +3,9 @@
 # seconds.
 #
 # **This is the crash-recovery mechanism, and its input is committed entity state.** The
-# post-commit enqueue in Github::IngestionRunner is a hint; a process killed between the
-# COMMIT and the enqueue, a worker that was down for an hour, a job that failed permanently
-# — all of them lose the hint, and none of them lose the work, because the entity rows still
-# say `pending` and the partial index that answers that predicate has existed since PR 3.
+# post-commit enqueue in Github::IngestionRunner is a hint. If a crash loses that hint while
+# the entity row remains eligible and `pending`, a later successful scheduled tick can find
+# it through the partial index that has answered that predicate since PR 3.
 #
 # **Entity-scoped, structurally.** It reads github_actors, github_repositories and one
 # github_api_budget row through Github::Enrichment::Dispatch, and never push_events. §8's

app/models/push_event.rb

@@ -27,8 +27,8 @@ class PushEvent < ApplicationRecord
             format: { with: SHA_FORMAT,
                       message: "must be 40 or 64 hexadecimal characters" }
 
-  # The idempotent insert the whole durability story rests on (§7, §8). Returns the
-  # new row's id, or nil when the event was already persisted — and the caller uses
+  # The duplicate-event insert gate the accepted-row guarantee rests on (§7, §8). Returns
+  # the new row's id, or nil when the event was already persisted — and the caller uses
   # exactly that distinction to decide whether entity activity may be updated, so a
   # re-polled window cannot resurrect skipped entities.
   #

app/services/github.rb

@@ -12,7 +12,7 @@
 #
 # Configuration is memoised per process and cleared by config/initializers/github.rb
 # on every reload, so a development edit takes effect without a restart while a
-# production process reads the environment exactly once.
+# production process retains its first environment-derived configuration.
 module Github
   class << self
     def configuration

app/services/github/budget_ledger.rb

@@ -488,9 +488,8 @@ def debit!(request_class, now:, share_cap:, borrow:)
     #
     # Emitted from the debit that *reached* the allowance, not from denial_reason. A
     # denial recurs on every attempt — under PR 8's recurring task that is a line a minute
-    # for the rest of the window, burying the stream §11 explicitly sizes. The debit that
-    # takes used to allowance happens exactly once per class per window, which is what a
-    # transition means.
+    # for the rest of the window, burying the stream §11 explicitly sizes. Only the debit
+    # that first takes used to allowance represents the class transition for that window.
     def log_class_exhausted(budget, request_class)
       used = class_used(budget, request_class)
       allowance = class_allowance(budget, request_class)
@@ -511,8 +510,8 @@ def log_class_exhausted(budget, request_class)
     #
     # Two edges worth naming. With a guarantee of zero the INFO never fires — the
     # post-debit share is 1 and never 0 — which is correct, because nothing transitioned:
-    # that class was borrowing from its first request. And under a sustained borrow the
-    # INFO still fires exactly once, at the guarantee, with budget.class_exhausted
+    # that class was borrowing from its first request. Under a sustained borrow, the INFO
+    # fires on the debit that first reaches the guarantee, with budget.class_exhausted
     # following later at the class cap.
     def log_share_transitions(budget, request_class, borrow:)
       return unless enrichment?(request_class)

app/services/github/enrichment/dispatch.rb

@@ -7,8 +7,9 @@ module Enrichment
     # "is there durable enrichment work this class could do right now?" — and the answer is
     # read from the committed entity rows and the ledger, never from the queue. That is what
     # makes the enqueue a *hint*: §2A's outbox-style recovery says "the committed entity
-    # state is the durable record of pending work", so a process killed between the COMMIT
-    # and the enqueue loses the hint and never the work.
+    # state is the durable record of pending work". A process killed between the COMMIT and
+    # enqueue can lose that hint while leaving eligible pending entity state discoverable by
+    # a later successful reconciler tick.
     #
     # **At most one job per class per call**, however deep the backlog. §5 gives each class
     # one job and Github::EnrichmentRunner enriches at most one entity per call, so the

app/services/github/ingestion_runner.rb

@@ -48,8 +48,8 @@ module Github
   # jobs being class-scoped: the runner chooses the entity by fairness, so N enqueues carry
   # no more information than one, and §8's property that matters — the enqueue happens after
   # the rows are durable — is stronger here, where every commit of the run is behind us.
-  # Losing the dispatch to a crash loses nothing: ReconcilePendingEnrichmentsJob sweeps the
-  # same committed state every 60 seconds.
+  # If a crash loses this dispatch hint, the committed pending entity state remains.
+  # ReconcilePendingEnrichmentsJob can rediscover eligible work on a later successful tick.
   class IngestionRunner
     MONOTONIC = -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) }
 

app/services/inspection/push_event_view.rb

@@ -3,8 +3,8 @@ module Inspection
   #
   # A module of functions rather than a method on PushEvent, following the convention this
   # codebase already holds twenty times over: there are twenty #to_log methods and not one
-  # of them lives on a model. A model owns its columns, its constraints and its idempotent
-  # write; how a row is shaped for a reader belongs to the reader — and there are two
+  # of them lives on a model. A model owns its columns, constraints, and duplicate-event
+  # insert gate; how a row is shaped for a reader belongs to the reader — and there are two
   # readers here whose answers deliberately differ, so a single #to_api would need a mode
   # flag on the model.
   #

config/application.rb

@@ -76,9 +76,9 @@ class Application < Rails::Application
     # long for in-flight work and no longer. docker-compose.yml pairs it with
     # stop_grace_period: 30s, leaving margin for the supervisor to reap its children.
     #
-    # A job still *waiting* for the gate has reserved nothing and is safe to kill: §8's
-    # at-least-once execution with idempotent writes is what makes that true, and both
-    # advisory locks die with the session.
+    # A job still *waiting* for the gate has not reserved budget, so terminating it cannot
+    # leak a debit. A redelivery re-enters the normal selector, and both advisory locks die
+    # with the session.
     config.solid_queue.shutdown_timeout = 20.seconds
   end
 end

config/recurring.yml

@@ -4,14 +4,15 @@
 # PollEventSourceJob computes effective_poll_time and no-ops unless a poll is due". A tick
 # is not a poll: against the default POLL_INTERVAL_SECONDS (300) roughly four ticks in five
 # find nothing due and cost one indexed SELECT. That ratio is the point — it is what makes a
-# source that becomes due at T get polled within a minute of T, without a cadence that the
-# allowance formula (12 poll requests an hour, zero headroom) cannot pay for.
+# source that becomes due at T become eligible on the next successful scheduled tick,
+# without a cadence that the allowance formula (12 poll requests an hour, zero headroom)
+# cannot pay for.
 #
 # reconcile_pending_enrichments — §8 step 11, the sweep behind the outbox-style recovery.
 # The post-commit enqueue in Github::IngestionRunner is only a hint; a SIGKILL between the
-# COMMIT and the enqueue loses the hint and never the work, because the committed entity
-# rows are the durable record. This task rediscovers that work, entity-scoped, and enqueues
-# nothing when there is none or when the ledger says the class is spent.
+# COMMIT and the enqueue can lose the hint while the committed pending entity state remains.
+# On a later successful tick, this task can rediscover eligible work from that state and
+# enqueues nothing when there is none or when the ledger says the class is spent.
 #
 # The cadences live here rather than in an environment variable because §9's cadence is
 # POLL_INTERVAL_SECONDS and it is enforced in the scheduling components, not in the tick.

docker-compose.yml

@@ -133,10 +133,9 @@ services:
   # stop_grace_period pairs with SolidQueue.shutdown_timeout (20s, set in
   # config/application.rb from §2A's pinned HTTP timeouts): 20s for an in-flight
   # attempt that already holds the request gate, and 10s of margin for the
-  # supervisor to reap its children. A job still *waiting* for the gate has
-  # reserved nothing and is safe to kill — §8's at-least-once execution with
-  # idempotent writes is what makes that true, and both advisory locks die with
-  # the session.
+  # supervisor to reap its children. A job still *waiting* for the gate has not
+  # reserved budget, so terminating it cannot leak a debit; a redelivery re-enters
+  # the normal selector, and both advisory locks die with the session.
   #
   # No healthcheck. Nothing depends on this service, `unless-stopped` already
   # covers process death, and the only probe that could tell a hung worker from