Make the scheduled jobs incremental, with a --full sweep for manual edits

[wasabipesto-bot]

The 6-hourly jobs run takes ~80 minutes at 40–70 GB RSS and causes an hour-long window of API pool-timeout 503s every run (prometheus shows it plainly: 10:48–12:08 UTC today, every 5-minute bucket elevated). Root causes, quantified against production:

• Consensus re-evaluates history. get_fields_in_base_with_detailed_subs has no notion of "new" — 3.70M fields for base 49, ~7M across bases, one query each, every run.
• Downsampling loads a base at a time. 3.58M canon submissions for base 49 in one load(), materialized as decoded serde_json::Values (~10–20 KB each against 1.5 KB on disk) — tens of GB transient — then the converted set was cloned wholesale for base-level stats, plus a .cloned() per chunk. Evicting the page cache (postgres runs 128 MB shared_buffers against a 305 GB database) is what turns this into API 503s.

Neither pass needs history: consensus is a pure function of a field's submissions, and chunk/base stats derive from field check levels + canon submissions, which move only at submit time or in the consensus pass itself.

What this does

Incremental by default, bounded by a one-row job_state watermark (highest processed submission id):

• Consensus runs only for fields with new detailed submissions in the window — found by one pkey range scan over the submissions tail, not per-base sweeps.
• Statistics recompute only for chunks whose fields received any submission (niceonly submissions bump check levels 0→1 at submit time, so both modes dirty a chunk). Untouched bases are skipped outright.
• Base totals come from the ~100 freshly-updated chunk rows instead of re-aggregating every field.
• The watermark advances at the end of a successful run, held back by a 10k-id safety margin: MAX(id) is not a transactionally safe boundary (an id allocated before the snapshot can commit after it), and everything keyed on the watermark is idempotent, so re-examining a few seconds' worth of stragglers is the cheap side of that trade. A run that fails partway leaves the watermark untouched and the window is redone.

Streaming everywhere it still reads bulk data. Canon submissions are fetched one chunk at a time and folded through new DistributionAccumulator / NumbersAccumulator types — batch-order-independent by construction (the histogram is a count-sum; a top-N of a set is contained in the top-N of every superset), with unit tests asserting fold-in-batches ≡ single-pass. Peak memory now tracks the largest chunk (~tens of MB) instead of the largest base, in full mode too. The old whole-base query and its double-clone are deleted.

--full / just jobs-full: the previous behavior, kept for manual edits. Disqualifying a submission creates no new submission, so the watermark cannot see it — the full sweep re-evaluates every field with detailed submissions and recomputes every chunk and base, then advances the watermark like any run.

Verified end-to-end against a seeded database

The disqualification lifecycle specifically, since that's the workflow that must stay trustworthy:

step	result
fresh fixture, incremental run	140 fields evaluated/updated, only the 2 dirty chunks recomputed, chunk 3 untouched, base stats + top-numbers correct
rerun with nothing new	Skipped 1 bases with no new submissions, 0 work, 0.0s
one new submission (chunk 2)	exactly 1 field evaluated (consensus → CL3), only chunk 2 recomputed
disqualify a canon submission	incremental run: no change (expected — this is the documented gap)
just jobs-full	field reset to CL1/no-canon with the WARNING line, chunk checked_detailed 130000→129000, distribution total follows, the disqualified submission's nice number drops out of chunk and base top-numbers, base totals update

Also covered: the full sweep trued up a chunk whose minimum_cl had never been written (seeded state no real flow produces), confirming its audit role. Accumulator equivalence has pure unit tests, including across the top-N compaction boundary; the full nice_common suite passes (105 tests).

Deploying

1. Apply schema/migrations/2026-08-20_job_state.sql (creates the table and initializes the watermark to the current max submission id — no CONCURRENTLY, no locks of note, safe anytime). It assumes stats are current as of applying; if in doubt, just jobs-full once.
2. Pull. The cron entry needs no change — just jobs is now incremental; just jobs-full is the manual sweep.

Expected steady state: minutes per run at <100 MB, no more 6-hourly 503 windows. The full sweep keeps its old ~hour duration (consensus is still one query per field there — an acceptable cost for a manual audit path, and its memory problem is fixed by the streaming regardless).

Logging drops from one line per field/chunk — millions of lines per run through cronic — to one line per change plus per-base summaries. Measured honestly: that's log hygiene, not performance; stdout writes were never a material cost.

🤖 Generated with Claude Code