From 085f5af13ee2ca391de83f14ea5cdfabd4b48254 Mon Sep 17 00:00:00 2001 From: Tomas Valenta <49156497+ValentaTomas@users.noreply.github.com> Date: Sat, 25 Jul 2026 15:25:06 -0700 Subject: [PATCH 1/2] fix(db): make rare status groups visible to the planner on env_builds (#3393) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary `InvalidateUnstartedTemplateBuilds` runs 2–60s instead of milliseconds because of a statistics blind spot, not scan cost: `status_group='pending'` is ~0.006% of env_builds (measured 19,906 of ~333M rows), so the default ~30k-row statistics sample never catches it — the value is absent from the column's MCV list and the planner estimates ~11 rows. That drives the update's join from the pending-scan side (~20k probes into env_build_assignments per execution, fresh-session EXPLAIN attached below) instead of one selective `(env_id, tag)` lookup. Every template-build registration then queues behind it on the envs row lock — the current upsert convoy (20–70s waits). Raising the per-column statistics target to 2000 (~600k-row sample) makes rare-but-hot status groups visible and flips the join order; the migration ANALYZEs immediately so the fix takes effect at promote rather than at the next autoanalyze. Same per-table-settings precedent as the autovacuum migrations. Fresh-session plan today (store-resident columnar unit present and unused — the misestimate, not columnar, is the bug): `Index Scan using idx_env_builds_status_group (rows=11 estimated, rows=19905 actual)` → nested-loop probing env_build_assignments ×19,905. ## Verification (post-draft) - **Scratch replay**: full migration chain via `scripts/migrator.go` on Postgres 16 applies cleanly; `goose down` (SET STATISTICS -1 + ANALYZE) and re-up also clean. - **Live read-only plan capture** (`PREPARE` + `EXPLAIN EXECUTE` ×7): custom plans (executions 1–5) are the pending-driven nested loop — est 11 rows, cost 662, the convoy plan; the generic plan (execution 6+) already flips to the assignments `(env_id, tag)` drive at cost 223. Production's pooled prepared statements re-run custom plans on every fresh connection, so busy templates keep landing on the bad side — this migration corrects the one wrong number in that comparison. - **Worst case bounded**: env_build_assignments is 323M rows; the heaviest template has ~438k assignment rows and the largest single `(env_id, tag)` group is ~219k. With truthful stats on both sides the planner picks the cheaper of the two known join orders — ≈ status-quo cost for that one template, ~1000× improvement for typical ones. - **Lock envelope**: ALTER…SET STATISTICS and ANALYZE both take SHARE UPDATE EXCLUSIVE (no read/DML blocking; queues behind a concurrent autovacuum on the same table at worst), bounded by the 1h timeout and rerunnable. Down is an instant revert. ## Postscript: the mechanism was observed live the same day At 16:09Z a routine autoanalyze happened to catch 'pending' into the MCV (frequency 0.0001) and the convoy vanished within the half-hour — rows returned collapsed from 14–17M/s (steady since 08:00Z) to ~0.5M/s, with no deploy, no columnar change, and the pending count unchanged. That is the dice roll this migration removes: at 0.0001 the value is one bad sample away from disappearing again, and every autoanalyze re-flips the coin. Target 2000 makes the catch deterministic (~36 expected sample hits). --- ...500_env_builds_status_group_statistics.sql | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 packages/db/migrations/20260725100500_env_builds_status_group_statistics.sql diff --git a/packages/db/migrations/20260725100500_env_builds_status_group_statistics.sql b/packages/db/migrations/20260725100500_env_builds_status_group_statistics.sql new file mode 100644 index 0000000000..8bb1e60933 --- /dev/null +++ b/packages/db/migrations/20260725100500_env_builds_status_group_statistics.sql @@ -0,0 +1,35 @@ +-- +goose Up +-- +goose NO TRANSACTION + +-- ANALYZE cannot run inside a transaction; bound the pass so it neither +-- hangs unbounded nor dies to a short session default. +SET statement_timeout = '1h'; + +-- status_group='pending' rows are ~0.006% of the table (measured 19,906 of +-- ~333M), which the default statistics sample (~30k rows) statistically +-- never catches: 'pending' is absent from the column's MCV list, so the +-- planner estimates ~11 rows and drives InvalidateUnstartedTemplateBuilds +-- from the pending-scan side — ~20k index probes into env_build_assignments +-- per execution instead of one selective (env_id, tag) lookup. A larger +-- per-column sample makes rare-but-hot status groups visible and flips the +-- join order back. +ALTER TABLE public.env_builds ALTER COLUMN status_group SET STATISTICS 2000; + +-- Rebuild the column stats immediately: autoanalyze would otherwise wait +-- for the change threshold, leaving the misestimate live for hours. +ANALYZE public.env_builds; + +-- The migrator session is reused for subsequent migrations: restore its +-- baseline (scripts/migrator.go sets 3h per connection). +SET statement_timeout = '3h'; + +-- +goose Down +-- +goose NO TRANSACTION + +SET statement_timeout = '1h'; + +ALTER TABLE public.env_builds ALTER COLUMN status_group SET STATISTICS -1; + +ANALYZE public.env_builds; + +SET statement_timeout = '3h'; From 8682f561a665215440999af051e9787043066afd Mon Sep 17 00:00:00 2001 From: Tomas Valenta <49156497+ValentaTomas@users.noreply.github.com> Date: Sat, 25 Jul 2026 15:32:56 -0700 Subject: [PATCH 2/2] chore(db): reword stats-migration comment --- ...100500_env_builds_status_group_statistics.sql | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/packages/db/migrations/20260725100500_env_builds_status_group_statistics.sql b/packages/db/migrations/20260725100500_env_builds_status_group_statistics.sql index 8bb1e60933..15cc5bc353 100644 --- a/packages/db/migrations/20260725100500_env_builds_status_group_statistics.sql +++ b/packages/db/migrations/20260725100500_env_builds_status_group_statistics.sql @@ -5,14 +5,14 @@ -- hangs unbounded nor dies to a short session default. SET statement_timeout = '1h'; --- status_group='pending' rows are ~0.006% of the table (measured 19,906 of --- ~333M), which the default statistics sample (~30k rows) statistically --- never catches: 'pending' is absent from the column's MCV list, so the --- planner estimates ~11 rows and drives InvalidateUnstartedTemplateBuilds --- from the pending-scan side — ~20k index probes into env_build_assignments --- per execution instead of one selective (env_id, tag) lookup. A larger --- per-column sample makes rare-but-hot status groups visible and flips the --- join order back. +-- status_group='pending' is rare enough (well under a hundredth of a percent +-- of the table, measured in production) that the default statistics sample +-- statistically never catches it: 'pending' is absent from the column's MCV +-- list, so the planner underestimates it by orders of magnitude and drives +-- InvalidateUnstartedTemplateBuilds from the pending-scan side — tens of +-- thousands of index probes into env_build_assignments per execution instead +-- of one selective (env_id, tag) lookup. A larger per-column sample makes +-- rare-but-hot status groups visible and flips the join order back. ALTER TABLE public.env_builds ALTER COLUMN status_group SET STATISTICS 2000; -- Rebuild the column stats immediately: autoanalyze would otherwise wait