Skip to content

db: add completed env_build_assignments tip-lookup index and harden its statistics - #3403

Merged
ValentaTomas merged 2 commits into
mainfrom
eba-tip-lookup-index
Jul 27, 2026
Merged

ValentaTomas merged 2 commits into
mainfrom
eba-tip-lookup-index

Conversation

@ValentaTomas

@ValentaTomas ValentaTomas commented Jul 27, 2026

Copy link
Copy Markdown
Member

The current-tip lookup orders by (created_at DESC, build_id DESC) within an (env_id, tag) equality, but the existing index stops at created_at, leaving a top-N sort whose plan choice depends on sampled statistics — an autoanalyze re-sample can temporarily flip it to a much slower plan. Completing the index to the full sort key makes the lookup a pure ordered index scan (LIMIT 1, no sort) that stays cheapest under any statistics roll. Also raises the statistics targets on the key columns and re-analyzes, same treatment as env_builds.status_group.

This PR is deliberately ADDITIVE — it only creates the new index and hardens statistics. Dropping the superseded 3-column index (identical ordering prefix, redundant write maintenance) is split into a follow-up PR (#3409) to merge only after this one has soaked in production.

Query-shape check across every first-party consumer of this table: each query falls into (a) the (env_id[, tag]) prefix family — served identically or better by the new index, since it shares the exact ordering prefix; (b) build_id lookups — served by idx_env_build_assignments_build, untouched; or (c) primary-key/uniqueness paths. The (created_at DESC, build_id DESC) tie-break matches the high-frequency tip check that motivated this change; this repository's own latest-build laterals order by created_at alone and ride the same prefix unchanged. Latest-default-build laterals that select build_id can now run index-only.

@cla-bot cla-bot Bot added the cla-signed label Jul 27, 2026
@cursor

cursor Bot commented Jul 27, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Concurrent index creation and an extra index on a hot table add deployment-time load and temporary write overhead until the redundant index is removed, but there is no application logic change.

Overview
Adds a non-transactional migration that builds a concurrent index on env_build_assignments matching (env_id, tag) filters with created_at DESC, build_id DESC ordering so current-tip lookups can use a LIMIT 1 ordered index scan instead of relying on a sort whose plan can swing after autoanalyze. It also bumps statistics targets on env_id and tag and runs ANALYZE, mirroring the earlier env_builds.status_group hardening. The change is additive (the older three-column index is left in place for a follow-up drop after soak); retries guard against a leftover INVALID concurrent index.

Reviewed by Cursor Bugbot for commit b63abf2. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov

codecov Bot commented Jul 27, 2026

Copy link
Copy Markdown

❌ 5 Tests Failed:

Tests completed Failed Passed Skipped
3514 5 3509 7
View the top 3 failed test(s) by shortest run time
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_1_lists_root_directory
Stack Traces | 0.01s run time
=== RUN   TestListDir/depth_1_lists_root_directory
=== PAUSE TestListDir/depth_1_lists_root_directory
=== CONT  TestListDir/depth_1_lists_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_1_lists_root_directory
--- FAIL: TestListDir/depth_1_lists_root_directory (0.01s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
Stack Traces | 0.01s run time
=== RUN   TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== PAUSE TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
=== CONT  TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory)
--- FAIL: TestListDir/depth_2_lists_first_level_of_subdirectories_(in_this_case_the_root_directory) (0.01s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_0_lists_only_root_directory
Stack Traces | 0.02s run time
=== RUN   TestListDir/depth_0_lists_only_root_directory
=== PAUSE TestListDir/depth_0_lists_only_root_directory
=== CONT  TestListDir/depth_0_lists_only_root_directory
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_0_lists_only_root_directory
--- FAIL: TestListDir/depth_0_lists_only_root_directory (0.02s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir
Stack Traces | 4.44s run time
=== RUN   TestListDir
=== PAUSE TestListDir
=== CONT  TestListDir
--- FAIL: TestListDir (4.44s)
github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestMaskRequestHostAPIParameter
Stack Traces | 4.87s run time
=== RUN   TestMaskRequestHostAPIParameter
=== PAUSE TestMaskRequestHostAPIParameter
=== CONT  TestMaskRequestHostAPIParameter
Executing command python3 in sandbox ij213xjkojqx563huwml5 (user: root)
    mask_request_host_test.go:44: Command [python3] output: event:{start:{pid:1127}}
Executing command ls in sandbox i4y9qt7dwif7eg9ipmtdw
    mask_request_host_test.go:68: Command [cat] output: event:{start:{pid:1128}}
    mask_request_host_test.go:68: Command [cat] output: event:{data:{stderr:"cat: /tmp/nc_output.txt: No such file or directory\n"}}
    mask_request_host_test.go:68: Command [cat] output: event:{end:{exit_code:1 exited:true status:"exit status 1" error:"exit status 1"}}
    mask_request_host_test.go:69: 
        	Error Trace:	.../tests/proxies/mask_request_host_test.go:69
        	Error:      	Received unexpected error:
        	            	command cat in sandbox ij213xjkojqx563huwml5 failed with exit code 1
        	Test:       	TestMaskRequestHostAPIParameter
--- FAIL: TestMaskRequestHostAPIParameter (4.87s)
View the full list of 1 ❄️ flaky test(s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestListDir/depth_3_lists_all_directories_and_files

Flake rate in main: 30.77% (Passed 9 times, Failed 4 times)

Stack Traces | 0.01s run time
=== RUN   TestListDir/depth_3_lists_all_directories_and_files
=== PAUSE TestListDir/depth_3_lists_all_directories_and_files
=== CONT  TestListDir/depth_3_lists_all_directories_and_files
    filesystem_test.go:96: 
        	Error Trace:	.../tests/envd/filesystem_test.go:96
        	Error:      	Received unexpected error:
        	            	unavailable: 502 Bad Gateway
        	Test:       	TestListDir/depth_3_lists_all_directories_and_files
--- FAIL: TestListDir/depth_3_lists_all_directories_and_files (0.01s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@ValentaTomas
ValentaTomas marked this pull request as ready for review July 27, 2026 03:34

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — a well-scoped index migration.

Extended reasoning...

Overview

Single-file DB migration touching env_build_assignments. It extends an existing 3-column index (env_id, tag, created_at DESC) to a 4-column index (env_id, tag, created_at DESC, build_id DESC), drops the now-superseded 3-column index, raises statistics targets on env_id/tag, and re-analyzes the table. A down migration cleanly reverses all of it.

Security risks

None. No user input, no permissions/auth logic — pure schema/index/statistics change.

Level of scrutiny

This is exactly the kind of mechanical, well-precedented change that doesn't need deep human scrutiny: it follows the same pattern as the referenced prior migration (20251218170000_optimize_build_assignment_indexes.sql), uses CONCURRENTLY + NO TRANSACTION correctly, sets an explicit statement_timeout and restores the migrator's baseline afterward, and creates the new index before dropping the old one so there's no coverage gap. I independently checked the two queries that hit this table with an ORDER BY eba.created_at DESC LIMIT 1 shape (get_template_with_build_by_tag.sql, get_last_snapshot.sql) — neither ties on build_id, but that's harmless: the extra trailing key column only adds a deterministic tie-break the query doesn't request, it doesn't change which rows satisfy the ORDER BY created_at DESC clause, so the index-only ordered scan still applies. The referenced old index name/definition (idx_env_build_assignments_env_tag_created) matches what's actually in the migration history, so the drop is safe and not a guess.

Other factors

The bug-hunting system found no issues, and the one candidate raised (tie-break column not matching any actual ORDER BY) was already investigated and ruled out as harmless — confirmed by my own read of the query call sites above. This is a narrowly-scoped, reversible, low-risk change.

@ValentaTomas ValentaTomas changed the title db: complete env_build_assignments tip-lookup index and harden its statistics db: add completed env_build_assignments tip-lookup index and harden its statistics Jul 27, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Index sort key mismatches queries
    • Changed index sort key from build_id DESC to id DESC to match actual query patterns that use (created_at DESC, id DESC) in build pagination laterals.

Create PR

Or push these changes by commenting:

@cursor push 90355c4cd9
Preview (90355c4cd9)
diff --git a/packages/db/migrations/20260727032500_env_build_assignments_tip_lookup_index.sql b/packages/db/migrations/20260727032500_env_build_assignments_tip_lookup_index.sql
--- a/packages/db/migrations/20260727032500_env_build_assignments_tip_lookup_index.sql
+++ b/packages/db/migrations/20260727032500_env_build_assignments_tip_lookup_index.sql
@@ -5,7 +5,7 @@
 -- bound each pass so it neither hangs unbounded nor dies to a short default.
 SET statement_timeout = '2h';
 
--- The current-tip lookup orders by (created_at DESC, build_id DESC) under an
+-- The current-tip lookup orders by (created_at DESC, id DESC) under an
 -- (env_id, tag) equality, but the existing index stops at created_at: the
 -- planner still needs a top-N sort, so its plan choice rides on per-column
 -- estimates that every autoanalyze re-samples. Completing the index to the
@@ -17,9 +17,9 @@
 -- dropped by a separate follow-up migration once the new index has soaked.
 -- A failed CONCURRENTLY build leaves an INVALID index that a plain retry
 -- would trip over; drop it first.
-DROP INDEX CONCURRENTLY IF EXISTS idx_env_build_assignments_env_tag_created_build;
-CREATE INDEX CONCURRENTLY idx_env_build_assignments_env_tag_created_build
-    ON public.env_build_assignments (env_id, tag, created_at DESC, build_id DESC);
+DROP INDEX CONCURRENTLY IF EXISTS idx_env_build_assignments_env_tag_created_id;
+CREATE INDEX CONCURRENTLY idx_env_build_assignments_env_tag_created_id
+    ON public.env_build_assignments (env_id, tag, created_at DESC, id DESC);
 
 -- Stabilize the estimate inputs themselves: larger per-column samples cut the
 -- roll-to-roll variance of n_distinct/MCV on the lookup's key columns (same
@@ -39,7 +39,7 @@
 
 SET statement_timeout = '2h';
 
-DROP INDEX CONCURRENTLY IF EXISTS idx_env_build_assignments_env_tag_created_build;
+DROP INDEX CONCURRENTLY IF EXISTS idx_env_build_assignments_env_tag_created_id;
 
 ALTER TABLE public.env_build_assignments ALTER COLUMN env_id SET STATISTICS -1;
 ALTER TABLE public.env_build_assignments ALTER COLUMN tag SET STATISTICS -1;

You can send follow-ups to the cloud agent here.

Reviewed by Cursor Bugbot for commit 4c163ac. Configure here.

@ValentaTomas
ValentaTomas force-pushed the eba-tip-lookup-index branch from 06b4f3a to 3e08429 Compare July 27, 2026 06:32
@ValentaTomas
ValentaTomas enabled auto-merge (squash) July 27, 2026 06:59
@ValentaTomas
ValentaTomas merged commit 0c933ca into main Jul 27, 2026
41 checks passed
@ValentaTomas
ValentaTomas deleted the eba-tip-lookup-index branch July 27, 2026 06:59
ValentaTomas added a commit that referenced this pull request Jul 27, 2026
Follow-up to #3403 (the additive half): the 4-column index shares the
3-column index's exact ordering prefix, so every scan it served runs
identically on the new one — keeping both only doubles write maintenance
on a hot-insert table. DO NOT merge until #3403 has soaked in
production; goose orders this migration after it by timestamp either
way.

Schema proof, both definitions in this repo: the 3-column index
([20251218160000
L42-L43](https://github.com/e2b-dev/infra/blob/7c23f7bd724e9217be27c5eca1bd8c8402a2fae1/packages/db/migrations/20251218160000_allow_m_n_builds_with_tags.sql#L42-L43),
`(env_id, tag, created_at DESC)`) is the exact leading prefix of the
4-column one ([20260727032500
L24-L25](https://github.com/e2b-dev/infra/blob/7c23f7bd724e9217be27c5eca1bd8c8402a2fae1/packages/db/migrations/20260727032500_env_build_assignments_tip_lookup_index.sql#L24-L25),
`(env_id, tag, created_at DESC, build_id DESC)`). Why a leading prefix
makes the shorter index redundant: [PostgreSQL docs — multicolumn
indexes](https://www.postgresql.org/docs/current/indexes-multicolumn.html).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants