Skip to content

db: drop the superseded env_build_assignments tip-lookup index - #3409

Merged
ValentaTomas merged 1 commit into
mainfrom
drop-superseded-eba-tip-index
Jul 27, 2026
Merged

ValentaTomas merged 1 commit into
mainfrom
drop-superseded-eba-tip-index

Conversation

@ValentaTomas

@ValentaTomas ValentaTomas commented Jul 27, 2026

Copy link
Copy Markdown
Member

Follow-up to #3403 (the additive half): the 4-column index shares the 3-column index's exact ordering prefix, so every scan it served runs identically on the new one — keeping both only doubles write maintenance on a hot-insert table. DO NOT merge until #3403 has soaked in production; goose orders this migration after it by timestamp either way.

Schema proof, both definitions in this repo: the 3-column index (20251218160000 L42-L43, (env_id, tag, created_at DESC)) is the exact leading prefix of the 4-column one (20260727032500 L24-L25, (env_id, tag, created_at DESC, build_id DESC)). Why a leading prefix makes the shorter index redundant: PostgreSQL docs — multicolumn indexes.

@cla-bot cla-bot Bot added the cla-signed label Jul 27, 2026
@cursor

cursor Bot commented Jul 27, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches index lifecycle on a high-write table; safe only if the additive 4-column index is already live in production, otherwise tip lookups could regress until rollback.

Overview
This follow-up migration drops the older (env_id, tag, created_at DESC) index on env_build_assignments now that the 4-column tip-lookup index covers the same leading prefix, so writes on that hot table no longer maintain duplicate indexes. The up path uses DROP INDEX CONCURRENTLY with a bounded statement_timeout, and the down path can rebuild the dropped index concurrently with timeout disabled and a pre-drop cleanup for any invalid leftover from an interrupted build.

Reviewed by Cursor Bugbot for commit ce2ba97. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov

codecov Bot commented Jul 27, 2026

Copy link
Copy Markdown

❌ 4 Tests Failed:

Tests completed Failed Passed Skipped
3514 4 3510 7
View the top 3 failed test(s) by shortest run time
github.com/e2b-dev/infra/tests/integration/internal/tests/api/templates::TestUpdateTemplateNotOwnedByTeam
Stack Traces | 156s run time
=== RUN   TestUpdateTemplateNotOwnedByTeam
=== PAUSE TestUpdateTemplateNotOwnedByTeam
=== CONT  TestUpdateTemplateNotOwnedByTeam
    template_update_test.go:205: Build failed: {<nil> An internal error occurred. Please try again or contact support with the build ID. <nil>}
--- FAIL: TestUpdateTemplateNotOwnedByTeam (155.77s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/templates::TestTemplateBuildFromTemplateStartCommand
Stack Traces | 170s run time
=== RUN   TestTemplateBuildFromTemplateStartCommand
=== PAUSE TestTemplateBuildFromTemplateStartCommand
=== CONT  TestTemplateBuildFromTemplateStartCommand
    build_template_test.go:133: test-ubuntu-start-base: [info] Building template pzzw0240lsbn8wowjcwj/52305415-e0e0-4f09-9219-e194ff0ecfd7
    build_template_test.go:133: test-ubuntu-start-base: [info] [base] FROM ubuntu:22.04 [f9f564014e009a9561a82bf8c84f9314242971e833fb019936654ecba452f184]
    build_template_test.go:133: test-ubuntu-start-base: [info] Base Docker image size: 30 MB
    build_template_test.go:133: test-ubuntu-start-base: [info] Creating file system and pulling Docker image
    build_template_test.go:133: test-ubuntu-start-base: [info] Uncompressing layer sha256:d6834b4a794c03efa2c998853e64969fa8851b11b2ade63292268872a37759d0 30 MB
    build_template_test.go:133: test-ubuntu-start-base: [info] Uncompressing layer sha256:1fe7d5a1fffe336c9a84a3207ff14f5eb40f1dfeb351811e722a7f804a8876f4 13 MB
    build_template_test.go:133: test-ubuntu-start-base: [info] Uncompressing layer sha256:8c4b1b28875140ed3abacaf16ad0d696f6bef912f52d2148f261a23e3349465b 168 B
    build_template_test.go:133: test-ubuntu-start-base: [info] Layers extracted
    build_template_test.go:133: test-ubuntu-start-base: [info] Root filesystem structure: bin, boot, dev, etc, home, lib, lib32, lib64, libx32, media, mnt, opt, proc, root, run, sbin, srv, sys, tmp, usr, var
    build_template_test.go:133: test-ubuntu-start-base: [info] Provisioning sandbox template
    build_template_test.go:133: test-ubuntu-start-base: [info] Provisioning was successful, cleaning up
    build_template_test.go:133: test-ubuntu-start-base: [info] Sandbox template provisioned
    build_template_test.go:133: test-ubuntu-start-base: [info] [base] DEFAULT USER user [49e586c2171254c6bc4a09e84eedac32dbcf113a158c24248129af2f49cbed74]
    build_template_test.go:133: test-ubuntu-start-base: [info] [builder 1/2] ENV START_ENV start_value [22ecb2c2ce1b1d262e37d2cd71b90e6d33fdf99ecfca361e1a2c284afb7fc08a]
    build_template_test.go:133: test-ubuntu-start-base: [info] [builder 2/2] WORKDIR /start/workdir [f841574929b92d8b6e15a8fc6477f0f1e568eac3d30ae7110a831bbcea62e921]
    build_template_test.go:133: test-ubuntu-start-base: [info] [finalize] Finalizing template build [a509f6bf83c53e4844e4d62e9e93f7dae0d5706e070f3f756fe6d227a04976e4]
    build_template_test.go:133: test-ubuntu-start-base: [error] Build failed: An internal error occurred. Please try again or contact support with the build ID.
    build_template_test.go:700: Build failed: {<nil> An internal error occurred. Please try again or contact support with the build ID. <nil>}
--- FAIL: TestTemplateBuildFromTemplateStartCommand (169.99s)
github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestCommandKillNextApp
Stack Traces | 288s run time
=== RUN   TestCommandKillNextApp
=== PAUSE TestCommandKillNextApp
=== CONT  TestCommandKillNextApp
    process_test.go:30: Build failed: {<nil> An internal error occurred. Please try again or contact support with the build ID. <nil>}
--- FAIL: TestCommandKillNextApp (287.74s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/templates::TestTemplateBuildInstalledPackagesAvailable
Stack Traces | 300s run time
=== RUN   TestTemplateBuildInstalledPackagesAvailable
=== PAUSE TestTemplateBuildInstalledPackagesAvailable
=== CONT  TestTemplateBuildInstalledPackagesAvailable
    build_template_test.go:133: test-ubuntu-packages-available: [info] Building template etapf4uu2cvoxp8afukh/512df65e-6b0f-42a3-887e-d1e27856226b
    build_template_test.go:133: test-ubuntu-packages-available: [info] [base] FROM ubuntu:22.04 [f9f564014e009a9561a82bf8c84f9314242971e833fb019936654ecba452f184]
    build_template_test.go:133: test-ubuntu-packages-available: [info] Base Docker image size: 30 MB
    build_template_test.go:133: test-ubuntu-packages-available: [info] Creating file system and pulling Docker image
    build_template_test.go:133: test-ubuntu-packages-available: [info] Uncompressing layer sha256:d6834b4a794c03efa2c998853e64969fa8851b11b2ade63292268872a37759d0 30 MB
    build_template_test.go:133: test-ubuntu-packages-available: [info] Uncompressing layer sha256:1fe7d5a1fffe336c9a84a3207ff14f5eb40f1dfeb351811e722a7f804a8876f4 13 MB
    build_template_test.go:133: test-ubuntu-packages-available: [info] Uncompressing layer sha256:8c4b1b28875140ed3abacaf16ad0d696f6bef912f52d2148f261a23e3349465b 168 B
    build_template_test.go:133: test-ubuntu-packages-available: [info] Layers extracted
    build_template_test.go:133: test-ubuntu-packages-available: [info] Root filesystem structure: bin, boot, dev, etc, home, lib, lib32, lib64, libx32, media, mnt, opt, proc, root, run, sbin, srv, sys, tmp, usr, var
    build_template_test.go:133: test-ubuntu-packages-available: [info] Provisioning sandbox template
    build_template_test.go:133: test-ubuntu-packages-available: [info] Provisioning was successful, cleaning up
    build_template_test.go:133: test-ubuntu-packages-available: [info] Sandbox template provisioned
    build_template_test.go:133: test-ubuntu-packages-available: [info] [base] DEFAULT USER user [49e586c2171254c6bc4a09e84eedac32dbcf113a158c24248129af2f49cbed74]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 1/15] RUN dpkg-query -W -f='${Status}' systemd | grep -q 'install ok installed' [de826f48fc8f28496580ff9e64f4cd6b59aaeff2811a9bae9b4fffe6645c289e]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 2/15] RUN dpkg-query -W -f='${Status}' systemd-sysv | grep -q 'install ok installed' [15d29149113620d83e56ad1512042bc0b247306802dcdb366ccf1b398b4c3cdb]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 3/15] RUN dpkg-query -W -f='${Status}' openssh-server | grep -q 'install ok installed' [c7ce729a2e1d50236dd8fa2338bcd463909c353853df395eeae5a6d86c5aaaff]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 4/15] RUN dpkg-query -W -f='${Status}' sudo | grep -q 'install ok installed' [401020ed635dcfa25a0d55aed045a457439f6d3712e439ecc3b33e4ba9db14eb]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 5/15] RUN dpkg-query -W -f='${Status}' chrony | grep -q 'install ok installed' [a355929061ef18ba0b427ce239bb340d317d7a078f06d70889300591280426e8]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 6/15] RUN dpkg-query -W -f='${Status}' socat | grep -q 'install ok installed' [d7e232005823e31f2c8a1e83ee175812553529abba39cf318b950f8963d1f969]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 7/15] RUN dpkg-query -W -f='${Status}' curl | grep -q 'install ok installed' [37a6ea018deed85204e0c0105881f83ca383e092f6e16b7fab1a42bb4b3254bc]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 8/15] RUN dpkg-query -W -f='${Status}' ca-certificates | grep -q 'install ok installed' [cf67f41976592227bbb1514b70f29ff6c3f187d36c227b7982bb823c86c78962]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 9/15] RUN dpkg-query -W -f='${Status}' fuse3 | grep -q 'install ok installed' [b0257640d1594b1b9ea66664d67097cbdaf423169d1769d43d697e0f8f44ae5b]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 10/15] RUN dpkg-query -W -f='${Status}' iptables | grep -q 'install ok installed' [1e7b717aefd403d504c973491220d70475fc8b7fa762444d2b6e37bfcaa2e84a]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 11/15] RUN dpkg-query -W -f='${Status}' git | grep -q 'install ok installed' [564561d2d945e6366c96a35e281a547dfd07bbfc4096352651cbfd121930d20f]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 12/15] RUN dpkg-query -W -f='${Status}' less | grep -q 'install ok installed' [7ed9ff27623a2f82daaec96e1f91e9d6b468447b732e0bcd5b61a948a0ae3c26]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 13/15] RUN dpkg-query -W -f='${Status}' nftables | grep -q 'install ok installed' [fab2299ee2f5ef1eb4f33ab12637b71234a2f7a21c8fdcd2bae150612084624f]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 14/15] RUN dpkg-query -W -f='${Status}' iputils-ping | grep -q 'install ok installed' [ad11c2e7f602b13f86e8f844936230c94e79bd96164a4abbdc5cf39f5fd5ba9a]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [builder 15/15] RUN dpkg-query -W -f='${Status}' jq | grep -q 'install ok installed' [13dad491042119a252a9b8d1861921889102ffdb6e0521e4940798ddf88a38b5]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [finalize] Finalizing template build [eb7b37d59c5e9b33cba4e503ad5ee98acad3960c365c3c520ed6d346004f79cb]
    build_template_test.go:133: test-ubuntu-packages-available: [info] [optimize] Optimizing template [f1746441596e199e31ff320cbc21dbca1a7bb77389dd59f9a3378b002c496de1]
    build_template_test.go:1060: 
        	Error Trace:	.../api/templates/build_template_test.go:96
        	            				.../api/templates/build_template_test.go:1060
        	Error:      	Received unexpected error:
        	            	Get "http://localhost:.../builds/512df65e-6b0f-42a3-887e-d1e27856226b/status?logsOffset=30&level=info": context deadline exceeded
        	Test:       	TestTemplateBuildInstalledPackagesAvailable
--- FAIL: TestTemplateBuildInstalledPackagesAvailable (300.00s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

ValentaTomas added a commit that referenced this pull request Jul 27, 2026
…ts statistics (#3403)

The current-tip lookup orders by (created_at DESC, build_id DESC) within
an (env_id, tag) equality, but the existing index stops at created_at,
leaving a top-N sort whose plan choice depends on sampled statistics —
an autoanalyze re-sample can temporarily flip it to a much slower plan.
Completing the index to the full sort key makes the lookup a pure
ordered index scan (LIMIT 1, no sort) that stays cheapest under any
statistics roll. Also raises the statistics targets on the key columns
and re-analyzes, same treatment as env_builds.status_group.

This PR is deliberately ADDITIVE — it only creates the new index and
hardens statistics. Dropping the superseded 3-column index (identical
ordering prefix, redundant write maintenance) is split into a follow-up
PR (#3409) to merge only after this one has soaked in production.

Query-shape check across every first-party consumer of this table: each
query falls into (a) the (env_id[, tag]) prefix family — served
identically or better by the new index, since it shares the exact
ordering prefix; (b) build_id lookups — served by
idx_env_build_assignments_build, untouched; or (c)
primary-key/uniqueness paths. The (created_at DESC, build_id DESC)
tie-break matches the high-frequency tip check that motivated this
change; this repository's own latest-build laterals order by created_at
alone and ride the same prefix unchanged. Latest-default-build laterals
that select build_id can now run index-only.
@ValentaTomas
ValentaTomas marked this pull request as ready for review July 27, 2026 07:18

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6b914eff81

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No bugs found, but this is a production index-maintenance migration on a hot-insert table (env_build_assignments) with an explicit operational precondition — the PR description says not to merge until the prerequisite additive migration (#3403) has soaked in production. That sequencing/timing judgment isn't something CI or automated review can verify, so it's worth a human sign-off before merge.

What was reviewed: the up/down migration logic (CONCURRENTLY drop/recreate, statement_timeout bounds and restoration, INVALID-index cleanup on rollback), and the claimed index-coverage rationale (4-column index shares the 3-column index's ordering prefix). The concerns raised by the bug-hunting system about connection-pool statement_timeout scoping and the prerequisite index not yet existing in this checkout were investigated and ruled out — the latter is because #3403 is intentionally not yet merged, not a bug.

Extended reasoning...

Overview

Single-file migration that drops the old 3-column tip-lookup index (idx_env_build_assignments_env_tag_created) on env_build_assignments now that a 4-column index with the same ordering prefix supersedes it. Uses DROP INDEX CONCURRENTLY with a bounded 1h statement_timeout on the up path, and a down path that clears any leftover INVALID index before rebuilding.

Security risks

None — this is a pure schema/index change with no new query paths, auth logic, or user-controllable input.

Level of scrutiny

Migrations that run CONCURRENTLY against a hot-insert production table warrant more scrutiny than an average PR, even though the SQL itself is mechanically simple. The bigger risk here isn't correctness of the SQL (which looks sound: NO TRANSACTION mode, bounded timeout, proper restoration to the 3h baseline, IF EXISTS/IF NOT EXISTS guards, INVALID-index cleanup on rollback) but the operational precondition stated in the PR description: this must not be merged until the prerequisite additive migration (#3403) has soaked in production. That's a human judgment call about deployment timing that automated review can't validate, so I'm deferring rather than approving outright.

Other factors

The bug-hunting system found no bugs. Candidate concerns it raised and ruled out were: whether the replacement 4-column index actually exists (it doesn't in this checkout, but that's expected — #3403 hasn't merged yet, consistent with the PR's stated precondition), and whether SET statement_timeout reliably applies to the connection executing DROP INDEX CONCURRENTLY given the migrator uses a pgxpool with MaxConns=4 (packages/db/scripts/migrator.go) — plausible in theory but not something I can confirm is a real issue without knowing whether goose pins a single connection per migration, and even if the SET didn't apply, the connection would just fall back to the pool's 3h AfterConnect default rather than being unbounded, so worst case is a longer wait, not correctness risk.

@ValentaTomas
ValentaTomas force-pushed the drop-superseded-eba-tip-index branch from 6b914ef to ce2ba97 Compare July 27, 2026 08:18
@ValentaTomas
ValentaTomas enabled auto-merge (squash) July 27, 2026 08:26
@ValentaTomas
ValentaTomas merged commit 6e25493 into main Jul 27, 2026
41 checks passed
@ValentaTomas
ValentaTomas deleted the drop-superseded-eba-tip-index branch July 27, 2026 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants