Replace standalone bundle with a fork-based runner in the file-by-file CI part - #1293
Merged
Conversation
The run_specs_file_by_file CI part boots RSpec once per spec file (288 files) to verify each file can run in isolation. To avoid paying for bundler on every boot, it used `bundle install --standalone` with binstubs, which loads gems via a static $LOAD_PATH file without ever activating them through RubyGems. That leaves `Gem.loaded_specs` empty, which breaks gems that consult it -- most recently elastic-transport 8.5.2+, which crashed on every request and forced us to pin the gem (see #1290 and elastic/elastic-transport-ruby#128). Instead, boot bundler and rspec-core once in a small runner script and run each spec file in a fork of that parent. The parent loads no ElasticGraph or spec code, so each child still starts from the same clean slate as a freshly booted rspec process, preserving the isolation guarantee -- but children inherit a normally-activated bundle, so gems that consult `Gem.loaded_specs` behave as they do in production, and we can drop the standalone workaround.
Booting bundler in the runner parent mutates ENV (BUNDLE_GEMFILE, RUBYOPT, etc.) and captures that mutated state in `Bundler::ORIGINAL_ENV`, which forked children inherit. A freshly booted rspec process has neither: it loads bundler lazily with the unmodified environment. This difference broke a schema_definition spec that shells out to `bundle` inside `Bundler.with_original_env` -- the shelled command picked up the repo root's BUNDLE_GEMFILE instead of the temp project's Gemfile. Snapshot the environment before booting bundler and restore both ENV and `Bundler::ORIGINAL_ENV` afterwards, so children observe the same environment a fresh process would.
jwils
marked this pull request as ready for review
July 3, 2026 23:49
jwils
requested review from
BrianSigafoos-SQ,
bsorbo,
ellisandrews-toast,
jwondrusch,
marcdaniels-toast,
myronmarston and
rossroberts-toast
as code owners
July 3, 2026 23:49
myronmarston
approved these changes
Jul 6, 2026
myronmarston
left a comment
myronmarston
left a comment
Collaborator
There was a problem hiding this comment.
This is awesome! Thanks @jwils.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
run_specs_file_by_fileCI part boots RSpec once per spec file (currently 288 files) to verify every spec file can run in isolation. To avoid paying bundler's boot cost 288 times, it usedbundle install --standalone+ binstubs, which loads gems from a static $LOAD_PATH file without activating them through RubyGems — soGem.loaded_specsis empty at runtime.That's a subtly nonstandard gem environment, and it recently bit us: elastic-transport 8.5.2+ consults
Gem.loaded_specs['multi_json']on every request and crashed only in this CI part, forcing a version pin in #1290 (upstream fix: elastic/elastic-transport-ruby#128).What
A small runner script (
script/rspec_file_by_file) boots bundler and rspec-core once, then runs each spec file in aforkof that parent:rspecprocess — the isolation guarantee this build part exists to check is preserved (verified: a missing-require failure and a deliberately failing spec file both still fail the run, and subsequent files don't run, matching today's fail-fast behavior).Gem.loaded_specsis populated and gems behave exactly as in production. No more standalone-only failure modes.ENVandBundler::ORIGINAL_ENV) after booting bundler, so children — and anything they shell out to viaBundler.with_original_env— observe the same environment a freshrspecprocess would. (The first draft missed this and this build part correctly caught it, via schema_definition'srake_tasks_spec; all 7 env-sensitive spec files in the repo now pass under the runner locally.)Timing
CI (the number that matters): this PR's
run_specs_file_by_filejob ran in 20m49s, vs the last four runs of the current approach on main: 23m17s, 25m55s, 26m18s, 26m29s (median ~26m). That's a ~5 minute / ~20% improvement, faster than every recent baseline run.Local measurements (M-series macOS, warm caches, ES 9.4.2,
NO_VCR=1):bundle execper fileThe local admin regression is a macOS-specific fork penalty on socket I/O (raw
Net::HTTPto local ES is ~40% slower in a forked child on macOS; pure-CPU and allocation-heavy benchmarks show zero fork penalty, and the same benchmark on Linux shows ~no penalty) — consistent with the CI job getting faster while local datastore-heavy gems get slower. Something to be aware of when running the file-by-file loop locally on a Mac, but CI is where this build part runs.