Add serial_default_lang option for parallel-unsafe plugins - #310
Open
rathboma wants to merge 2 commits into
Open
Add serial_default_lang option for parallel-unsafe plugins#310rathboma wants to merge 2 commits into
rathboma wants to merge 2 commits into
Conversation
When parallel_localization is on, polyglot forks one Ruby process per language. Each fork independently initializes jekyll-assets, whose Sprockets::Cache calls FileUtils.rm_r on the shared .jekyll-cache/assets/ directory on first use. Multiple forks racing on the same clear leaves all but one with `Errno::ENOENT @ apply2files` and the build fails. The new serial_default_lang option (default false) tells polyglot to process the default language synchronously in the parent before forking the rest. That lets jekyll-assets initialize and write its manifest exactly once; the language forks then inherit the populated @sprockets via fork(2) copy-on-write, so the destructive clear never runs in any fork. No behavior change unless the option is explicitly enabled.
Reframe the docs around parallel-unsafe plugins in general rather than the jekyll-assets/sprockets internals. Keeps jekyll-assets as the named example but drops the sprockets-specific traceback and implementation detail that won't age well.
untra
reviewed
Aug 2, 2026
Comment on lines
+13
to
+19
| # When true (and parallel_localization is also true), the default | ||
| # language is processed synchronously in the parent before any forks | ||
| # are spawned for the other languages. This makes parallel builds | ||
| # safe for plugins that do expensive one-time setup and share state | ||
| # across the site (e.g. jekyll-assets), which otherwise race when | ||
| # every fork runs that setup at once. See README for details. | ||
| @serial_default_lang = config.fetch('serial_default_lang', false) |
Owner
There was a problem hiding this comment.
something I've noticed and don't like with claude made code contributions, is it writes a lot of comments.
Code comments rot rapidly, and then become lies that downstream AI reads, and then comments that are misleading cause problems.
can you remove some of these extraneous excessive code comments in the site.rb and spec ? they stand out like a sore thumb. Otherwise this PR is good, and I'm inclined to merge it soon with the next release, after I can adjust some verbiage and make the upcoming blogpost.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds a new config option,
serial_default_lang(defaultfalse). When enabled alongsideparallel_localization: true, Polyglot processes the default language first, on its own, in the parent process — and only then forks for the remaining languages.Why
With
parallel_localization: true, Polyglot forks one process per language and runs them all at once. That breaks plugins that do expensive one-time setup and share state across the whole site.The case I hit is
jekyll-assets: each fork independently initializes its Sprockets cache, and on startup it clears the shared.jekyll-cache/assets/directory. When several forks race to clear the same directory at the same time, all but one die mid-traversal with:…and the whole build fails. The only workaround today is
parallel_localization: false, which means giving up parallelism entirely.With
serial_default_lang: true, the expensive setup happens once during the default-language pass in the parent. The language forks then inherit the finished state viafork(2)copy-on-write, so there's nothing left to race over. It isn't jekyll-assets specific — any plugin with the same "shared cache, cleared on init" shape benefits.Performance
Tested on a real site — the Beekeeper Studio website: ~770 markdown source files, 10 languages, jekyll-assets in the pipeline.
parallel_localization: false(serial — previously the only thing that worked here)parallel_localization: true+serial_default_lang: trueRoughly a 3.6x speedup, and more to the point it makes
parallel_localizationusable on this site at all. I diffed the parallel output against a serial build and confirmed it is content-identical: same file count, and every byte-level difference is ordinary build nondeterminism (timestamps, asset content-hashes, randomized DOM ids) that shows up between any two builds regardless of this change.The cost is one fewer concurrent fork — the default language no longer overlaps with the others. On a multi-language site that is a small fraction of what parallelism buys back.
Compatibility
Defaults to
false, so behavior is unchanged unless you explicitly opt in.Tests
bash test.sh(RuboCop + RSpec) passes locally.