Skip to content

docs(env-vars): raise the SCAN_WORKERS default to 4 - #131

Merged
gantoine merged 2 commits into
rommapp:mainfrom
Spinnich:docs/scan-workers-recommendation
Sep 8, 2026
Merged

docs(env-vars): raise the SCAN_WORKERS default to 4#131
gantoine merged 2 commits into
rommapp:mainfrom
Spinnich:docs/scan-workers-recommendation

Conversation

@Spinnich

@Spinnich Spinnich commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Description

Updates the SCAN_WORKERS row in the environment-variables reference, which currently shows a default of 1. Hashing a real 1.96 GiB library measured 2.2x faster at 4 concurrent workers than at 1, and a single worker is a large part of why first scans feel slow.

Live page: https://docs.romm.app/latest/reference/environment-variables/

Before After
Default 1 4
Description How many ROMs a scan processes at once How many ROMs a scan processes at once; mainly speeds up hashing, as metadata providers pace themselves

Why

SCAN_WORKERS is an asyncio.Semaphore around per-ROM identification. Inside it, hashing is offloaded with asyncio.to_thread and hashlib/zlib release the GIL, so more workers genuinely use more cores. 67 real ROM files, 1.96 GiB, warm page cache, core counts below 16 pinned with taskset:

Cores 1 worker 2 workers 4 workers 8 workers
16 108 MiB/s 197 (1.8x) 240 (2.2x) 154 (1.4x)
4 108 MiB/s 204 (1.9x) 264 (2.4x) 226 (2.1x)
2 110 MiB/s 176 (1.6x) 173 (1.6x) -

4 is the peak at both 16 and 4 cores and still 1.6x at 2 cores, so it does not assume a large host.

The added half-sentence is the part users most need: raising this mainly speeds up hashing. Metadata providers cap themselves independently (IGDB 4/s, RetroAchievements 4/s, MobyGames 1/s, Steam 0.6/s, and ScreenScraper via a concurrency limiter set to the account's advertised thread count), so extra workers queue there rather than scraping faster.

It deliberately does not name a ceiling. The practical limit today is FILE_READ_CHUNK_SIZE = 8 KiB, which rommapp/romm#4387 raises to 256 KiB; past 4 workers the 8 KiB chunk turns added concurrency into GIL contention, and at 256 KiB throughput instead scales to 666 MiB/s at 8 workers and stays flat out to 20. Wording the row around what the setting buys rather than where it stops keeps it correct before and after that change.

About editing a generated file

env-vars.md carries the AUTOGENERATED by scripts/gen_env_vars.py: do not edit header, so this needs a word of explanation.

The real fix is rommapp/romm#4386, which changes the env.template line this table is generated from. That is the source of truth and should be merged first. But scripts/sources.toml pins ref = "5.2.0", so the published table keeps showing 1 until the next release triggers romm-release-bump.yml. This PR closes that window.

To make it safe, the edit is not free-hand. I ran scripts.gen_env_vars against the env.template from rommapp/romm#4386, formatted the result, and the Scans & Tasks section is byte-identical to what is committed here, so the next bump reproduces this exact content rather than fighting it:

$ diff <(awk '/^### Scans & Tasks/,/^### Sync/' generated-check.md) \
       <(awk '/^### Scans & Tasks/,/^### Sync/' docs/resources/snippets/env-vars.md)
IDENTICAL for the Scans & Tasks section

I confirmed the two versions of that section differ only in the SCAN_WORKERS row, so nothing else in it drifts in from unreleased master. The rest of the diff is only Prettier re-padding the table to the new column width. No other section is touched, and sources.toml is deliberately left at 5.2.0 (re-pinning to master would pull in env vars that are not in a release yet).

If you would rather not carry a manual edit at all, feel free to close this and let the bump handle it after rommapp/romm#4386 merges. The fix is complete without this PR, just delayed by one release.

Verification

  • uv run mkdocs build --strict passes
  • uv run python -m scripts.check_redirects passes (all 46 targets exist)
  • trunk fmt / trunk check clean
  • Confirmed the built site/reference/environment-variables/index.html renders 4 and the new description

Related: rommapp/romm#4386 (the source-of-truth change), rommapp/romm#4387 (the read-chunk fix that moves the practical ceiling)

AI disclosure

The benchmarking, analysis and wording here were produced with Claude Code. I reviewed and verified every claim against the source before submitting.

🤖 Generated with Claude Code

The reference lists SCAN_WORKERS as 1, which measured 2.2x slower than 4
concurrent workers when hashing a real library. rommapp/romm#4386 changes
the env.template line this table is generated from.

Regenerated from that branch rather than hand-written: the Scans & Tasks
section here is byte-identical to the generator's output, so the next
release bump reproduces it. sources.toml stays pinned at 5.2.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gantoine
gantoine merged commit 595bb79 into rommapp:main Sep 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants