RomM version
5.2.0 (recommendation text is unchanged on current master)
Describe the bug
Two places document WEB_SERVER_CONCURRENCY with a worker-count recommendation that makes
RomM slower and can exhaust memory on a typical self-hosted box:
env.template:133 - WEB_SERVER_CONCURRENCY=1 # Number of worker processes (recommended: 2 × CPU cores + 1)examples/docker-compose.example.yml:23 - WEB_SERVER_CONCURRENCY=3 # How many workers you want available for API calls and tasks we recommend 2 x CPU + 1
Three problems:
-
The heuristic is for the wrong worker class. 2 x CPU cores + 1 is gunicorn's
guidance for sync workers, where each worker serves exactly one request at a time and
you over-provision to hide blocking I/O. RomM runs
--worker-class uvicorn_worker.UvicornWorker with --worker-connections 1000
(docker/init_scripts/init:140-146), and its endpoints are plain def (for example
get_roms at backend/endpoints/roms/__init__.py:411), so Starlette already serves them
concurrently in its threadpool inside a single worker. The rule does not transfer.
-
"and tasks" is inaccurate. WEB_SERVER_CONCURRENCY appears exactly once in the
entrypoint, on the gunicorn API server (docker/init_scripts/init:141). Background tasks
run in separate RQ worker processes started by start_bin_rq_worker /
start_bin_rq_scan_worker (init:262-270), which this variable does not govern.
-
The two files disagree on the default (1 vs 3), and neither mentions the RAM cost.
To Reproduce
- Run RomM 5.2.0 on a host with N cores and a library where a single platform holds a few
thousand ROMs (I used 4,644 ROMs over 4 platforms).
- Set
WEB_SERVER_CONCURRENCY to 2 * N + 1 as the docs recommend.
- Load the gallery, which fires roughly 14 concurrent API calls.
- Compare against
WEB_SERVER_CONCURRENCY=4.
Expected behavior
Following the documented recommendation should improve throughput, or at least not hurt it,
and should not risk exhausting host memory.
Actual behavior
Measured with real gunicorn, 14 concurrent gallery requests, RomM 5.2.0, 4,644 ROMs over
4 platforms, 16-core / 7.4 GB host (so the documented value is 33), slowest TTFB in the
burst:
| Workers |
Slowest TTFB |
Peak DB connections |
Errors |
| 1 (current default) |
6.474 s |
5 |
0 |
| 2 |
3.515 s |
8 |
0 |
| 4 |
1.939 s |
14 |
0 |
| 8 |
2.007 s |
14 |
0 |
| 16 |
1.784 s |
16 |
0 |
33 (2 x cores + 1) |
5.767 s |
17 |
0 |
Returns flatten after about 4 workers. At 33, processes contend for 16 cores on CPU-bound
work and lose more than they gain: 3x slower than 4 workers, and barely better than the
default of 1.
Memory is the harder limit. 33 workers measured 5.31 GB PSS (~160 MB per worker, shared
pages counted once), leaving 587 MB free on a 7.4 GB host. On an 8-core NAS the
recommendation gives 17 workers, roughly 2.7 GB, competing with every other container.
Server (please complete the following information):
- OS: Ubuntu 24.04 (Docker host), 16 cores / 7.4 GB RAM
- Version: RomM 5.2.0, MariaDB 11.3.2
Additional context
There is also a connection-pool ceiling worth documenting, though I could not provoke it in
practice. Each worker process gets its own SQLAlchemy QueuePool with the library defaults
pool_size=5, max_overflow=10, never overridden at
backend/handler/database/base_handler.py:10-12, so 15 connections per process. At 33
workers that ceilings at 495 connections, plus 15 each for the RQ worker and RQ scan worker,
against MariaDB's default max_connections of 151. Pools only grow on demand, so realistic
page-load bursts peaked at just 14-17 connections even with 16 workers and produced no
errors, but the setting is over-subscribed on paper and would bite a busy multi-user
instance.
Suggested wording, replacing the current note in both files:
Number of API worker processes. Start at 4. Raise it only if you have several people
using the instance at once, and budget roughly 160 MB of RAM per worker. This does not
control background tasks, which run in their own RQ worker processes.
This came out of investigating #4382, where the default of 1 worker turned out to be the
reason a single gallery request measured 3-4 s.
AI disclosure: the measurements and analysis here were produced with Claude Code.
RomM version
5.2.0 (recommendation text is unchanged on current
master)Describe the bug
Two places document
WEB_SERVER_CONCURRENCYwith a worker-count recommendation that makesRomM slower and can exhaust memory on a typical self-hosted box:
env.template:133-WEB_SERVER_CONCURRENCY=1 # Number of worker processes (recommended: 2 × CPU cores + 1)examples/docker-compose.example.yml:23-WEB_SERVER_CONCURRENCY=3 # How many workers you want available for API calls and tasks we recommend 2 x CPU + 1Three problems:
The heuristic is for the wrong worker class.
2 x CPU cores + 1is gunicorn'sguidance for sync workers, where each worker serves exactly one request at a time and
you over-provision to hide blocking I/O. RomM runs
--worker-class uvicorn_worker.UvicornWorkerwith--worker-connections 1000(
docker/init_scripts/init:140-146), and its endpoints are plaindef(for exampleget_romsatbackend/endpoints/roms/__init__.py:411), so Starlette already serves themconcurrently in its threadpool inside a single worker. The rule does not transfer.
"and tasks" is inaccurate.
WEB_SERVER_CONCURRENCYappears exactly once in theentrypoint, on the gunicorn API server (
docker/init_scripts/init:141). Background tasksrun in separate RQ worker processes started by
start_bin_rq_worker/start_bin_rq_scan_worker(init:262-270), which this variable does not govern.The two files disagree on the default (1 vs 3), and neither mentions the RAM cost.
To Reproduce
thousand ROMs (I used 4,644 ROMs over 4 platforms).
WEB_SERVER_CONCURRENCYto2 * N + 1as the docs recommend.WEB_SERVER_CONCURRENCY=4.Expected behavior
Following the documented recommendation should improve throughput, or at least not hurt it,
and should not risk exhausting host memory.
Actual behavior
Measured with real gunicorn, 14 concurrent gallery requests, RomM 5.2.0, 4,644 ROMs over
4 platforms, 16-core / 7.4 GB host (so the documented value is 33), slowest TTFB in the
burst:
2 x cores + 1)Returns flatten after about 4 workers. At 33, processes contend for 16 cores on CPU-bound
work and lose more than they gain: 3x slower than 4 workers, and barely better than the
default of 1.
Memory is the harder limit. 33 workers measured 5.31 GB PSS (~160 MB per worker, shared
pages counted once), leaving 587 MB free on a 7.4 GB host. On an 8-core NAS the
recommendation gives 17 workers, roughly 2.7 GB, competing with every other container.
Server (please complete the following information):
Additional context
There is also a connection-pool ceiling worth documenting, though I could not provoke it in
practice. Each worker process gets its own SQLAlchemy
QueuePoolwith the library defaultspool_size=5, max_overflow=10, never overridden atbackend/handler/database/base_handler.py:10-12, so 15 connections per process. At 33workers that ceilings at 495 connections, plus 15 each for the RQ worker and RQ scan worker,
against MariaDB's default
max_connectionsof 151. Pools only grow on demand, so realisticpage-load bursts peaked at just 14-17 connections even with 16 workers and produced no
errors, but the setting is over-subscribed on paper and would bite a busy multi-user
instance.
Suggested wording, replacing the current note in both files:
This came out of investigating #4382, where the default of 1 worker turned out to be the
reason a single gallery request measured 3-4 s.