Skip to content

feat(install): apply the Workerman kernel tuning on install and --update - #602

Merged
detain merged 1 commit into
masterfrom
feat/kernel-tuning-install
Aug 1, 2026
Merged

feat(install): apply the Workerman kernel tuning on install and --update#602
detain merged 1 commit into
masterfrom
feat/kernel-tuning-install

Conversation

@detain

@detain detain commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Implements the applicable parts of Workerman's kernel-optimization appendix in scripts/install.sh, on both the fresh-install path and --update, so an install that predates this picks the tuning up on the next update.

The finding

On the live box both phlix daemons were running with a soft RLIMIT_NOFILE of 1024:

$ grep 'Max open files' /proc/$(systemctl show -p MainPID --value phlix-server)/limits
Max open files            1024                 524288               files

The appendix prescribes /etc/security/limits.conf for this — but systemd does not read limits.conf at all. The daemon's ceiling comes from LimitNOFILE= in the unit, which was unset, so the service inherited systemd's soft default of 1024. Workerman holds one fd per live connection, and an HTTP worker serving HLS also holds the open segment files, so 1024 is reachable under load and surfaces as "Too many open files" with clients dropped mid-stream.

Applied

setting was now
LimitNOFILE= (systemd unit) unset → soft 1024 1048576
net.core.somaxconn 4096 65535
net.ipv4.tcp_max_syn_backlog 4096 262144
net.core.netdev_max_backlog 1000 30000
net.ipv4.ip_local_port_range 32768 60999 (28,232 ports) 16384 65535 (49,152 ports)
PAM nofile (root + service user) distro default soft 65536 / hard 1048576

somaxconn was the other silent one — it clamps every listen() backlog, visible as Send-Q 4096 on all phlix listen sockets in ss -ltn.

The port range is a widening (+74%); the floor stops at 16384 rather than the page's 10240 so the registered-port band — which contains every port Phlix listens on (2206, 8096, 8097, 8800, 8802-8805) — stays outside the ephemeral range, making an ip_local_reserved_ports entry unnecessary.

Deliberately NOT applied

Each with the reason recorded inline in the generated sysctl file, so a future reader doesn't "restore" them:

  • fs.file-max=6815744 — live default is 9223372036854775807. Writing the page's value is a ~10¹²× downgrade.
  • tcp_max_tw_buckets=20000 — live default is 262144, and this is a cap: once exceeded the kernel destroys TIME_WAIT sockets early and logs an overflow. 13× downgrade.
  • tcp_tw_recycle=0 — removed from Linux in 4.12 (2017). The key doesn't exist; listing it makes sysctl --system exit non-zero.
  • nf_conntrack_max=2621440 — only meaningful behind a stateful firewall (ufw is inactive; nf_conntrack_count is 107 against a max of 262144). Pins ~1 GB of unswappable kernel memory and needs a matching nf_conntrack_buckets bump or it just lengthens the hash chains.

Design notes

  • The sysctl/limits drop-ins are owned outright and rewritten wholesale rather than editing the shared /etc/sysctl.conf, which makes re-runs idempotent. --uninstall removes them.
  • The --update LimitNOFILE retrofit fires only when the directive is absent, so an operator who tuned it by hand keeps their value.
  • Containers get a read-only/namespaced /proc/sys; the function detects that and skips with a warning rather than failing the install.
  • The PAM drop-in's account is passed to the function as an argument rather than by reassigning $SERVICE_USERdo_update's chown steps on either side of the call site read that global, and moving it would silently re-target them.

Verification

The retrofit sed was extracted verbatim from this script (so the test can't drift from shipped code) and run against the real live unit files in three distinct shapes — has LimitMEMLOCK, lacks LimitMEMLOCK, already patched:

=== phlix-server / live-server.service  [A: has LimitMEMLOCK] ===
    PASS one LimitNOFILE=1048576 / one ExecStart= / inside [Service]
    PASS diff has ZERO deleted lines      PASS every added line is ours
    PASS idempotent on re-run             PASS systemd-analyze clean
    PASS systemd parses LimitNOFILE (1048576)
=== phlix-server / old-server.service   [B: no LimitMEMLOCK]  === all PASS
=== phlix-hub    / live-hub.service     [A: has LimitMEMLOCK] === all PASS

Also verified: all four sysctl keys exist on the target kernel (so sysctl -p cannot fail on them); the generated limits.conf is well-formed (4 fields/line); the user argument is honoured with no global leak; bash -n clean; and the --uninstall loop is safe under set -euo pipefail in all three file-presence cases.

Paired with detain/phlix-hub#(same change).

🤖 Generated with Claude Code

Implements the applicable parts of Workerman's kernel-optimization appendix
(https://www.workerman.net/doc/workerman/appendices/kernel-optimization.html)
in scripts/install.sh, on both the fresh-install path and `--update`, so an
install that predates this picks the tuning up on the next update.

The finding that motivated this: on the live box both phlix daemons were
running with a soft RLIMIT_NOFILE of 1024. The appendix prescribes
/etc/security/limits.conf for this, but systemd does not read limits.conf at
all — the daemon's ceiling comes from LimitNOFILE= in the unit, which was
unset, so the service inherited systemd's soft default of 1024. Workerman
holds one fd per live connection, and an HTTP worker serving HLS also holds
the open segment files, so 1024 is reachable under load and surfaces as
"Too many open files" with clients dropped mid-stream.

Applied:
  - LimitNOFILE=1048576 in the unit template, plus an idempotent `--update`
    retrofit that only fires when the directive is absent (an operator who
    tuned it by hand keeps their value). Anchored after LimitMEMLOCK= when
    present, else inserted before ExecStart=.
  - /etc/sysctl.d/99-phlix-net.conf: somaxconn 4096 -> 65535 (this value
    silently clamps every listen() backlog), tcp_max_syn_backlog 4096 ->
    262144, netdev_max_backlog 1000 -> 30000, ip_local_port_range
    32768-60999 -> 16384-65535.
  - /etc/security/limits.d/99-phlix.conf: nofile soft 65536 / hard 1048576
    for root and the service user. PAM-only, so this covers login shells and
    hand-run CLI scripts (scripts/*.php) but explicitly NOT the daemon.

Deliberately NOT applied, with the reason recorded inline in the generated
sysctl file so a future reader does not "restore" them:
  - fs.file-max=6815744 — the live default is 9223372036854775807; writing
    the page's value is a ~10^12x downgrade.
  - tcp_max_tw_buckets=20000 — the live default is 262144, and this is a cap:
    once exceeded the kernel kills TIME_WAIT sockets early and logs an
    overflow. 13x downgrade.
  - tcp_tw_recycle=0 — removed from Linux in 4.12; the key does not exist and
    listing it makes `sysctl --system` exit non-zero.
  - nf_conntrack_max=2621440 — only meaningful behind a stateful firewall
    (ufw is inactive here, conntrack_count is 107 against a max of 262144),
    pins ~1 GB of unswappable kernel memory, and needs a matching
    nf_conntrack_buckets bump or it just lengthens the hash chains.

The sysctl/limits drop-ins are owned outright and rewritten wholesale rather
than editing the shared /etc/sysctl.conf, which makes re-runs idempotent;
--uninstall removes them. Containers get a read-only or namespaced /proc/sys,
so the function detects that and skips with a warning instead of failing the
install. The PAM drop-in's account is passed to the function as an ARGUMENT
rather than by reassigning $SERVICE_USER, because do_update's chown steps on
either side of the call site also read that global.

Verified: the retrofit sed was extracted verbatim from this script and run
against the real live unit files in three shapes (has LimitMEMLOCK, lacks
LimitMEMLOCK, already patched). All assert one LimitNOFILE line, correct
[Service] placement, a purely additive diff (zero deleted lines), idempotency
on re-run, and a clean `systemd-analyze verify`. All four sysctl keys were
confirmed to exist on the target kernel, so `sysctl -p` cannot fail on them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@codacy-production

codacy-production Bot commented Aug 1, 2026

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Coverage ∅ diff coverage · -0.05% coverage variation

Metric Results
Coverage variation -0.05% coverage variation (-1.00%)
Diff coverage diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (d620c87) 63701 42022 65.97%
Head commit (43d55e4) 63701 (+0) 41991 (-31) 65.92% (-0.05%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#602) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@detain
detain merged commit 9405471 into master Aug 1, 2026
17 checks passed
@detain
detain deleted the feat/kernel-tuning-install branch August 1, 2026 05:02
@codecov

codecov Bot commented Aug 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.76%. Comparing base (d620c87) to head (43d55e4).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #602      +/-   ##
============================================
- Coverage     66.80%   66.76%   -0.05%     
  Complexity    21657    21657              
============================================
  Files           668      668              
  Lines         67864    67864              
============================================
- Hits          45339    45307      -32     
- Misses        22525    22557      +32     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant