Merge dev into master: #78 whitelist arc + single-instance lock (#433) + port-in-use fix (#434)

.github/workflows/smoke.yml

@@ -157,6 +157,9 @@ jobs:
       - name: Run etc_blocklist unit test
         run: lua5.4 tests/unit/etc_blocklist_test.lua
 
+      - name: Run etc_whitelist unit test
+        run: lua5.4 tests/unit/etc_whitelist_test.lua
+
       - name: Run cmd_usercleaner orphan unit test
         run: lua5.4 tests/unit/cmd_usercleaner_orphan_test.lua
 
@@ -217,6 +220,16 @@ jobs:
       - name: Run blocklist unit test
         run: lua5.4 tests/unit/blocklist_test.lua
 
+      # #78 allowlist: core/whitelist.lua global IP/CIDR allowlist.
+      # 63 checks: add/remove/list/count, v4+v6+CIDR match, v4-mapped
+      # v6 lookup, expires_at, reload-from-disk, disabled-engine,
+      # empty-store fast path, plus a blocklist integration block
+      # proving Model-A precedence (whitelist overrides an automated
+      # block, a manual pin still wins) - the integration checks
+      # provably fail on a blocklist without the whitelist hook.
+      - name: Run whitelist unit test
+        run: lua5.4 tests/unit/whitelist_test.lua
+
       # #78 Phase D1: core/mmdb.lua pure-Lua MaxMind DB reader. 75
       # checks against MaxMind's canonical test databases (all three
       # record sizes 24/28/32, both IP families, v4-in-v6 embedding,
@@ -421,6 +434,10 @@ jobs:
         shell: msys2 {0}
         run: lua5.4 tests/unit/etc_blocklist_test.lua
 
+      - name: Run etc_whitelist unit test
+        shell: msys2 {0}
+        run: lua5.4 tests/unit/etc_whitelist_test.lua
+
       - name: Run cmd_usercleaner orphan unit test
         shell: msys2 {0}
         run: lua5.4 tests/unit/cmd_usercleaner_orphan_test.lua
@@ -457,6 +474,10 @@ jobs:
         shell: msys2 {0}
         run: lua5.4 tests/unit/blocklist_test.lua
 
+      - name: Run whitelist unit test
+        shell: msys2 {0}
+        run: lua5.4 tests/unit/whitelist_test.lua
+
       # #78 Phase D1: pure-Lua MaxMind DB reader (see Linux job note).
       - name: Run mmdb unit test
         shell: msys2 {0}

.gitignore

@@ -36,4 +36,6 @@ desktop.ini
 # tree. This entry is a safety net for older binaries or unusual
 # invocation paths that could still drop the file at the repo root.
 exception.txt
+# Single-instance lock file, created at runtime in the install tree.
+luadch.lock
 __pycache__/

CHANGELOG.md

@@ -65,6 +65,10 @@ land on `release/3.1.x` per
 
 ### Bugfixes
 
+- **clear "port already in use" message + Windows port-hijack fix** (Sopor follow-up). When a listener could not bind its port the hub was unhelpful: on Linux it logged only the cryptic luasocket line (`...bind: address already in use`) then silently retried every 30s; on Windows the pre-bind `SO_REUSEADDR` (added for the Linux #128 TIME_WAIT fast-restart) has `SO_REUSEPORT` semantics, so a second process re-bound the SAME port with no error at all - two hubs then silently shared it, and any local process could hijack a luadch port. Two changes: (1) `core/server.lua` sets `SO_REUSEADDR` only on Linux (detected via `util.path_sep()`); on Windows it is skipped both pre- and post-bind, so a same-port second bind now fails visibly with `WSAEADDRINUSE` and the port can no longer be hijacked. Linux keeps the #128 fast-restart behaviour unchanged. (2) `core/hub.lua` `add_server_handler` now logs a clear operator line (`server: IPv4 TLS port 5001 is already in use - another process (or a second hub) holds it. Stop it or change the port in cfg/cfg.tbl; retrying every 30s.`) plus a generic message for any other listener-start failure that was previously swallowed. Verified that dropping `SO_REUSEADDR` on Windows does NOT regress the fast-restart: the full Windows smoke restart battery (plaintext / tier / `+reload` / dual-stack-same-port #107) stays green. Smoke: a new stage starts a second hub (separate dir, same ports as the live first hub) and asserts the guidance appears, provably failing pre-fix (§1a.7 - the assertion is on the new `change the port in cfg/cfg.tbl` phrase that the pre-fix cryptic line lacks; pre-fix Windows silently re-binds and logs nothing). Complements the single-instance lock (same Sopor thread): the lock stops a second copy in the SAME dir; this makes a second hub in a DIFFERENT dir with clashing ports fail loudly instead of silently. 3.2.x only.
+
+- **single-instance lock: a second hub started against the same install tree now refuses to start** (Sopor). Nothing stopped two `Luadch.exe` (or `./luadch`) processes running from one install directory at once - on Windows especially, where `core/server.lua` sets `SO_REUSEADDR` before bind (a deliberate fast-restart fix, #128) and Windows gives `SO_REUSEADDR` `SO_REUSEPORT` semantics, so the second process re-binds the same port and both hubs then share (and race on) `cfg/user.tbl`, `master.key`, the plugin `scripts/data/*.tbl` stores and the logs; interleaved `saveusers()`/`savetable()` writes can corrupt them (data loss). Fixed in the C launcher (`hub/hub.c`) with a per-install lock file (`luadch.lock` in the install dir): unix `flock(LOCK_EX|LOCK_NB)` + `O_CLOEXEC`, Windows exclusive `CreateFile` (deny-share, non-inheritable handle, delete-on-close). Held open for the whole process lifetime and released by the OS on crash (no stale-lock cleanup step), retained by the daemon child across the `-d` fork and across `+reload` (the fd survives the `restart()`/atexit re-entry). Scope is per install directory - a test hub and a prod hub in separate directories on one box still run in parallel; only a second copy of the SAME install is refused (non-zero exit + "another instance is already running" to stderr and `log/exception.txt`). Fail-open: only the definitive already-held signal (unix `EWOULDBLOCK`, Windows `ERROR_SHARING_VIOLATION`) blocks boot; an un-creatable lock file (read-only fs and the like) warns and continues. Deployment note: a supervisor that starts the replacement before the old process has finished its shutdown drain sees the refusal - stop must complete before start (systemd / `docker restart` are sequential and unaffected). Smoke: a new stage starts a second hub against the live staging tree and asserts it exits non-zero with the refusal while the first stays up, provably failing pre-fix (§1a.7 - on the unpatched Windows binary the second instance boots and runs concurrently). Two-pass review (§1a.6) caught the missing `O_CLOEXEC` (a spawned child could otherwise inherit the fd and brick the next restart) and the `ERROR_ACCESS_DENIED`-is-not-a-live-instance classification. 3.2.x only, not backported (hardening, not a security/crash fix; per §8 the 3.1.x bar is high). Operator guide: [`docs/BUILDING.md`](docs/BUILDING.md).
+
 - **`usr_hide_share`: also hide the shared-file count, not just the share size.** The plugin zeroed only the ADC INF `SS` (share size) for a hidden user, never `SF` (shared-file count), so clients still displayed how many files the user shared. Fixed by mirroring every `SS`-zeroing write with an `SF`-zeroing one at all three sites (the `onInf`/`onConnect`/`onStart`/`onExit` listener and the manual `+hideshare` command) and adding `SF0` to the `BINF` broadcast. v0.4 -> v0.5. Intended side effect (a consistency fix): a hidden user now also contributes 0 to the hub's aggregate file counts (PING reply to hublist scrapers, Prometheus `/metrics`, HTTP stats), matching the pre-existing behaviour for aggregate share size; no new kick path (`usr_share` min-share enforcement keys on `SS` only). §1a.7: diff-is-self-evidence (the unpatched plugin never touched `SF`; the fix is a 1:1 mirror of the already-working `SS` path) plus testhub validation. 3.2.x; not backported (self-contained plugin, a 3.1.x drop-in is possible on request).
 
 - [#419](https://github.com/luadch-ng/luadch/issues/419) (Kcchouette) - **the ADC parser now discards messages containing unknown escape sequences** (ADC 1.0 section 3.1: only `\s` / `\n` / `\\` are defined; "any message containing unknown escapes must be discarded"). `core/adc.lua parse()` validated UTF-8, structure and field values but never escapes, so a message carrying `\q` (or a trailing/unescaped backslash) was accepted and relayed - other clients then interpret the unknown escape differently (undefined per spec), a cross-client display-impersonation vector (`admin\q` may render as `admin` on a client that drops it; complements the #315 `unescape` hardening, which kept the receiving-side lenient - this closes the relay side). Validated **pairwise** (strip the valid `\s`/`\n`/`\\` first; any leftover backslash is then unknown): a naive "`\` not followed by s/n/`\`" scan is wrong - it false-positives on `\\q` (an escaped backslash + literal `q`, i.e. the valid wire form of the text `\q`) and misses a lone trailing backslash. A backslash-free message (most protocol commands) skips the gsub via a single plain `find`; a message with escapes (e.g. any multi-word chat, which carries `\s`) takes one bounded single-pass gsub - both cheap. Compliant clients never emit unknown escapes and are unaffected (and a plugin that hand-builds a raw message body should escape backslashes as `\\` - the `hub.broadcast` / `user:reply` / `cmd:adcstring` helpers already do); a drop is logged to `event.log`. Smoke: 3 checks on both legs - `\q` discarded + trailing backslash discarded (both provably fail pre-fix, §1a.7) + a positive control that a valid `\\` is NOT discarded (guards the false-positive). Spec-compliance, no hub-crash / no remote-DoS: 3.2.x only, not backported.
@@ -140,6 +144,14 @@ land on `release/3.1.x` per
 
 ### Features
 
+- **#78 allowlist, Phase D: HTTP API for the whitelist (`etc_whitelist` v0.02).** Four endpoints mirroring the blocklist HTTP API: `GET /v1/whitelist` (read, filter/sort/paginate via `http_filter`), `GET /v1/whitelist/counts` (read), `POST /v1/whitelist` (admin, body `{cidr, source?, reason?, expires_at?}`, source enum `{manual, pinger}`, host-bits-set CIDRs rejected), `DELETE /v1/whitelist/{id}` (admin). HTTP admin tokens bypass the ADC hierarchy guard (the token is the trust surface; entries attributed `by_nick = <token label>` / `by_level = 100`). `etc_whitelist_test` gains 32 HTTP checks (registration + scope + schema + all four handlers, happy + error paths); a seed-aware `test_http_phase_d_whitelist` smoke does a real token-authed POST -> GET -> DELETE roundtrip (baseline count captured so it stays robust to the pinger seed). Docs: `SCRIPTS.md` (new `etc_whitelist` section) + `BLOCKLIST.md` (allowlist layer). Completes the 4-PR #78 allowlist arc. 3.2.x only.
+
+- **#78 allowlist, Phase C: the IP-blocking plugins now consult the whitelist first.** `etc_geoip` (country/ASN kick), `etc_proxydetect` (proxy kick + the paid provider query) and `usr_hubs` (invalid-hubcount kick + hub-limit ban) each gained a one-line `if whitelist.is_whitelisted(ip) then return end` guard at the earliest point of their per-connection decision, so a whitelisted IP (hublist pinger / trusted infra) is exempt from these three - and for `etc_proxydetect` the guard sits BEFORE the cache / quota / HTTP so a trusted IP never costs a provider query. This silences the recurring `PROXYDETECT` / `GEOIP` / `USER HUBS` log lines for whitelisted IPs. Scope: this covers the IP-reputation blockers (plus the unified blocklist store via Phase A) and the hub-limit ban; the share / slots / nick-policy plugins (`usr_share` / `usr_slots` / `usr_nick_*`) are NOT yet whitelist-aware (a whitelisted IP still faces those), a possible follow-up. `etc_geoip_test` and `etc_proxydetect_test` gain whitelist-exemption regressions that provably fail without the guard (§1a.7); `usr_hubs` has no unit harness so its guard is the identical one-line mirror (diff-is-self-evidence). 3.2.x only.
+
+- **#78 allowlist, Phase B: the `+whitelist` admin plugin + bundled hublist-pinger seed (`etc_whitelist.lua`).** Operator-facing chat command over the Phase A engine: `+whitelist show [source] | add <cidr|ip> [reason="..."] [expires=YYYY-MM-DD] | del <id> | count | export | import <path>`. Mirrors `etc_blocklist`'s structure (verb dispatch, `by_level` hierarchy guard on del, JSONL export/import with control-byte stripping + a master-only `etc_whitelist_import_min_level`, opchat report + audit fire-sites `whitelist.add`/`.remove`/`.export`/`.import`) minus stealth/HTTP. **Bundled seed:** on the FIRST run (store `.tbl` missing) the plugin seeds a small set of known hublist-pinger IPs (source=`pinger`, v6 as `/64`, v4 as `/32`) so the pingers are exempt from the automated blockers out of the box; seed-on-missing only (operator edits/deletions are never re-seeded), disable with `etc_whitelist_seed = false`. Ships enabled in `examples/cfg/cfg.tbl`. cfg keys `etc_whitelist_oplevel` / `_show_limit` / `_seed` / `_report[_hubbot|_opchat]` / `_llevel` / `_import_min_level`; lang en + de. `tests/unit/etc_whitelist_test.lua` (53 checks: parse / add / del-hierarchy / sanitize / format / seed-on-missing / export-import) + a `+whitelist` roundtrip smoke that also validates the seed lands on first boot; both registered on both smoke.yml legs. 3.2.x only.
+
+- **#78 allowlist, Phase A: a global IP/CIDR whitelist engine (`core/whitelist.lua`).** The allowlist deferred from the unified-blocklist arc. A new core module - a stripped-down sibling of `core/blocklist.lua` (same bucketed radix cache, hex-encoded `.tbl` store, v4-mapped-v6 lookup; no source-priority / stealth / rollup) - exposes `whitelist.is_whitelisted(ip)` as a sandbox global so every IP-blocking path can consult it. `blocklist.check_ip` now applies **Model-A precedence**: a whitelisted IP overrides an AUTOMATED block (GeoIP / proxydetect / external feed) but NOT a deliberate manual pin (`source = "manual"` wins), so an operator can still block a specific IP that sits inside a whitelisted range. Full IPv4/IPv6 + CIDR support (a v6 `/64` covers a rotating pinger host); `expires_at` TTL; survives `+reload` via a cfg-reload listener. cfg keys `whitelist_enabled` (default true) + `whitelist_store_path` (`scripts/data/etc_whitelist.tbl`); the engine ships enabled with an empty store (zero overhead via a single `next()` fast path until an entry is added). `tests/unit/whitelist_test.lua` (63 checks incl. a blocklist-integration block whose precedence assertions provably fail on a blocklist without the hook, §1a.7); registered on both smoke.yml legs. Phase A of a 4-PR feature (B: `+whitelist` admin plugin + bundled hublist-pinger seed; C: per-plugin guards in etc_geoip / etc_proxydetect / usr_hubs; D: HTTP `/v1/whitelist`). 3.2.x only.
+
 - [#420](https://github.com/luadch-ng/luadch/issues/420) (#398 follow-up) - **`etc_webhook`: per-endpoint `conditions` filter on a decoded body field** (v0.02 -> v0.03). The event filter only matched the event *header*, but two real cases need a *body* field: a GitHub `release` webhook fires for every action (`created` / `edited` / `published` / `released`) under one `x-github-event: release`, and a Discourse new topic fires both `topic_created` AND `post_created` (its auto opening post) - so a release announced 5x and a new topic twice. New optional `conditions = { { path = "dotted.path", equals = X }, ... }` (or `not_equals`); ALL must hold to announce, evaluated against the decoded body via the existing `resolve_path` extraction (no new parsing surface). Values compare as strings (a number or string works); an unresolved path is `nil`, so `not_equals` passes when the field is absent (a `topic_created` carries no `post.post_number`, so the topic still announces while its opening post is dropped). Conditions apply endpoint-wide. The example config now filters GitHub to `action = "released"` (one clean line + release URL) and drops the Discourse opening post via `post.post_number != 1` (real replies still announce). `tests/unit/etc_webhook_test.lua` gains 6 checks, 3 of which provably fail pre-fix (§1a.7 - the unpatched plugin announces the opening post, a float `1.0` opening post, and `action=created`; the other 3 are positive controls that announce on both). Numbers compare numerically (a config `1` matches JSON `1` or `1.0`), otherwise as strings. Docs: `docs/WEBHOOKS.md` + `examples/cfg/webhooks.tbl`. 3.2.x only.
 
 - [#398](https://github.com/luadch-ng/luadch/issues/398) - **`etc_webhook.lua`: a generic INBOUND webhook receiver.** An external service (a Discourse forum, GitHub, GitLab, CI, monitoring, ...) POSTs an HMAC-SHA256-signed JSON body to a hub HTTP endpoint; the hub verifies the signature over the raw body (`adclib.constant_time_eq`), applies an event filter + delivery-id dedup, renders a `{dotted.path}` template and announces the event in chat as a named bot. The inbound-push mirror of `etc_status_push` (outbound heartbeat) and the complement of `etc_prometheus` (pull `/metrics`). **Multi-endpoint:** the operator-edited `cfg/webhooks.tbl` holds an array of endpoints, each with its own path / signature+event+id headers / event filter / bot nick / min-level / templates, so Discourse and GitHub can be received in parallel; `cfg.tbl` carries only the master switch `etc_webhook_activate` (keeps it lean). Per-endpoint secrets resolve env-var-first (`LUADCH_ETC_WEBHOOK_<NAME>_SECRET`), else the `etc_webhook_<name>_secret` cfg key, else an inline `secret` in `cfg/webhooks.tbl`; an endpoint with no resolvable secret is skipped (never runs unsigned). First plugin to register a **`scope="none"`** HTTP route - the router's bearer-token gate is skipped and the handler does its OWN HMAC auth; a missing/wrong signature is 401. Untrusted template fields are control-byte-stripped + length-capped, a per-minute flood cap bounds chat spam, and duplicate deliveries are deduped by the delivery-id header. Built on `core/hmac.lua` (the #398 precursor). Off by default (`etc_webhook_activate=false`; whitelisted `enabled=false` - advanced, needs `cfg/webhooks.tbl` + a reverse proxy). Unit-tested with the REAL hmac module (`tests/unit/etc_webhook_test.lua`, both smoke legs, 23 checks: HMAC pass/wrong/missing, `sha256=` prefix, event filter, dedup, template `{dotted.path}`/`{event}`/missing-path, control-byte strip + truncation, flood cap, min_level gating, no-secret-skip, activate=false inert). Operator guide `docs/WEBHOOKS.md`; security model in `docs/SECURITY.md`. 3.2.x only.