diff --git a/.github/workflows/smoke.yml b/.github/workflows/smoke.yml index 617f8c5d..7b846364 100644 --- a/.github/workflows/smoke.yml +++ b/.github/workflows/smoke.yml @@ -157,6 +157,9 @@ jobs: - name: Run etc_blocklist unit test run: lua5.4 tests/unit/etc_blocklist_test.lua + - name: Run etc_whitelist unit test + run: lua5.4 tests/unit/etc_whitelist_test.lua + - name: Run cmd_usercleaner orphan unit test run: lua5.4 tests/unit/cmd_usercleaner_orphan_test.lua @@ -217,6 +220,16 @@ jobs: - name: Run blocklist unit test run: lua5.4 tests/unit/blocklist_test.lua + # #78 allowlist: core/whitelist.lua global IP/CIDR allowlist. + # 63 checks: add/remove/list/count, v4+v6+CIDR match, v4-mapped + # v6 lookup, expires_at, reload-from-disk, disabled-engine, + # empty-store fast path, plus a blocklist integration block + # proving Model-A precedence (whitelist overrides an automated + # block, a manual pin still wins) - the integration checks + # provably fail on a blocklist without the whitelist hook. + - name: Run whitelist unit test + run: lua5.4 tests/unit/whitelist_test.lua + # #78 Phase D1: core/mmdb.lua pure-Lua MaxMind DB reader. 75 # checks against MaxMind's canonical test databases (all three # record sizes 24/28/32, both IP families, v4-in-v6 embedding, @@ -421,6 +434,10 @@ jobs: shell: msys2 {0} run: lua5.4 tests/unit/etc_blocklist_test.lua + - name: Run etc_whitelist unit test + shell: msys2 {0} + run: lua5.4 tests/unit/etc_whitelist_test.lua + - name: Run cmd_usercleaner orphan unit test shell: msys2 {0} run: lua5.4 tests/unit/cmd_usercleaner_orphan_test.lua @@ -457,6 +474,10 @@ jobs: shell: msys2 {0} run: lua5.4 tests/unit/blocklist_test.lua + - name: Run whitelist unit test + shell: msys2 {0} + run: lua5.4 tests/unit/whitelist_test.lua + # #78 Phase D1: pure-Lua MaxMind DB reader (see Linux job note). - name: Run mmdb unit test shell: msys2 {0} diff --git a/.gitignore b/.gitignore index 79df0ad0..7e6dbf7b 100644 --- a/.gitignore +++ b/.gitignore @@ -36,4 +36,6 @@ desktop.ini # tree. This entry is a safety net for older binaries or unusual # invocation paths that could still drop the file at the repo root. exception.txt +# Single-instance lock file, created at runtime in the install tree. +luadch.lock __pycache__/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 43c3d54f..3665fabf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -65,6 +65,10 @@ land on `release/3.1.x` per ### Bugfixes +- **clear "port already in use" message + Windows port-hijack fix** (Sopor follow-up). When a listener could not bind its port the hub was unhelpful: on Linux it logged only the cryptic luasocket line (`...bind: address already in use`) then silently retried every 30s; on Windows the pre-bind `SO_REUSEADDR` (added for the Linux #128 TIME_WAIT fast-restart) has `SO_REUSEPORT` semantics, so a second process re-bound the SAME port with no error at all - two hubs then silently shared it, and any local process could hijack a luadch port. Two changes: (1) `core/server.lua` sets `SO_REUSEADDR` only on Linux (detected via `util.path_sep()`); on Windows it is skipped both pre- and post-bind, so a same-port second bind now fails visibly with `WSAEADDRINUSE` and the port can no longer be hijacked. Linux keeps the #128 fast-restart behaviour unchanged. (2) `core/hub.lua` `add_server_handler` now logs a clear operator line (`server: IPv4 TLS port 5001 is already in use - another process (or a second hub) holds it. Stop it or change the port in cfg/cfg.tbl; retrying every 30s.`) plus a generic message for any other listener-start failure that was previously swallowed. Verified that dropping `SO_REUSEADDR` on Windows does NOT regress the fast-restart: the full Windows smoke restart battery (plaintext / tier / `+reload` / dual-stack-same-port #107) stays green. Smoke: a new stage starts a second hub (separate dir, same ports as the live first hub) and asserts the guidance appears, provably failing pre-fix (§1a.7 - the assertion is on the new `change the port in cfg/cfg.tbl` phrase that the pre-fix cryptic line lacks; pre-fix Windows silently re-binds and logs nothing). Complements the single-instance lock (same Sopor thread): the lock stops a second copy in the SAME dir; this makes a second hub in a DIFFERENT dir with clashing ports fail loudly instead of silently. 3.2.x only. + +- **single-instance lock: a second hub started against the same install tree now refuses to start** (Sopor). Nothing stopped two `Luadch.exe` (or `./luadch`) processes running from one install directory at once - on Windows especially, where `core/server.lua` sets `SO_REUSEADDR` before bind (a deliberate fast-restart fix, #128) and Windows gives `SO_REUSEADDR` `SO_REUSEPORT` semantics, so the second process re-binds the same port and both hubs then share (and race on) `cfg/user.tbl`, `master.key`, the plugin `scripts/data/*.tbl` stores and the logs; interleaved `saveusers()`/`savetable()` writes can corrupt them (data loss). Fixed in the C launcher (`hub/hub.c`) with a per-install lock file (`luadch.lock` in the install dir): unix `flock(LOCK_EX|LOCK_NB)` + `O_CLOEXEC`, Windows exclusive `CreateFile` (deny-share, non-inheritable handle, delete-on-close). Held open for the whole process lifetime and released by the OS on crash (no stale-lock cleanup step), retained by the daemon child across the `-d` fork and across `+reload` (the fd survives the `restart()`/atexit re-entry). Scope is per install directory - a test hub and a prod hub in separate directories on one box still run in parallel; only a second copy of the SAME install is refused (non-zero exit + "another instance is already running" to stderr and `log/exception.txt`). Fail-open: only the definitive already-held signal (unix `EWOULDBLOCK`, Windows `ERROR_SHARING_VIOLATION`) blocks boot; an un-creatable lock file (read-only fs and the like) warns and continues. Deployment note: a supervisor that starts the replacement before the old process has finished its shutdown drain sees the refusal - stop must complete before start (systemd / `docker restart` are sequential and unaffected). Smoke: a new stage starts a second hub against the live staging tree and asserts it exits non-zero with the refusal while the first stays up, provably failing pre-fix (§1a.7 - on the unpatched Windows binary the second instance boots and runs concurrently). Two-pass review (§1a.6) caught the missing `O_CLOEXEC` (a spawned child could otherwise inherit the fd and brick the next restart) and the `ERROR_ACCESS_DENIED`-is-not-a-live-instance classification. 3.2.x only, not backported (hardening, not a security/crash fix; per §8 the 3.1.x bar is high). Operator guide: [`docs/BUILDING.md`](docs/BUILDING.md). + - **`usr_hide_share`: also hide the shared-file count, not just the share size.** The plugin zeroed only the ADC INF `SS` (share size) for a hidden user, never `SF` (shared-file count), so clients still displayed how many files the user shared. Fixed by mirroring every `SS`-zeroing write with an `SF`-zeroing one at all three sites (the `onInf`/`onConnect`/`onStart`/`onExit` listener and the manual `+hideshare` command) and adding `SF0` to the `BINF` broadcast. v0.4 -> v0.5. Intended side effect (a consistency fix): a hidden user now also contributes 0 to the hub's aggregate file counts (PING reply to hublist scrapers, Prometheus `/metrics`, HTTP stats), matching the pre-existing behaviour for aggregate share size; no new kick path (`usr_share` min-share enforcement keys on `SS` only). §1a.7: diff-is-self-evidence (the unpatched plugin never touched `SF`; the fix is a 1:1 mirror of the already-working `SS` path) plus testhub validation. 3.2.x; not backported (self-contained plugin, a 3.1.x drop-in is possible on request). - [#419](https://github.com/luadch-ng/luadch/issues/419) (Kcchouette) - **the ADC parser now discards messages containing unknown escape sequences** (ADC 1.0 section 3.1: only `\s` / `\n` / `\\` are defined; "any message containing unknown escapes must be discarded"). `core/adc.lua parse()` validated UTF-8, structure and field values but never escapes, so a message carrying `\q` (or a trailing/unescaped backslash) was accepted and relayed - other clients then interpret the unknown escape differently (undefined per spec), a cross-client display-impersonation vector (`admin\q` may render as `admin` on a client that drops it; complements the #315 `unescape` hardening, which kept the receiving-side lenient - this closes the relay side). Validated **pairwise** (strip the valid `\s`/`\n`/`\\` first; any leftover backslash is then unknown): a naive "`\` not followed by s/n/`\`" scan is wrong - it false-positives on `\\q` (an escaped backslash + literal `q`, i.e. the valid wire form of the text `\q`) and misses a lone trailing backslash. A backslash-free message (most protocol commands) skips the gsub via a single plain `find`; a message with escapes (e.g. any multi-word chat, which carries `\s`) takes one bounded single-pass gsub - both cheap. Compliant clients never emit unknown escapes and are unaffected (and a plugin that hand-builds a raw message body should escape backslashes as `\\` - the `hub.broadcast` / `user:reply` / `cmd:adcstring` helpers already do); a drop is logged to `event.log`. Smoke: 3 checks on both legs - `\q` discarded + trailing backslash discarded (both provably fail pre-fix, §1a.7) + a positive control that a valid `\\` is NOT discarded (guards the false-positive). Spec-compliance, no hub-crash / no remote-DoS: 3.2.x only, not backported. @@ -140,6 +144,14 @@ land on `release/3.1.x` per ### Features +- **#78 allowlist, Phase D: HTTP API for the whitelist (`etc_whitelist` v0.02).** Four endpoints mirroring the blocklist HTTP API: `GET /v1/whitelist` (read, filter/sort/paginate via `http_filter`), `GET /v1/whitelist/counts` (read), `POST /v1/whitelist` (admin, body `{cidr, source?, reason?, expires_at?}`, source enum `{manual, pinger}`, host-bits-set CIDRs rejected), `DELETE /v1/whitelist/{id}` (admin). HTTP admin tokens bypass the ADC hierarchy guard (the token is the trust surface; entries attributed `by_nick = ` / `by_level = 100`). `etc_whitelist_test` gains 32 HTTP checks (registration + scope + schema + all four handlers, happy + error paths); a seed-aware `test_http_phase_d_whitelist` smoke does a real token-authed POST -> GET -> DELETE roundtrip (baseline count captured so it stays robust to the pinger seed). Docs: `SCRIPTS.md` (new `etc_whitelist` section) + `BLOCKLIST.md` (allowlist layer). Completes the 4-PR #78 allowlist arc. 3.2.x only. + +- **#78 allowlist, Phase C: the IP-blocking plugins now consult the whitelist first.** `etc_geoip` (country/ASN kick), `etc_proxydetect` (proxy kick + the paid provider query) and `usr_hubs` (invalid-hubcount kick + hub-limit ban) each gained a one-line `if whitelist.is_whitelisted(ip) then return end` guard at the earliest point of their per-connection decision, so a whitelisted IP (hublist pinger / trusted infra) is exempt from these three - and for `etc_proxydetect` the guard sits BEFORE the cache / quota / HTTP so a trusted IP never costs a provider query. This silences the recurring `PROXYDETECT` / `GEOIP` / `USER HUBS` log lines for whitelisted IPs. Scope: this covers the IP-reputation blockers (plus the unified blocklist store via Phase A) and the hub-limit ban; the share / slots / nick-policy plugins (`usr_share` / `usr_slots` / `usr_nick_*`) are NOT yet whitelist-aware (a whitelisted IP still faces those), a possible follow-up. `etc_geoip_test` and `etc_proxydetect_test` gain whitelist-exemption regressions that provably fail without the guard (§1a.7); `usr_hubs` has no unit harness so its guard is the identical one-line mirror (diff-is-self-evidence). 3.2.x only. + +- **#78 allowlist, Phase B: the `+whitelist` admin plugin + bundled hublist-pinger seed (`etc_whitelist.lua`).** Operator-facing chat command over the Phase A engine: `+whitelist show [source] | add [reason="..."] [expires=YYYY-MM-DD] | del | count | export | import `. Mirrors `etc_blocklist`'s structure (verb dispatch, `by_level` hierarchy guard on del, JSONL export/import with control-byte stripping + a master-only `etc_whitelist_import_min_level`, opchat report + audit fire-sites `whitelist.add`/`.remove`/`.export`/`.import`) minus stealth/HTTP. **Bundled seed:** on the FIRST run (store `.tbl` missing) the plugin seeds a small set of known hublist-pinger IPs (source=`pinger`, v6 as `/64`, v4 as `/32`) so the pingers are exempt from the automated blockers out of the box; seed-on-missing only (operator edits/deletions are never re-seeded), disable with `etc_whitelist_seed = false`. Ships enabled in `examples/cfg/cfg.tbl`. cfg keys `etc_whitelist_oplevel` / `_show_limit` / `_seed` / `_report[_hubbot|_opchat]` / `_llevel` / `_import_min_level`; lang en + de. `tests/unit/etc_whitelist_test.lua` (53 checks: parse / add / del-hierarchy / sanitize / format / seed-on-missing / export-import) + a `+whitelist` roundtrip smoke that also validates the seed lands on first boot; both registered on both smoke.yml legs. 3.2.x only. + +- **#78 allowlist, Phase A: a global IP/CIDR whitelist engine (`core/whitelist.lua`).** The allowlist deferred from the unified-blocklist arc. A new core module - a stripped-down sibling of `core/blocklist.lua` (same bucketed radix cache, hex-encoded `.tbl` store, v4-mapped-v6 lookup; no source-priority / stealth / rollup) - exposes `whitelist.is_whitelisted(ip)` as a sandbox global so every IP-blocking path can consult it. `blocklist.check_ip` now applies **Model-A precedence**: a whitelisted IP overrides an AUTOMATED block (GeoIP / proxydetect / external feed) but NOT a deliberate manual pin (`source = "manual"` wins), so an operator can still block a specific IP that sits inside a whitelisted range. Full IPv4/IPv6 + CIDR support (a v6 `/64` covers a rotating pinger host); `expires_at` TTL; survives `+reload` via a cfg-reload listener. cfg keys `whitelist_enabled` (default true) + `whitelist_store_path` (`scripts/data/etc_whitelist.tbl`); the engine ships enabled with an empty store (zero overhead via a single `next()` fast path until an entry is added). `tests/unit/whitelist_test.lua` (63 checks incl. a blocklist-integration block whose precedence assertions provably fail on a blocklist without the hook, §1a.7); registered on both smoke.yml legs. Phase A of a 4-PR feature (B: `+whitelist` admin plugin + bundled hublist-pinger seed; C: per-plugin guards in etc_geoip / etc_proxydetect / usr_hubs; D: HTTP `/v1/whitelist`). 3.2.x only. + - [#420](https://github.com/luadch-ng/luadch/issues/420) (#398 follow-up) - **`etc_webhook`: per-endpoint `conditions` filter on a decoded body field** (v0.02 -> v0.03). The event filter only matched the event *header*, but two real cases need a *body* field: a GitHub `release` webhook fires for every action (`created` / `edited` / `published` / `released`) under one `x-github-event: release`, and a Discourse new topic fires both `topic_created` AND `post_created` (its auto opening post) - so a release announced 5x and a new topic twice. New optional `conditions = { { path = "dotted.path", equals = X }, ... }` (or `not_equals`); ALL must hold to announce, evaluated against the decoded body via the existing `resolve_path` extraction (no new parsing surface). Values compare as strings (a number or string works); an unresolved path is `nil`, so `not_equals` passes when the field is absent (a `topic_created` carries no `post.post_number`, so the topic still announces while its opening post is dropped). Conditions apply endpoint-wide. The example config now filters GitHub to `action = "released"` (one clean line + release URL) and drops the Discourse opening post via `post.post_number != 1` (real replies still announce). `tests/unit/etc_webhook_test.lua` gains 6 checks, 3 of which provably fail pre-fix (§1a.7 - the unpatched plugin announces the opening post, a float `1.0` opening post, and `action=created`; the other 3 are positive controls that announce on both). Numbers compare numerically (a config `1` matches JSON `1` or `1.0`), otherwise as strings. Docs: `docs/WEBHOOKS.md` + `examples/cfg/webhooks.tbl`. 3.2.x only. - [#398](https://github.com/luadch-ng/luadch/issues/398) - **`etc_webhook.lua`: a generic INBOUND webhook receiver.** An external service (a Discourse forum, GitHub, GitLab, CI, monitoring, ...) POSTs an HMAC-SHA256-signed JSON body to a hub HTTP endpoint; the hub verifies the signature over the raw body (`adclib.constant_time_eq`), applies an event filter + delivery-id dedup, renders a `{dotted.path}` template and announces the event in chat as a named bot. The inbound-push mirror of `etc_status_push` (outbound heartbeat) and the complement of `etc_prometheus` (pull `/metrics`). **Multi-endpoint:** the operator-edited `cfg/webhooks.tbl` holds an array of endpoints, each with its own path / signature+event+id headers / event filter / bot nick / min-level / templates, so Discourse and GitHub can be received in parallel; `cfg.tbl` carries only the master switch `etc_webhook_activate` (keeps it lean). Per-endpoint secrets resolve env-var-first (`LUADCH_ETC_WEBHOOK__SECRET`), else the `etc_webhook__secret` cfg key, else an inline `secret` in `cfg/webhooks.tbl`; an endpoint with no resolvable secret is skipped (never runs unsigned). First plugin to register a **`scope="none"`** HTTP route - the router's bearer-token gate is skipped and the handler does its OWN HMAC auth; a missing/wrong signature is 401. Untrusted template fields are control-byte-stripped + length-capped, a per-minute flood cap bounds chat spam, and duplicate deliveries are deduped by the delivery-id header. Built on `core/hmac.lua` (the #398 precursor). Off by default (`etc_webhook_activate=false`; whitelisted `enabled=false` - advanced, needs `cfg/webhooks.tbl` + a reverse proxy). Unit-tested with the REAL hmac module (`tests/unit/etc_webhook_test.lua`, both smoke legs, 23 checks: HMAC pass/wrong/missing, `sha256=` prefix, event filter, dedup, template `{dotted.path}`/`{event}`/missing-path, control-byte strip + truncation, flood cap, min_level gating, no-secret-skip, activate=false inert). Operator guide `docs/WEBHOOKS.md`; security model in `docs/SECURITY.md`. 3.2.x only. diff --git a/CLAUDE.md b/CLAUDE.md index 4ab8c369..9df69397 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -130,7 +130,7 @@ order (its inline comments explain each ordering constraint). | Subsystem | Modules | Responsibility | |---|---|---| | Boot + config | `init`, `const`, `cfg`, `cfg_defaults`, `cfg_users`, `cfg_lang`, `cfg_secret`, `secrets` | Restricted env + module loader; program constants; settings/user.tbl/language handling; AES-256-GCM at-rest crypto; env-var-first secret lookup | -| Network + ADC | `server`, `iostream`, `adc`, `hub`, `hub_dispatch`, `hub_user_object`, `hub_bot_object`, `hbri`, `ratelimit`, `blocklist`, `ipmatch` | select() loop + SSL; framing pipeline; ADC parse/escape/format; main loop + login; command dispatch; user/bot objects; dual-stack secondary-IP verification; DoS limits; pre-handshake IP/CIDR blocklist; IP/CIDR primitives | +| Network + ADC | `server`, `iostream`, `adc`, `hub`, `hub_dispatch`, `hub_user_object`, `hub_bot_object`, `hbri`, `ratelimit`, `blocklist`, `whitelist`, `ipmatch` | select() loop + SSL; framing pipeline; ADC parse/escape/format; main loop + login; command dispatch; user/bot objects; dual-stack secondary-IP verification; DoS limits; pre-handshake IP/CIDR blocklist; global allowlist (whitelist beats automated blocks, not manual pins); IP/CIDR primitives | | HTTP API | `http`, `http_router`, `http_client`, `http_filter`, `http_events`, `util_http` | Inbound HTTP/JSON API + router + auth; non-blocking OUTBOUND client; filter/sort/paginate helper; deferred-event endpoints; plugin endpoint helper | | Crypto + boot trust | `sha256`, `hmac`, `cert_bootstrap`, `cacert_bootstrap` | Pure-Lua SHA-256; HMAC-SHA256 (RFC 2104, sandbox-exposed for signed-webhook auth, #398); first-boot TLS-cert auto-gen (#77); CA-bundle reconciliation | | Infra | `util`, `out`, `mem`, `signal`, `types`, `scripts`, `audit`, `sysinfo`, `mmdb`, `geoip_update`, `bloom`, `ensuredirs`, `doc` (disabled), `hci` (stub) | File I/O + table helpers; logging; GC; timers; ADC type validation; plugin loader + sandbox + listener registry; onAudit JSONL log; system info; MaxMind DB reader + in-hub GeoLite2 auto-update; bloom filter; boot-time runtime-dir self-heal; dormant | @@ -261,7 +261,14 @@ ADC parser now discards messages with unknown escape sequences per ADC 3.1 (#419) + hub-bot INF `EM` escaping (#423), and an `etc_webhook` body-field `conditions` filter (#420 - fixed a live double-announce by filtering on a JSON body field like a GitHub release `action=released` or a Discourse -opening post, not just the event header). A +opening post, not just the event header). On `dev` (PR #432): +`core/sysinfo.lua` falls back `Get-CimInstance` -> `Get-WmiObject` so +old-Windows hubowners (Server 2008 R2 / Win7 = PowerShell 2.0, which has +no `Get-CimInstance`) get real `+hubinfo` OS/CPU/RAM instead of +`` (Sopor; the pre-refactor 3.1.x plugin also CRASHED on the +nil-concat - shipped a v0.30 `cmd_hubinfo` drop-in per §8). Many +hubowners run ancient Windows (the UCRT release build also needs +KB2999226 there - the Universal C Runtime). A recurring pattern this era: a **periodic-fetch plugin must persist its next-fetch deadline across `+reload`**, or every reload re-hits a rate-limited provider - fixed twice @@ -272,6 +279,21 @@ from source per §1a.3/4, since many are old-version reports asking "fixed in 3.x?"). GitFlow A per fix; batch dev->master as a MERGE commit (never squash - see §8 branch hygiene). +**On `dev`: the #78 allowlist (global whitelist) - 4-PR arc A-D MERGED +(PRs #427/#431/#429/#430), pending testhub -> dev->master.** The allowlist deferred from the +unified-blocklist arc. A `core/whitelist.lua` engine consulted by every +IP-blocking path so trusted infrastructure (hublist pingers etc.) is exempt +from the AUTOMATED blockers (GeoIP / proxydetect / feeds / hub-limit) - but +NOT from a deliberate manual `+ban` / `+blocklist` (Model A: a manual block +wins). Phase A (engine + `blocklist.check_ip` precedence + +`whitelist.is_whitelisted` sandbox global), B (`+whitelist` plugin + +bundled hublist-pinger seed, `etc_whitelist`), C (per-plugin guards in +etc_geoip / etc_proxydetect / usr_hubs - where the log goes quiet), D (HTTP +`/v1/whitelist`). NOT extended to the share / slots / nick-policy plugins +(`usr_share` / `usr_slots` / `usr_nick_*`). Source of truth: the +whitelist-arc PRs (deferred item of +[#78](https://github.com/luadch-ng/luadch/issues/78)). + **HTTP-endpoint authoring** (which helper for which endpoint shape, envelope contract, preflight): see [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) §3 and [`docs/HTTP_API.md`](docs/HTTP_API.md) §5. diff --git a/core/blocklist.lua b/core/blocklist.lua index 896d5811..f71120d2 100644 --- a/core/blocklist.lua +++ b/core/blocklist.lua @@ -83,6 +83,7 @@ local use = use local type = use "type" +local pcall = use "pcall" local next = use "next" local pairs = use "pairs" local ipairs = use "ipairs" @@ -121,6 +122,12 @@ local cfg_get local out_put local out_error +-- Whitelist (core/whitelist.lua) precedence hook. Captured in init() +-- once all _core modules are present; nil until then (and in a unit +-- test that stubs `use` without the whitelist module - the check_ip +-- guard below is a no-op in that case, preserving legacy behaviour). +local whitelist_is_whitelisted + -- Source priority: higher number = higher priority. When multiple -- entries match the same IP, the winner is the highest-priority. -- New sources for Phase D/E/F register into this table. @@ -474,6 +481,22 @@ local function check_ip( ip ) local entry = _resolve_decision( ip ) if not entry then return false end + -- Whitelist precedence (#78 allowlist, Model A): a whitelisted IP + -- overrides an AUTOMATED block (geoip / proxycheck / external feed) + -- but NOT a deliberate manual pin. source == "manual" is the + -- operator's explicit "block this" and wins, so a manual + -- +blocklist/+ban still applies inside a whitelisted range. On an + -- override we return BEFORE the per-attempt log + rollup so a + -- trusted IP produces zero block noise (the point of the allowlist). + -- pcall-guarded: check_ip runs on the accept path (server.lua -> + -- hub.loop) with no pcall in the chain, so a throw in the whitelist + -- module would kill the hub loop (the #401 accept-path crash class). + -- Fail CLOSED - an error keeps the block, the safe direction. + if entry.source ~= "manual" and whitelist_is_whitelisted then + local ok_wl, hit = pcall( whitelist_is_whitelisted, ip ) + if ok_wl and hit then return false end + end + local now = socket_gettime( ) -- Per-attempt visible log when NOT stealth; the rollup picks -- up the count regardless. @@ -942,6 +965,15 @@ local function init( ) out_put = out_mod.put out_error = out_mod.error + -- Capture whitelist.is_whitelisted for the check_ip precedence hook. + -- Post-load (every _core module present); pcall-guarded so a unit + -- test that stubs `use` without the whitelist module still loads and + -- the guard stays a no-op (legacy block-only behaviour). + local ok_wl, wl = pcall( use, "whitelist" ) + if ok_wl and type( wl ) == "table" then + whitelist_is_whitelisted = wl.is_whitelisted + end + _resnapshot_cfg( ) reload( ) diff --git a/core/cfg_defaults.lua b/core/cfg_defaults.lua index ad5c675e..269cf775 100644 --- a/core/cfg_defaults.lua +++ b/core/cfg_defaults.lua @@ -4563,6 +4563,74 @@ local defaults = { return value >= 60 and value <= 86400 end }, + -- #78 allowlist (core/whitelist.lua): a global IP/CIDR allowlist + -- consulted by every IP-blocking path. A match exempts the IP from + -- the AUTOMATED blockers (GeoIP / proxydetect / feeds / hub-limit) + -- and from automated blocklist-store entries, but NOT from a + -- deliberate manual +blocklist/+ban (a manual block wins). Engine + -- ships enabled with an empty store; Phase B's `+whitelist` plugin + -- seeds the bundled hublist-pinger allowlist on first run. + whitelist_enabled = { true, + function( value ) + return types_boolean( value, nil, true ) + end + }, + whitelist_store_path = { "scripts/data/etc_whitelist.tbl", + function( value ) + return types_utf8( value, nil, true ) + end + }, + -- #78 allowlist, Phase B: `+whitelist` admin plugin + -- (etc_whitelist.lua). Operator-facing chat command + JSONL + -- export/import + a bundled hublist-pinger seed. The engine in + -- core/whitelist.lua is independent; these keys gate the plugin. + etc_whitelist_oplevel = { 80, + function( value ) + return types_number( value, nil, true ) + end + }, + etc_whitelist_show_limit = { 200, + function( value ) + if not types_number( value, nil, true ) then return false end + return value >= 1 and value <= 10000 + end + }, + -- Seed the bundled known-hublist-pinger allowlist on the FIRST run + -- (store .tbl missing). Seed-on-missing only; operator edits are + -- never overwritten. Set false to boot with an empty whitelist. + etc_whitelist_seed = { true, + function( value ) + return types_boolean( value, nil, true ) + end + }, + etc_whitelist_report = { true, + function( value ) + return types_boolean( value, nil, true ) + end + }, + etc_whitelist_report_hubbot = { false, + function( value ) + return types_boolean( value, nil, true ) + end + }, + etc_whitelist_report_opchat = { true, + function( value ) + return types_boolean( value, nil, true ) + end + }, + etc_whitelist_llevel = { 60, + function( value ) + return types_number( value, nil, true ) + end + }, + -- Minimum level to RUN `+whitelist import `. Same threat + -- model as etc_blocklist_import_min_level: JSONL rows may carry + -- master-added entries; default 100 = master-only. + etc_whitelist_import_min_level = { 100, + function( value ) + return types_number( value, nil, true ) + end + }, -- #78 Phase B: `+blocklist` admin plugin (etc_blocklist.lua). -- Operator-facing chat command + JSONL export/import. The -- engine in core/blocklist.lua is independent; these keys diff --git a/core/hub.lua b/core/hub.lua index ba1059d8..ef2c4112 100755 --- a/core/hub.lua +++ b/core/hub.lua @@ -1660,7 +1660,19 @@ add_server_handler = function( p ) local hndl, err = server.addserver( p ) if hndl then _servers[ hndl ] = true - elseif err and err:find("address already in use") then + return + end + -- addserver failed. server.addserver already logged the raw luasocket + -- error; add a clear operator-facing line so a port clash is obvious. + -- The #1 cause is a second hub (or another program) on the same port - + -- on Windows the SO_REUSEADDR skip in server.lua makes that clash + -- surface here instead of the second bind silently succeeding. + local proto = p.sslctx and "TLS" or "plain" + local fam = ( p.family == "ipv6" ) and "IPv6" or "IPv4" + if err and err:find( "address already in use" ) then + out_error( "server: ", fam, " ", proto, " port ", p.port, + " is already in use - another process (or a second hub) holds it.", + " Stop it or change the port in cfg/cfg.tbl; retrying every 30s." ) local starttime = os.time() server.addtimer( coroutine.create( @@ -1679,6 +1691,9 @@ add_server_handler = function( p ) end ) ) + else + out_error( "server: could not start the ", fam, " ", proto, + " listener on port ", p.port, ": ", err or "unknown error" ) end end diff --git a/core/init.lua b/core/init.lua index 5c5d4bb2..abe21030 100755 --- a/core/init.lua +++ b/core/init.lua @@ -123,6 +123,13 @@ _core = { -- luadch core, order is important -- + CIDR parse + prefix-match primitives. No deps beyond -- stdlib; load order is just before its consumer blocklist. "ipmatch", + -- core/whitelist.lua (#78 allowlist): global IP/CIDR allowlist + -- consulted by blocklist.check_ip and the IP-blocking plugins. + -- Loaded AFTER ipmatch (uses its parse/match primitives) and + -- BEFORE blocklist, which captures whitelist.is_whitelisted in + -- its init() to apply the "whitelist overrides an automated block" + -- precedence. Same passive-at-load / init()-reads-cfg contract. + "whitelist", -- core/blocklist.lua (Phase A of #78 arc): unified pre-handshake -- IP/CIDR blocklist (in-memory bucketed cache + scripts/data/etc_blocklist.tbl -- persistent store + decision API). Loaded BEFORE ratelimit + diff --git a/core/scripts.lua b/core/scripts.lua index bd89ec61..9cb724c6 100755 --- a/core/scripts.lua +++ b/core/scripts.lua @@ -245,6 +245,12 @@ local SANDBOX_GLOBALS = { -- webhook tokens) call `secrets.lookup(cfg_key)` so Docker -- operators can set keys via env vars instead of cfg.tbl. "secrets", + -- core/whitelist.lua (#78 allowlist): global IP/CIDR allowlist. + -- Every IP-blocking plugin (etc_geoip / etc_proxydetect / usr_hubs + -- / ...) calls whitelist.is_whitelisted(ip) before its own block so + -- trusted infrastructure (hublist pingers etc.) is exempt from the + -- automated blockers. Phase B's etc_whitelist.lua manages entries. + "whitelist", -- core/blocklist.lua (#78 Phase A): unified pre-handshake -- IP/CIDR blocklist engine. Phase B's etc_blocklist.lua and -- Phase D/E/F's auto-feed plugins call blocklist.add / diff --git a/core/server.lua b/core/server.lua index 630f7dc9..7e3a4db4 100755 --- a/core/server.lua +++ b/core/server.lua @@ -87,6 +87,7 @@ local ssl_newcontext = ( luasec and luasec.newcontext ) local cfg = use "cfg" local out = use "out" +local util = use "util" local mem = use "mem" local signal = use "signal" local ratelimit = use "ratelimit" @@ -102,6 +103,14 @@ local out_error = out.error local signal_set = signal.set local signal_get = signal.get local ratelimit_accept_ip = ratelimit.accept_ip +-- Platform gate for SO_REUSEADDR (see addserver): Linux needs it for the +-- TIME_WAIT fast-restart (#128); on Windows SO_REUSEADDR instead lets a +-- SECOND process re-bind a port already in use (SO_REUSEPORT-like +-- semantics), so a same-port second hub binds silently and any process +-- could hijack a luadch port. Skipping it on Windows makes that second +-- bind fail with WSAEADDRINUSE ("address already in use") so the operator +-- gets a clear message instead of two hubs racing. +local _is_windows = ( util.path_sep( ) == "\\" ) local ratelimit_release_ip = ratelimit.release_ip local blocklist_check_ip = blocklist.check_ip local ratelimit_handshake_started = ratelimit.handshake_started @@ -1098,18 +1107,22 @@ addserver = function( p ) -- listeners, port, addr, pattern, sslctx, maxconnecti out_error( "server.lua: function 'addserver', luasocket cannot create master obejct: ", err ) return nil, err end - -- Set SO_REUSEADDR BEFORE bind. On Linux the option only takes - -- effect on the next bind() call; setting it after bind is a - -- no-op and a fast restart hits "address already in use" if the - -- previous listener left the port in TIME_WAIT. Surfaced by - -- the #128 plaintext-mode smoke test which stops + restarts the - -- hub against the same staging tree. Pre-existing latent bug; - -- the post-bind setoption stays for completeness but the - -- pre-bind one is what actually unblocks the rebind. - local _, prebind_err = server:setoption( "reuseaddr", true ) - if prebind_err then - out_error( "server.lua: function 'addserver', luasocket socket pre-bind setoption: ", prebind_err ) - return nil, prebind_err + -- Set SO_REUSEADDR BEFORE bind (Linux only - see _is_windows above). + -- On Linux the option only takes effect on the next bind() call; + -- setting it after bind is a no-op and a fast restart hits "address + -- already in use" if the previous listener left the port in + -- TIME_WAIT. Surfaced by the #128 plaintext-mode smoke test which + -- stops + restarts the hub against the same staging tree. Pre-existing + -- latent bug; the post-bind setoption stays for completeness but the + -- pre-bind one is what actually unblocks the rebind. Deliberately + -- SKIPPED on Windows so a same-port second bind fails visibly instead + -- of silently succeeding (SO_REUSEADDR = SO_REUSEPORT there). + if not _is_windows then + local _, prebind_err = server:setoption( "reuseaddr", true ) + if prebind_err then + out_error( "server.lua: function 'addserver', luasocket socket pre-bind setoption: ", prebind_err ) + return nil, prebind_err + end end local num, err = server:bind( p.addr, p.port ) if err then @@ -1128,11 +1141,18 @@ addserver = function( p ) -- listeners, port, addr, pattern, sslctx, maxconnecti return nil, err end server:settimeout( 0 ) - local _, err = server:setoption( "reuseaddr", true ) + -- reuseaddr on the bound socket is Linux-only for the same reason as + -- the pre-bind call above: on Windows it would re-open the port to + -- being hijacked by another process. keepalive is set on every + -- platform. + local err + if not _is_windows then + _, err = server:setoption( "reuseaddr", true ) + end local _, err2 = server:setoption( "keepalive", true ) if err or err2 then out_error( "server.lua: function 'addserver', luasocket socket setoption: ", err or err2 ) - return nil, err + return nil, err or err2 end _readlistlen = _readlistlen + 1 _readlist[ _readlistlen ] = server diff --git a/core/whitelist.lua b/core/whitelist.lua new file mode 100644 index 00000000..5351be78 --- /dev/null +++ b/core/whitelist.lua @@ -0,0 +1,542 @@ +--[[ + + core/whitelist.lua - global IP/CIDR allowlist consulted by every + IP-blocking path (the deferred #78 allowlist). + + A trusted-infrastructure allowlist: an IP/CIDR on this list is + exempt from the AUTOMATED blockers - GeoIP country/ASN, proxy + detection, external feeds (Tor / Spamhaus / AbuseIPDB), the + hub-limit ban - AND from the automated entries in the unified + blocklist store. It does NOT override a deliberate manual block + (a `+blocklist add` with source="manual" or a `+ban`): a + manual/operator block wins, so an operator can still block a + specific IP even if it falls in a whitelisted range. If you need + to block a whitelisted IP, remove it from the whitelist first. + + Structurally this is a stripped-down sibling of core/blocklist.lua: + same in-memory bucketed cache, same hex-encoded .tbl store, same + v4-mapped-v6 normalisation on lookup. It drops everything a + deny-engine needs but an allow-set does not: no source PRIORITY + (any match = allowed; the `source` field is a label only, e.g. + "manual" vs the bundled "pinger" seed), no stealth flag, no + aggregated-log rollup, no feed bulk_replace. + + Consumed by: + * core/blocklist.lua check_ip() - overrides an automated block + pre-handshake (manual pins excepted, see above). + * plugins that block by IP (etc_geoip / etc_proxydetect / + usr_hubs / ...) - each calls whitelist.is_whitelisted(ip) + before its own block. Exposed as the sandbox global + `whitelist` (mirrors `blocklist`). + + Public surface: + + whitelist.is_whitelisted(ip) -> bool + true if ip is covered by an active (non-expired) entry. + nil / empty ip -> false. Disabled or empty store -> false + via a single next() fast-path (zero meaningful overhead). + + whitelist.add(cidr_or_ip, opts) -> ok, id, err + opts = { source, reason, by_nick, by_level, expires_at, meta } + source defaults to "manual" (a label only, no priority). + + whitelist.remove(id) -> ok, err + By numeric id. false + "not_found" if absent. + + whitelist.list(filter_spec) -> rows (filter: { source }) + whitelist.count() -> { total, by_source } + whitelist.reload() re-read the .tbl + whitelist._resolve_match(ip) -> entry | nil (exposed for tests) + +]]-- + +local use = use + +local type = use "type" +local next = use "next" +local pairs = use "pairs" +local ipairs = use "ipairs" +local tostring = use "tostring" +local tonumber = use "tonumber" +local string = use "string" +local table = use "table" +local math = use "math" +local io = use "io" +local socket = use "socket" + +local string_byte = string.byte +local string_format = string.format +local string_sub = string.sub +local table_insert = table.insert +local table_remove = table.remove +local math_floor = math.floor +local io_open = io.open +local socket_gettime = socket.gettime + +local util = use "util" +local ipmatch = use "ipmatch" + +local util_loadtable = util.loadtable +local util_savetable = util.savetable +local ipmatch_parse_cidr = ipmatch.parse_cidr +local ipmatch_parse_ip = ipmatch.parse_ip +local ipmatch_match = ipmatch.match +local ipmatch_format_bytes = ipmatch.format_bytes + +-- Late-bound core deps (bound in init(), mirroring blocklist.lua). +local cfg_get +local out_put +local out_error + +---------------------------------------------------------------------- +-- State +---------------------------------------------------------------------- + +local _enabled = true +local _store_path = "scripts/data/etc_whitelist.tbl" + +local _entries = { } -- array of entry records, order = insert +local _next_id = 1 +local _buckets_v4 = { } -- [byte 1] -> array of entry refs +local _buckets_v6 = { } -- [byte1 * 256 + byte2] -> array of entry refs + +---------------------------------------------------------------------- +-- Bucket helpers (identical radix scheme to core/blocklist.lua: first +-- byte for v4, first two bytes for v6; short prefixes enroll in every +-- covering bucket so the lookup stays O(1) + linear-scan-of-bucket). +---------------------------------------------------------------------- + +local function _bucket_for( family, network_bytes, prefix_len ) + if family == 4 then + if prefix_len >= 8 then + return { string_byte( network_bytes, 1 ) } + end + local first_byte = string_byte( network_bytes, 1 ) + local span = ( 1 << ( 8 - prefix_len ) ) + local out = { } + for i = 0, span - 1 do + out[ #out + 1 ] = first_byte + i + end + return out + end + -- family == 6 + if prefix_len >= 16 then + local b1, b2 = string_byte( network_bytes, 1, 2 ) + return { b1 * 256 + b2 } + end + local b1 = string_byte( network_bytes, 1 ) + if prefix_len >= 8 then + local b2 = string_byte( network_bytes, 2 ) + local span = ( 1 << ( 16 - prefix_len ) ) + local base = b1 * 256 + b2 + local out = { } + for i = 0, span - 1 do + out[ #out + 1 ] = base + i + end + return out + end + local span = ( 1 << ( 8 - prefix_len ) ) + local out = { } + for i = 0, span - 1 do + local first = b1 + i + for j = 0, 255 do + out[ #out + 1 ] = first * 256 + j + end + end + return out +end + +local function _bucket_table( family ) + if family == 4 then return _buckets_v4 end + return _buckets_v6 +end + +local function _bucket_insert( entry ) + local tbl = _bucket_table( entry.family ) + for _, key in ipairs( entry._buckets ) do + local list = tbl[ key ] + if not list then + list = { } + tbl[ key ] = list + end + list[ #list + 1 ] = entry + end +end + +local function _bucket_remove( entry ) + local tbl = _bucket_table( entry.family ) + for _, key in ipairs( entry._buckets ) do + local list = tbl[ key ] + if list then + for i, e in ipairs( list ) do + if e == entry then + table_remove( list, i ) + break + end + end + if #list == 0 then tbl[ key ] = nil end + end + end +end + +---------------------------------------------------------------------- +-- Disk persistence (Lua-table dump; raw network bytes hex-encoded to +-- keep the .tbl text-safe, exactly as core/blocklist.lua does). +---------------------------------------------------------------------- + +local function _hex_encode( s ) + local out = { } + for i = 1, #s do + out[ i ] = string_format( "%02x", string_byte( s, i ) ) + end + return table.concat( out ) +end + +local function _hex_decode( s ) + if type( s ) ~= "string" or ( #s % 2 ) ~= 0 then return nil end + local out = { } + for i = 1, #s, 2 do + local byte_val = tonumber( s:sub( i, i + 1 ), 16 ) + if not byte_val then return nil end + out[ #out + 1 ] = string.char( byte_val ) + end + return table.concat( out ) +end + +local function _persist_one( e ) + return { + id = e.id, + cidr = e.cidr, + family = e.family, + network_b64 = e._network_b64, + prefix_len = e.prefix_len, + source = e.source, + reason = e.reason or "", + by_nick = e.by_nick or "", + by_level = e.by_level, + expires_at = e.expires_at, + created_at = e.created_at, + meta = e.meta, + } +end + +local function _save_to_disk( ) + local snapshot = { } + for i, e in ipairs( _entries ) do + snapshot[ i ] = _persist_one( e ) + end + local ok, err = util_savetable( snapshot, "whitelist", _store_path ) + if not ok then + if out_error then out_error( "whitelist: save failed: " .. tostring( err ) ) end + return false, err + end + return true +end + +---------------------------------------------------------------------- +-- Entry construction +---------------------------------------------------------------------- + +local function _rebuild_indices( ) + _buckets_v4 = { } + _buckets_v6 = { } + for _, e in ipairs( _entries ) do + e._buckets = _bucket_for( e.family, e.network_bytes, e.prefix_len ) + _bucket_insert( e ) + end +end + +local function _make_entry( cidr, opts ) + local family, network_bytes, prefix_len = ipmatch_parse_cidr( cidr ) + if not family then + return nil, network_bytes -- second value is the err string + end + local canonical_ip = ipmatch_format_bytes( family, network_bytes ) + if not canonical_ip then + return nil, "internal: cannot format canonical CIDR from bytes" + end + local canonical_cidr = canonical_ip .. "/" .. prefix_len + + opts = opts or { } + local now = socket_gettime( ) + local entry = { + id = _next_id, + cidr = canonical_cidr, + family = family, + network_bytes = network_bytes, + prefix_len = prefix_len, + -- `source` is a LABEL only (no priority): "manual" for an + -- operator add, "pinger" for the bundled seed, etc. Any match + -- means allowed regardless of source. + source = opts.source or "manual", + reason = opts.reason or "", + by_nick = opts.by_nick or "", + -- by_level = operator level at add time, for the Phase B + -- `+whitelist del` hierarchy check. nil for the seed. + by_level = tonumber( opts.by_level ), + expires_at = opts.expires_at, + created_at = math_floor( now ), + meta = opts.meta, + _network_b64 = _hex_encode( network_bytes ), + } + return entry +end + +local function _sweep_expired( now ) + local kept = { } + local removed_any = false + for _, e in ipairs( _entries ) do + if e.expires_at and e.expires_at <= now then + _bucket_remove( e ) + removed_any = true + else + kept[ #kept + 1 ] = e + end + end + if removed_any then _entries = kept end + return removed_any +end + +---------------------------------------------------------------------- +-- Lookup +---------------------------------------------------------------------- + +local _resolve_match + +local function is_whitelisted( ip ) + if not _enabled then return false end + if not ip then return false end + -- Fast path: an empty store is the common case for operators who + -- never add an allow entry - a single next() and we are out, no + -- parse, no bucket work. + if not next( _entries ) then return false end + return _resolve_match( ip ) ~= nil +end + +_resolve_match = function( ip ) + if type( ip ) ~= "string" or ip == "" then return nil end + local family, bytes = ipmatch_parse_ip( ip ) + if not family then return nil end + + -- v4-mapped-v6 normalisation, identical to core/blocklist.lua + -- _resolve_decision: a dual-stack listener delivers IPv4 clients + -- as `::ffff:a.b.c.d`, so re-key those to plain v4 or an + -- operator's v4 whitelist entry would never match a v4-over-v6 + -- client. Explicit `::ffff:.../128` v6 entries still work (they + -- were canonicalised to v6 bytes at add() time). + if family == 6 and #bytes == 16 then + local is_mapped = true + for i = 1, 10 do + if string_byte( bytes, i ) ~= 0 then is_mapped = false; break end + end + if is_mapped and string_byte( bytes, 11 ) == 0xff + and string_byte( bytes, 12 ) == 0xff then + family = 4 + bytes = string_sub( bytes, 13, 16 ) + end + end + + local bucket + if family == 4 then + bucket = _buckets_v4[ string_byte( bytes, 1 ) ] + else + local b1, b2 = string_byte( bytes, 1, 2 ) + bucket = _buckets_v6[ b1 * 256 + b2 ] + end + if not bucket then return nil end + + local now = socket_gettime( ) + for _, entry in ipairs( bucket ) do + if entry.expires_at and entry.expires_at <= now then + -- expired; filtered on every lookup, removed on next add/reload + elseif ipmatch_match( bytes, entry.network_bytes, entry.prefix_len ) then + -- No priority to resolve: first active match wins. + return entry + end + end + return nil +end + +---------------------------------------------------------------------- +-- Mutators +---------------------------------------------------------------------- + +local function add( cidr, opts ) + local entry, err = _make_entry( cidr, opts ) + if not entry then return false, nil, err end + _sweep_expired( socket_gettime( ) ) + entry._buckets = _bucket_for( entry.family, entry.network_bytes, entry.prefix_len ) + _next_id = _next_id + 1 + _entries[ #_entries + 1 ] = entry + _bucket_insert( entry ) + local ok, save_err = _save_to_disk( ) + if not ok then + _entries[ #_entries ] = nil + _next_id = _next_id - 1 + _bucket_remove( entry ) + return false, nil, "save failed: " .. tostring( save_err ) + end + return true, entry.id +end + +local function remove( id ) + if type( id ) ~= "number" then return false, "id must be a number" end + for i, e in ipairs( _entries ) do + if e.id == id then + table_remove( _entries, i ) + _bucket_remove( e ) + local ok, err = _save_to_disk( ) + if not ok then + table_insert( _entries, i, e ) + _bucket_insert( e ) + return false, "save failed: " .. tostring( err ) + end + return true + end + end + return false, "not_found" +end + +local _FILTER_KEYS = { source = true } + +local function list( filter_spec ) + local out = { } + filter_spec = filter_spec or { } + for k in pairs( filter_spec ) do + if not _FILTER_KEYS[ k ] then + if out_put then + out_put( "whitelist: list() ignoring unknown filter key '" .. + tostring( k ) .. "'" ) + end + end + end + for _, e in ipairs( _entries ) do + local include = true + if filter_spec.source and e.source ~= filter_spec.source then include = false end + if include then + local meta_copy + if type( e.meta ) == "table" then + meta_copy = { } + for k, v in pairs( e.meta ) do meta_copy[ k ] = v end + end + out[ #out + 1 ] = { + id = e.id, + cidr = e.cidr, + source = e.source, + reason = e.reason, + by_nick = e.by_nick, + by_level = e.by_level, + expires_at = e.expires_at, + created_at = e.created_at, + meta = meta_copy, + } + end + end + return out +end + +local function count( ) + local by_source = { } + for _, e in ipairs( _entries ) do + by_source[ e.source ] = ( by_source[ e.source ] or 0 ) + 1 + end + return { total = #_entries, by_source = by_source } +end + +---------------------------------------------------------------------- +-- Load / reload +---------------------------------------------------------------------- + +local function reload( ) + _entries = { } + _next_id = 1 + _buckets_v4 = { } + _buckets_v6 = { } + + -- Peek before util.loadtable to avoid the noisy checkfile error + -- line on a fresh install (mirrors blocklist.lua). Empty store is + -- the correct first-boot state; the Phase B plugin seeds the + -- bundled pinger allowlist on its first run, not here. + local probe = io_open( _store_path, "r" ) + if not probe then return end + probe:close( ) + + local data, err = util_loadtable( _store_path ) + if not data then + if err and not err:find( "No such file" ) then + if out_error then out_error( "whitelist: load failed: " .. tostring( err ) ) end + end + return + end + if type( data ) ~= "table" then return end + + local now = socket_gettime( ) + for _, row in ipairs( data ) do + local expired = row.expires_at and row.expires_at <= now + local network_bytes = _hex_decode( row.network_b64 or "" ) + if ( not expired ) and network_bytes and ( ( row.family == 4 and #network_bytes == 4 ) or ( row.family == 6 and #network_bytes == 16 ) ) then + local entry = { + id = tonumber( row.id ) or _next_id, + cidr = row.cidr, + family = row.family, + network_bytes = network_bytes, + prefix_len = tonumber( row.prefix_len ) or ( row.family == 4 and 32 or 128 ), + source = row.source or "manual", + reason = row.reason or "", + by_nick = row.by_nick or "", + by_level = tonumber( row.by_level ), + expires_at = row.expires_at, + created_at = row.created_at, + meta = row.meta, + _network_b64 = row.network_b64, + } + entry._buckets = _bucket_for( entry.family, entry.network_bytes, entry.prefix_len ) + _entries[ #_entries + 1 ] = entry + _bucket_insert( entry ) + if entry.id >= _next_id then _next_id = entry.id + 1 end + end + end +end + +---------------------------------------------------------------------- +-- Init - called from init.lua _core sweep +---------------------------------------------------------------------- + +local function _resnapshot_cfg( ) + local cfg_enabled = cfg_get "whitelist_enabled" + _enabled = ( cfg_enabled == nil ) or ( cfg_enabled and true ) or false + _store_path = cfg_get( "whitelist_store_path" ) or _store_path +end + +local function _reload_on_cfg_reload( ) + _resnapshot_cfg( ) + reload( ) +end + +local function init( ) + cfg_get = use( "cfg" ).get + local out_mod = use "out" + out_put = out_mod.put + out_error = out_mod.error + + _resnapshot_cfg( ) + reload( ) + + local cfg_mod = use "cfg" + if type( cfg_mod.registerevent ) == "function" then + cfg_mod.registerevent( "reload", _reload_on_cfg_reload ) + end +end + +return { + init = init, + is_whitelisted = is_whitelisted, + add = add, + remove = remove, + list = list, + count = count, + reload = reload, + _resolve_match = _resolve_match, + _hex_encode = _hex_encode, + _hex_decode = _hex_decode, +} diff --git a/docs/BLOCKLIST.md b/docs/BLOCKLIST.md index 61579e67..8766edea 100644 --- a/docs/BLOCKLIST.md +++ b/docs/BLOCKLIST.md @@ -7,6 +7,7 @@ luadch blocks unwanted connections in two complementary layers: | **Pre-handshake IP/CIDR blocklist** | TCP accept, before ADC/TLS | known-bad IPs and CIDR ranges (manual + future feeds) | `+blocklist` ([`SCRIPTS.md`](SCRIPTS.md) etc_blocklist) | | **Post-handshake bans** | after login | nick / CID / short-term IP bans | `+ban` (cmd_ban) | | **GeoIP policy** | on connect, post-handshake | country / ASN policy | `etc_geoip` (this doc) | +| **Allowlist (whitelist)** | consulted first by every automated blocker | trusted IPs / CIDRs (hublist pingers) exempted | `+whitelist` ([`SCRIPTS.md`](SCRIPTS.md) etc_whitelist) | The pre-handshake blocklist is your DoS/scanner shield (a hostile IP is dropped before it costs you a TLS handshake). GeoIP is *policy*: it @@ -15,6 +16,15 @@ kicks them post-handshake with a reason they can read - the same layer `+ban` and the client blocker use. This is deliberate; see the design note at the end. +The **allowlist** (`+whitelist`, the #78 allowlist) sits in front of all +of the automated layers: a whitelisted IP is exempt from GeoIP, proxy +detection, the external feeds, the hub-limit ban and the automated +blocklist-store entries - but NOT from a deliberate manual `+ban` / +`+blocklist add` (a manual block wins). A small set of known +hublist-pinger IPs is seeded by default so the pingers do not pollute +your log. It does NOT (yet) exempt from the share / slots / nick-policy +plugins. Full reference: [`SCRIPTS.md`](SCRIPTS.md) etc_whitelist. + --- ## GeoIP: country / ASN blocking (`etc_geoip`, #78 Phase D2) diff --git a/docs/BUILDING.md b/docs/BUILDING.md index c7cf26aa..a2993ae5 100644 --- a/docs/BUILDING.md +++ b/docs/BUILDING.md @@ -222,6 +222,30 @@ default account is hubowner — **delete it as soon as you have your own**. --- +## Single instance per install + +The hub refuses to start a second time from the **same install +directory**: the launcher holds an exclusive lock on `luadch.lock` in +that directory for its whole lifetime. A second `./luadch` / +`Luadch.exe` against the same tree exits non-zero with `another instance +is already running in this directory` (also written to +`log/exception.txt`). This stops two hubs racing on the shared +`cfg/user.tbl`, `master.key`, `scripts/data/*.tbl` and logs. + +- Running **several hubs on one machine** is fully supported - give each + its own install directory; each gets its own lock and they never + collide. +- The lock is released by the OS when the process exits or crashes, so + there is **no stale lock file to clean up** after a crash. +- `+reload` keeps the lock (it re-runs in place, it is not a new + process). +- If you script a restart, let the old process finish exiting before + starting the new one: during the old hub's shutdown drain the lock is + still held and the replacement is refused. `systemd` and + `docker restart` do this sequentially and are unaffected. + +--- + ## File permissions for secrets `cfg/user.tbl` (registered users with their cleartext passwords - see diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index eadba84f..c53e25fe 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -210,6 +210,19 @@ The conventions below are either not in it or are easy to get wrong: disabled per its default policy). Array order in `cfg.scripts` = listener-chain order; structural plugins (e.g. `hub_inf_manager`) must precede plugins that depend on their effect. +- **Auto-kick plugins consult the whitelist first (#78 allowlist).** Any plugin that + would auto-kick / ban / block trusted infrastructure - whether it decides on the + connecting IP (GeoIP, proxy detection, feed blocklists) or on a signal that trusted + infra legitimately trips (a hublist pinger's high hub count) - should call + `whitelist.is_whitelisted(user:ip())` at the top of its per-connection decision and + skip on a match, so operator-trusted infrastructure (hublist pingers etc.) is + exempt. `whitelist` is a sandbox global (mirrors `blocklist`). Precedence is + deliberate: the whitelist overrides AUTOMATED blocks only - a manual `+ban` / + `+blocklist` still applies (enforced in `core/blocklist.check_ip`). Put the guard + BEFORE any cache / quota / network step so a trusted IP costs nothing. Precedents: + the one-line guard in `etc_geoip`, `etc_proxydetect`, `usr_hubs`. Not yet extended + to the share / slots / nick-policy plugins (`usr_share` / `usr_slots` / + `usr_nick_*`) - a whitelisted IP still faces those unless a follow-up adds the guard. - **Audit fire-sites:** state-changing actions emit `audit.build` / `audit.fire` with the firstnick-canonical actor. See [`SECURITY.md`](SECURITY.md) and the `#84` audit-log conventions. @@ -292,6 +305,19 @@ passes on Linux + locally. Don't "fix" the code to satisfy a newer Lua - pin the CI back to 5.4.x. (Local dev uses a standalone Lua 5.4.8 = the same version.) +**Old-Windows hubowners are a real population (Server 2008 R2 / Windows 7).** +Two traps when supporting them: (1) the UCRT release build links the Universal +C Runtime (`api-ms-win-crt-*.dll`), absent there until **KB2999226** is +installed (symptom: a missing `api-ms-win-crt-*.dll` at startup). (2) Host-info +shell-outs (`core/sysinfo.lua`) must NOT rely on `Get-CimInstance` - it is +PowerShell 3.0+, and those OSes ship PowerShell 2.0. Query WMI with a +`powershell -Command "try { (Get-CimInstance X).P } catch { (Get-WmiObject X).P }"` +fallback: `Get-WmiObject` exists in every Windows PowerShell (2.0-5.1; +`powershell.exe`, not the PS-7/Core `pwsh`), so PS-2.0 hosts take the catch +branch. A nil result from such a probe must degrade to a sentinel +(`... or msg_unknown`), never reach a concatenation - the pre-refactor 3.1.x +`cmd_hubinfo` crashed exactly there (`attempt to concatenate a nil value`). + ### Restricted-env load check for a plugin A plugin unit test stubs the sandbox globals in `_G`, so it provides *every* diff --git a/docs/HTTP_API.md b/docs/HTTP_API.md index 66e33afd..7c5c0b03 100644 --- a/docs/HTTP_API.md +++ b/docs/HTTP_API.md @@ -1118,6 +1118,10 @@ is disabled in `cfg.scripts`, the endpoint returns 404 | GET | `/v1/blocklist/counts` | read | `etc_blocklist` (= ADC `+blocklist count`) - **landed (#78 Phase C)** [^http-blocklist-list-2] | | POST | `/v1/blocklist` | admin | `etc_blocklist` (= ADC `+blocklist add`) - **landed (#78 Phase C)** [^http-blocklist-list-3] | | DELETE | `/v1/blocklist/{id}` | admin | `etc_blocklist` (= ADC `+blocklist del`) - **landed (#78 Phase C)** [^http-blocklist-list-4] | +| GET | `/v1/whitelist` | read | `etc_whitelist` (= ADC `+whitelist show`) - **paginated** + **filter/sort** [^http-whitelist-1] | +| GET | `/v1/whitelist/counts` | read | `etc_whitelist` (= ADC `+whitelist count`) - **landed (#78 allowlist Phase D)** [^http-whitelist-2] | +| POST | `/v1/whitelist` | admin | `etc_whitelist` (= ADC `+whitelist add`) - **landed (#78 allowlist Phase D)** [^http-whitelist-3] | +| DELETE | `/v1/whitelist/{id}` | admin | `etc_whitelist` (= ADC `+whitelist del`) - **landed (#78 allowlist Phase D)** [^http-whitelist-4] | [^http-ban-1]: Returns `{ok:true, data:{bans:[entry, ...]}}`. Each entry carries `id` (1-based index into the live bans array - the same `{id}` accepted by DELETE), `nick`, `cid`, `hash`, `ip`, `reason`, `by_nick`, `by_level`, `ban_seconds`, `ban_start` (epoch seconds, hub clock), `remaining_seconds` (can be negative for not-yet-pruned-expired entries; pruning happens on `onConnect` of the banned user, not on a timer), and `expires_at` (ISO 8601 UTC, omitted when `remaining_seconds <= 0`). The `id` is NOT a stable surrogate key - it shifts every time a ban is removed (the underlying `bans` array is reindexed by `table.remove`). Operators / tooling MUST re-list before issuing a follow-up DELETE. @@ -1145,6 +1149,14 @@ is disabled in `cfg.scripts`, the endpoint returns 404 [^http-blocklist-list-4]: No request body. Path parameter `{id}` is the stable numeric id from GET /v1/blocklist entries. Returns 200 with `data: {action:"removed", id, removed:{cidr, source, reason, by_nick}}` per §7.1.1 so the operator's audit / undo flow has the snapshot of what was lifted. Returns **404 E_NOT_FOUND** if the id is not in the live store (idempotent 200 would mask the typo case); **400 E_BAD_INPUT** if `{id}` is not a positive integer. Unlike `/v1/bans/{id}` (which uses a 1-based array index that shifts on removal), blocklist ids are stable monotonic - GET-then-DELETE is safe against concurrent mutations of unrelated entries. The ADC-side hierarchy guard (a level-N operator cannot remove an entry added by level-(N+)) does NOT apply on the HTTP path: HTTP treats the token as master (level 100) so an admin token can undo ANY entry regardless of who added it. Operators wanting tighter control on HTTP-driven removes gate at the token-scope layer (revoking the token), not the entry-level layer. The unblock takes effect IMMEDIATELY (the engine's bucket cache is mutated in the same tick and re-persisted to `scripts/data/etc_blocklist.tbl` before the response returns). Emits `blocklist.remove` audit event with `meta: {id, cidr, source}` and the entry's original reason as the audit reason field (matching cmd_ban convention). +[^http-whitelist-1]: Returns `{ok:true, data:{entries:[entry, ...]}, pagination:{...}}`. Each entry carries `id` (stable monotonically-assigned integer, valid across removes; a re-add gets a fresh id), `cidr` (canonical form; host-bits-set CIDRs rejected at add), `source` (`manual` = operator add, `pinger` = bundled hublist-pinger seed - a LABEL only, no priority), `reason`, `by_nick`, `by_level` (nil for the seed), `expires_at` (unix ts, nil for permanent), `created_at`, `meta`. There is NO `stealth` field (the whitelist is allow-only). Supports http_filter query params `source=`, `cidr=`, `reason=`, `by_nick=`, `by_level[_min|_max]=`, `created_at[_min|_max]=`, `expires_at[_min|_max]=`; sort via `sort=` (default `id`, descending prefix `-`; sortable: `id`, `cidr`, `source`, `created_at`, `expires_at`). The ADC-side `etc_whitelist_oplevel` (default 80) does NOT apply on the HTTP path: the bearer token's `read` scope IS the authorisation gate. + +[^http-whitelist-2]: No request body. Returns 200 with `data: {total, by_source}` per §7.1 - `total` a flat integer, `by_source` a `{source: count}` map (empty `{}` on a store with no entries). Same auth as GET list (`read` scope). A fresh hub reports the bundled pinger seed (source `pinger`), so `total` is non-zero out of the box. + +[^http-whitelist-3]: Body `{cidr: string required (max 45; `1.2.3.4` = `/32`, `2001:db8::` = `/128`; host-bits-set CIDRs rejected - `1.2.3.4/24` -> error), source?: string (enum: manual / pinger; default `manual`), reason?: string (max 256, control-byte sanitised via `util.strip_control_bytes`), expires_at?: integer (unix ts, 1..2147483647; nil / absent for permanent; `0` rejected as a UX trap)}`. Returns **201 Created** with `data: {action:"added", id, cidr, source}` per §7.1.1. Error mapping: **400 E_BAD_INPUT** for malformed / host-bits-set / engine-rejected CIDR; **500 E_INTERNAL** for on-disk persistence failure. The ADC-side `etc_whitelist_oplevel` does NOT apply: the bearer token's `admin` scope IS the authorisation gate. HTTP-created entries are attributed `by_nick = `, `by_level = 100` (synthetic master) so a subsequent DELETE via the same token bypasses the hierarchy guard - the token IS the trust surface. Emits `whitelist.add` audit event with `meta: {cidr, source, id, by_level}`. Precedence reminder: a whitelisted IP overrides the AUTOMATED blockers (GeoIP / proxydetect / feeds / hub-limit + automated blocklist entries) but NOT a manual `+ban` / `+blocklist add` (enforced in `core/blocklist.check_ip`). + +[^http-whitelist-4]: No request body. Path parameter `{id}` is the stable numeric id from GET /v1/whitelist. Returns 200 with `data: {action:"removed", id, removed:{cidr, source, reason, by_nick}}` (control-byte sanitised) per §7.1.1 so the operator's audit / undo flow has the snapshot. Returns **404 E_NOT_FOUND** if the id is not in the live store; **400 E_BAD_INPUT** if `{id}` is not a positive integer. Whitelist ids are stable monotonic (GET-then-DELETE is safe against unrelated mutations). The ADC-side hierarchy guard does NOT apply on the HTTP path: HTTP treats the token as master (level 100) so an admin token can undo ANY entry regardless of who added it (including the bundled seed). Emits `whitelist.remove` audit event with `meta: {id, cidr, source}`. + #### Hub control | Method | Path | Scope | Plugin | diff --git a/docs/SCRIPTS.md b/docs/SCRIPTS.md index b8abf91b..ee658fb9 100644 --- a/docs/SCRIPTS.md +++ b/docs/SCRIPTS.md @@ -662,6 +662,77 @@ set `blocklist_store_path = "cfg/blocklist.tbl"` in `cfg.tbl` to pin the old location). No auto-migration is provided; the old-path window was small. +### etc_whitelist + +Operator-facing chat command for the global IP/CIDR allowlist (the +deferred #78 allowlist). Engine is `core/whitelist.lua` (Phase A); +this plugin ships the `+whitelist` admin CLI (Phase B) and the HTTP +API (Phase D). A whitelisted IP is exempt from the AUTOMATED +blockers - GeoIP, proxy detection, external feeds, the hub-limit ban +- and from automated blocklist-store entries, but NOT from a +deliberate manual `+ban` / `+blocklist add` (a manual block wins). To +block a whitelisted IP, remove it from the whitelist first. + +**Commands** (all require `etc_whitelist_oplevel`, default 80): + +- `+whitelist show [source]` - list active entries; optional filter + by source (`manual`, `pinger`). Capped at `etc_whitelist_show_limit` + (default 200). +- `+whitelist add [reason="..."] [expires=YYYY-MM-DD]` - + allow a CIDR (or single IP = `/32` v4 / `/128` v6). Host-bits-set + CIDRs are rejected (`1.2.3.4/24` -> use `1.2.3.0/24`). +- `+whitelist del ` - remove by numeric id; same `by_level` + hierarchy guard as `+blocklist`. +- `+whitelist count` - `{total, by_source}` summary. +- `+whitelist export` - write + `cfg/whitelist-export-YYYYMMDD-HHMMSS.jsonl` with the operator-added + (`source=manual`) entries. The bundled pinger seed is NOT exported + (it re-seeds itself on a fresh hub). +- `+whitelist import ` - read JSONL; every string field is + control-byte stripped; master-only via `etc_whitelist_import_min_level` + (default 100), rows reattributed to the importing operator. + +**Bundled pinger seed.** On the FIRST run (store `.tbl` missing) the +plugin seeds a small set of known hublist-pinger IPs as +`source=pinger` (v6 as `/64`, v4 as `/32`) so the pingers are not +flagged by the automated blockers out of the box. Seed-on-missing +ONLY - once the `.tbl` exists your edits (including deletions) are +authoritative and never re-seeded. Set `etc_whitelist_seed = false` +to boot with an empty whitelist. The list bit-rots (pinger IPs +rotate); review + extend via `+whitelist add`. A `/64` exempts a +whole subnet - each seeded range is meant to be a single per-VPS +pinger allocation; narrow to `/128` if that matters for your hub. + +**Precedence (which blocker wins).** whitelist > automated blockers +(GeoIP / proxydetect / feeds / hub-limit ban + automated +blocklist-store entries); a manual `+ban` / `+blocklist add` +(`source=manual`) > whitelist. Enforced in `core/blocklist.check_ip` +and in each blocker plugin's per-connection guard. NOT yet extended +to the share / slots / nick-policy plugins (`usr_share` / `usr_slots` +/ `usr_nick_*`) - a whitelisted IP still faces those. + +**Audit fires:** `whitelist.add` / `whitelist.remove` / +`whitelist.export` / `whitelist.import`. + +**Storage and reload.** `scripts/data/etc_whitelist.tbl`, written on +every add / remove; path via `whitelist_store_path`. `+reload` +re-snapshots `whitelist_enabled` / `whitelist_store_path` and +re-reads the .tbl (config-only changes need no restart). + +**HTTP API surface (v0.02 / Phase D).** Four endpoints mirroring the +blocklist API (in `docs/HTTP_API.md`): `GET /v1/whitelist` (read, +filter/sort/paginate), `GET /v1/whitelist/counts` (read), `POST +/v1/whitelist` (admin, body `{cidr, source?, reason?, expires_at?}`, +source enum `{manual, pinger}`), `DELETE /v1/whitelist/{id}` (admin). +HTTP admin tokens bypass the ADC hierarchy guard (token = trust +surface; entries attributed `by_nick = `, `by_level = 100`). + +**Enabling.** Ships enabled in `examples/cfg/cfg.tbl`. An operator +upgrading a pre-existing `cfg.tbl` adds +`{ "etc_whitelist.lua", enabled = true },` to `cfg.scripts` and runs +`+reload`; the cfg keys all have safe defaults via +`core/cfg_defaults.lua`. + ### etc_geoip Country / ASN policy blocking via a MaxMind GeoLite2 database (#78 diff --git a/examples/cfg/cfg.tbl b/examples/cfg/cfg.tbl index 85f357e5..9ab07b59 100755 --- a/examples/cfg/cfg.tbl +++ b/examples/cfg/cfg.tbl @@ -1011,6 +1011,18 @@ return { -- the settings are a lua table, do not tough this! etc_blocklist_llevel = 60, -- min level for the hubbot-PM report (integer) etc_blocklist_import_min_level = 100, -- min level to run `+blocklist import ` - JSONL files may carry entries added by higher-level masters; default 100 = master-only so a mid-level op cannot import + take ownership of master-tier entries (integer) + --------------------------------------------------------------------------------------------------------------------------------- + --// etc_whitelist.lua settings | `+whitelist` admin CLI for the global IP/CIDR allowlist (#78 allowlist, Phase B). Exempts trusted IPs from the automated blockers, NOT from a manual ban + + etc_whitelist_oplevel = 80, -- min level to run `+whitelist add / del / show / count / export / import` (integer) + etc_whitelist_show_limit = 200, -- hard cap on rows in `+whitelist show` output (integer 1..10000) + etc_whitelist_seed = true, -- seed the bundled known-hublist-pinger allowlist on the FIRST run (store .tbl missing); seed-on-missing only, never overwrites your edits. false = boot with an empty whitelist (boolean) + etc_whitelist_report = true, -- on every add / del / export / import, fire etc_report.send with the action summary so staff see it live (boolean) + etc_whitelist_report_hubbot = false, -- send the report as a hubbot PM to operators at or above etc_whitelist_llevel (boolean) + etc_whitelist_report_opchat = true, -- send the report into op-chat (boolean) + etc_whitelist_llevel = 60, -- min level for the hubbot-PM report (integer) + etc_whitelist_import_min_level = 100, -- min level to run `+whitelist import ` - same threat model as blocklist (JSONL may carry master-added rows); default 100 = master-only (integer) + --------------------------------------------------------------------------------------------------------------------------------- --// etc_geoip.lua settings | country / ASN blocking via MaxMind GeoLite2 (#78 Phase D2). Needs geoipupdate for the DB - see docs/BLOCKLIST.md @@ -1557,6 +1569,7 @@ return { -- the settings are a lua table, do not tough this! "hub_inf_manager.lua", { "etc_clientblocker.lua", enabled = true }, -- #81 client AP/VE blocklist. API-toggleable (#261). Must sit AFTER hub_inf_manager so structural BINF rejects (forbidden flags / identity spoofing) happen first; client-policy kicks then run on already-validated INFs. { "etc_blocklist.lua", enabled = true }, -- #78 Phase B `+blocklist` admin CLI for the unified pre-handshake IP/CIDR blocklist (engine = core/blocklist.lua, Phase A). API-toggleable (#261). Manages MANUAL entries; Phase D/E/F plugins (geoip / external feeds / proxydetect) will push auto-feed entries into the same store. + { "etc_whitelist.lua", enabled = true }, -- #78 allowlist, Phase B `+whitelist` admin CLI for the global IP/CIDR allowlist (engine = core/whitelist.lua). API-toggleable (#261). Exempts trusted IPs (hublist pingers, seeded on first run) from the automated blockers; a manual +ban still wins. Chain position not load-bearing (consulted by the blockers, not a listener itself). { "etc_geoip.lua", enabled = false }, -- #78 Phase D2 country / ASN blocking via MaxMind GeoLite2. OFF by default (needs a geoipupdate-managed DB - see docs/BLOCKLIST.md). Per-connection post-handshake lookup; must sit AFTER hub_inf_manager like etc_clientblocker. Enable here + set etc_geoip_enabled=true + populate etc_geoip_blocked_countries. { "etc_blocklist_feeds.lua", enabled = false }, -- #78 Phase E external IP/CIDR feeds (Tor exits, Spamhaus DROP) pulled over HTTPS into the blocklist store. OFF by default. Store WRITER (the engine blocks pre-handshake), so chain position is not load-bearing. Enable here + set etc_blocklist_feeds_enabled=true + turn on a feed - see docs/BLOCKLIST.md. { "etc_proxydetect.lua", enabled = false }, -- #78 Phase F live proxy/VPN/Tor detection via a provider API on connect. OFF by default (needs a provider + API key - see docs/BLOCKLIST.md). Per-connection post-handshake lookup; must sit AFTER hub_inf_manager like etc_geoip. Enable here + set etc_proxydetect_enabled=true + etc_proxydetect_action="block". diff --git a/hub/hub.c b/hub/hub.c index c64437fc..a63f6ec8 100755 --- a/hub/hub.c +++ b/hub/hub.c @@ -11,6 +11,7 @@ #ifdef __unix__ #include #include +#include /* flock */ #include #include #include @@ -342,6 +343,147 @@ static void handle_args(int argc, char **argv) } } +/* ---- single-instance lock ------------------------------------------- + * + * Refuse to start a second hub against the same install tree. Two hubs + * sharing one cfg/ race on user.tbl, master.key, the plugin .tbl stores + * and the logs; interleaved saveusers()/savetable() writes can corrupt + * them (data loss). The guard is a lock FILE in the install directory - + * chdir_to_binary_dir() anchored the CWD there, so a relative name + * resolves per-install automatically: a second hub in a DIFFERENT + * directory has its own lock and starts fine (a test hub + a prod hub on + * one box is a supported setup). The lock is held open for the whole + * process lifetime: + * - unix: flock(LOCK_EX | LOCK_NB). The kernel releases it when the + * process dies, so a crash never leaves a blocking stale + * lock. fork() shares the open file description, so the + * daemon child keeps the lock after the parent exits. + * - windows: CreateFile with dwShareMode = 0 (deny sharing) - a second + * open fails with ERROR_SHARING_VIOLATION. FILE_FLAG_DELETE_ + * ON_CLOSE removes the file on exit/crash. + * + * Fail-open: if the lock file cannot even be created (read-only fs and + * the like) we warn and continue rather than brick the hub over a + * missing convenience guard - only a definitive "already held" refuses + * startup. Survives +reload: restart() re-enters run_lua via atexit + * without closing this fd/handle, so the lock persists with no gap. */ +#define LOCKFILE_NAME "luadch.lock" + +#ifdef _WIN32 +static HANDLE g_lock_handle = INVALID_HANDLE_VALUE; +#elif defined(__unix__) +static int g_lock_fd = -1; +#endif + +/* 0 = we hold the lock (or the guard degraded to a no-op; continue). + * 1 = another instance already holds it (caller must abort startup). */ +static int acquire_single_instance_lock(void) +{ +#ifdef _WIN32 + HANDLE h = CreateFileA( + LOCKFILE_NAME, + GENERIC_READ | GENERIC_WRITE, + 0, /* dwShareMode = 0 -> exclusive */ + NULL, + OPEN_ALWAYS, + FILE_ATTRIBUTE_NORMAL | FILE_FLAG_DELETE_ON_CLOSE, + NULL); + if (h == INVALID_HANDLE_VALUE) + { + DWORD e = GetLastError(); + /* A concurrent instance holding the file with dwShareMode = 0 ALWAYS + * surfaces as ERROR_SHARING_VIOLATION - that is the only definitive + * "already running" signal. ERROR_ACCESS_DENIED means something else + * (a read-only or ACL-denied stale lock file, a directory of that + * name) with no instance necessarily running, so it must fail-open + * rather than wrongly refuse startup. */ + if (e == ERROR_SHARING_VIOLATION) + { + return 1; /* another instance holds the exclusive handle */ + } + fprintf(stderr, "warning: cannot create lock file '%s' (error %lu); " + "single-instance guard disabled\n", + LOCKFILE_NAME, (unsigned long)e); + fflush(stderr); + return 0; /* fail-open */ + } + g_lock_handle = h; + return 0; +#elif defined(__unix__) + /* O_CLOEXEC so a fork+exec child (io.popen / os.execute) does NOT + * inherit this fd: an inherited fd that outlives the hub would keep the + * flock held and wrongly refuse the next restart. daemonize() forks + * WITHOUT exec, so the daemon child still shares the open file + * description and correctly retains the lock. */ + int fd = open(LOCKFILE_NAME, O_CREAT | O_RDWR | O_CLOEXEC, 0640); + if (fd < 0) + { + fprintf(stderr, "warning: cannot create lock file '%s' (%s); " + "single-instance guard disabled\n", + LOCKFILE_NAME, strerror(errno)); + fflush(stderr); + return 0; /* fail-open */ + } + if (flock(fd, LOCK_EX | LOCK_NB) != 0) + { + if (errno == EWOULDBLOCK) + { + close(fd); + return 1; /* another instance holds the lock */ + } + fprintf(stderr, "warning: cannot lock '%s' (%s); " + "single-instance guard disabled\n", + LOCKFILE_NAME, strerror(errno)); + fflush(stderr); + close(fd); + return 0; /* fail-open */ + } + g_lock_fd = fd; + return 0; +#else + return 0; /* unknown platform: no guard */ +#endif +} + +/* Record the running process's PID in the (already-held) lock file for + * operator diagnostics. Called AFTER daemonize() so the value is the + * surviving process, not the pre-fork parent. Best-effort: a write + * failure is not fatal - the lock is the open handle, not the content. */ +static void write_lock_pid(void) +{ + char buf[32]; + int n; +#ifdef _WIN32 + if (g_lock_handle == INVALID_HANDLE_VALUE) return; + n = snprintf(buf, sizeof(buf), "%lu\n", + (unsigned long)GetCurrentProcessId()); + if (n > 0) + { + DWORD written = 0; + if (n >= (int)sizeof(buf)) n = (int)sizeof(buf) - 1; /* clamp */ + SetFilePointer(g_lock_handle, 0, NULL, FILE_BEGIN); + WriteFile(g_lock_handle, buf, (DWORD)n, &written, NULL); + SetEndOfFile(g_lock_handle); /* truncate: OPEN_ALWAYS may reopen a + * survived-hard-crash file with a + * longer stale PID */ + } +#elif defined(__unix__) + if (g_lock_fd < 0) return; + n = snprintf(buf, sizeof(buf), "%ld\n", (long)getpid()); + if (n > 0 && lseek(g_lock_fd, 0, SEEK_SET) == 0) + { + ssize_t w; + if (n >= (int)sizeof(buf)) n = (int)sizeof(buf) - 1; /* clamp */ + w = write(g_lock_fd, buf, (size_t)n); + if (w > 0) + { + int t = ftruncate(g_lock_fd, w); + (void)t; + } + } +#endif +} + int main(int argc, char **argv) { handle_signals(); @@ -354,7 +496,17 @@ int main(int argc, char **argv) fprintf(stderr, "warning: could not anchor cwd to binary directory; " "falling back to inherited cwd\n"); } + // Acquire BEFORE daemonize() so a foreground start reports the refusal + // straight to the invoking shell (non-zero exit), and a second `-d` + // start is rejected before it forks. + if (acquire_single_instance_lock() != 0) + { + log_error("luadch: another instance is already running in this " + "directory; refusing to start."); + return EXIT_FAILURE; + } daemonize(); + write_lock_pid(); run_lua(); return EXIT_SUCCESS; } diff --git a/scripts/etc_geoip.lua b/scripts/etc_geoip.lua index e7884e19..27529d6f 100644 --- a/scripts/etc_geoip.lua +++ b/scripts/etc_geoip.lua @@ -291,6 +291,10 @@ local check_geoip = function( user ) local ip = user:ip( ) if not ip or ip == "" then return end + -- #78 allowlist: a whitelisted IP (trusted infra / hublist pinger) + -- is exempt from the country/ASN block and its report + audit. + if whitelist.is_whitelisted( ip ) then return end + local country, asn, org = classify( ip ) local matched = resolve( country, asn ) if not matched then return end -- allowed diff --git a/scripts/etc_proxydetect.lua b/scripts/etc_proxydetect.lua index d900fb1b..d63de468 100644 --- a/scripts/etc_proxydetect.lua +++ b/scripts/etc_proxydetect.lua @@ -685,6 +685,12 @@ local check_proxydetect = function( user ) local ip, family = query_ip( user:ip( ) ) if not ip then return end -- unparseable IP -> skip (fail-open) + -- #78 allowlist: a whitelisted IP (trusted infra / hublist pinger) + -- is never sent to the provider (saves the paid query), cached, or + -- kicked. Checked before the cache + quota so a trusted IP costs + -- nothing. + if whitelist.is_whitelisted( ip ) then return end + -- Cache hit: decide against the LIVE block_types (policy changes -- take effect without waiting for expiry). local c = cache_get( ip ) diff --git a/scripts/etc_whitelist.lua b/scripts/etc_whitelist.lua new file mode 100644 index 00000000..d539577a --- /dev/null +++ b/scripts/etc_whitelist.lua @@ -0,0 +1,902 @@ +--[[ + + etc_whitelist.lua v0.01 by Aybo (#78 allowlist, Phase B) + + Operator-facing chat command for the global IP/CIDR allowlist + (`core/whitelist.lua`, shipped in Phase A). A whitelisted IP is + exempt from the AUTOMATED blockers (GeoIP / proxydetect / feeds / + hub-limit) and from automated blocklist-store entries, but NOT + from a deliberate manual `+ban` / `+blocklist` (a manual block + wins - see core/whitelist.lua). Six verbs under one command, plus + JSONL export / import for backup and cross-hub sync: + + +whitelist show [source] list active entries (optional + filter by source: manual / + pinger) + +whitelist add [reason="..."] [expires=YYYY-MM-DD] + allow a CIDR (or single IP = + /32 v4 or /128 v6). reason + + expires are key=value pairs + with quoted-string values. + +whitelist del remove entry by numeric id + from the +whitelist show output + +whitelist count {total, by_source} summary + +whitelist export write cfg/whitelist-export- + YYYYMMDD-HHMMSS.jsonl with the + operator-added (source=manual) + entries; the bundled pinger seed + re-seeds itself on a fresh hub, + so it is not exported + +whitelist import read JSONL, source="manual" + unless a row sets it; control- + byte stripping on every field + + Bundled pinger seed: on the FIRST run (the store .tbl missing on + disk) this plugin seeds a small set of known hublist-pinger IPs + as source="pinger" so they are not flagged by the automated + blockers out of the box. Seed-on-missing only - once the .tbl + exists the operator's edits (incl. deletions) are authoritative + and never re-seeded. Disable the seed entirely with + `etc_whitelist_seed = false`. The list bit-rots (pinger IPs + rotate); review + extend via `+whitelist add`. + + Hierarchy guard: a level-N operator cannot remove an entry added + by a level-(N+) master. The entry's `by_level` field, captured at + add time, is the comparison source. The bundled seed has no + by_level (= 0), so any operator can prune it. + + Phase D (v0.02) adds the HTTP API alongside the ADC surface: + GET /v1/whitelist (list + filter/sort/paginate), GET + /v1/whitelist/counts (summary), POST /v1/whitelist (add), + DELETE /v1/whitelist/{id} (remove). HTTP admin tokens bypass the + ADC hierarchy guard - the token is the trust surface, so a token + can undo any entry regardless of who added it. + +]]-- + + +-------------- +--[SETTINGS]-- +-------------- + +local scriptname = "etc_whitelist" +local scriptversion = "0.02" + +local cmd_main = "whitelist" + + +--// imports +local scriptlang = cfg.get( "language" ) +local lang, lang_err = cfg.loadlanguage( scriptlang, scriptname ) +lang = lang or { } +if lang_err then hub.debug( lang_err ) end + +local oplevel = cfg.get( "etc_whitelist_oplevel" ) or 80 +local show_limit = cfg.get( "etc_whitelist_show_limit" ) or 200 +local import_min_level = cfg.get( "etc_whitelist_import_min_level" ) or 100 +local seed_enabled = cfg.get( "etc_whitelist_seed" ) +if seed_enabled == nil then seed_enabled = true end + +local report_activate = cfg.get( "etc_whitelist_report" ) +local report_hubbot = cfg.get( "etc_whitelist_report_hubbot" ) +local report_opchat = cfg.get( "etc_whitelist_report_opchat" ) +local report_llevel = cfg.get( "etc_whitelist_llevel" ) + +-- Store path: core-owned cfg key. Used to detect first-run (file +-- missing) for the bundled-seed decision, mirroring the engine default. +local store_path = cfg.get( "whitelist_store_path" ) or "scripts/data/etc_whitelist.tbl" + +local report = hub.import( "etc_report" ) + + +--// table lookups +local hub_getbot = hub.getbot +local hub_import = hub.import +local hub_debug = hub.debug +local util_strip = util.strip_control_bytes +local util_safe_path = util.safe_path +local utf_match = utf.match +local utf_format = utf.format +local table_concat = table.concat +local string_format = string.format + +--// dkjson: required for JSONL export/import; the plugin still loads +--// without it and the JSONL verbs degrade to an error reply. +local json = dkjson + + +--// Bundled hublist-pinger allowlist. Seeded on first run only (see +--// header). v6 pingers use a /64 because the host rotates within the +--// range; v4 pingers use an exact /32. Observed on a live 3.2.x hub +--// 2026-07-13; extend via +whitelist add. These are exempt from the +--// automated blockers, never from a manual +ban. +--// +--// Breadth caveat: a /64 exempts a whole subnet. Each range below is +--// meant to be a per-VPS pinger allocation (OVH etc. hand out a /64 +--// per host); if a provider ever shares a /64 across tenants, the +--// co-tenants inherit the automated-blocker exemption too (still never +--// a manual-ban exemption). Narrow a range to /128 via +whitelist, or +--// set etc_whitelist_seed=false to skip the seed entirely. +local BUNDLED_SEED = { + { cidr = "2001:41d0:a:f8b3::/64", reason = "DCpinger (hublist)" }, + { cidr = "2607:5300:201:3000::/64", reason = "PingerDC (hublist)" }, + { cidr = "2602:fed2:731b:25::/64", reason = "[ADC]Proxima (hublist)" }, + { cidr = "78.40.117.229", reason = "HublistPinger" }, + { cidr = "142.54.190.133", reason = "PWiAM_Pinger" }, + { cidr = "5.252.102.106", reason = "TEPinger" }, +} + + +--// lang +local help_title = lang.help_title or "etc_whitelist.lua - allowlist" +local help_usage = lang.help_usage or "[+!#]whitelist show|add|del|count|export|import ..." +local help_desc = lang.help_desc or "Manage the global IP/CIDR allowlist (exempts trusted IPs from the automated blockers, not from a manual ban). Run `+whitelist show`." + +local ucmd_menu_show = lang.ucmd_menu_show or { "Hub", "Whitelist", "show" } +local ucmd_menu_add = lang.ucmd_menu_add or { "Hub", "Whitelist", "add" } +local ucmd_menu_del = lang.ucmd_menu_del or { "Hub", "Whitelist", "remove by id" } +local ucmd_menu_count = lang.ucmd_menu_count or { "Hub", "Whitelist", "count by source" } +local ucmd_menu_export = lang.ucmd_menu_export or { "Hub", "Whitelist", "export to JSONL" } + +local ucmd_popup_cidr = lang.ucmd_popup_cidr or "CIDR or IP (e.g. 192.0.2.0/24):" +local ucmd_popup_reason = lang.ucmd_popup_reason or "Reason (optional):" +local ucmd_popup_id = lang.ucmd_popup_id or "Entry id from `+whitelist show`:" + +local msg_denied = lang.msg_denied or "You are not allowed to use this command." +local msg_usage = lang.msg_usage or "Usage: +whitelist show|add|del|count|export|import ..." +local msg_usage_add = lang.msg_usage_add or 'Usage: +whitelist add [reason="..."] [expires=YYYY-MM-DD]' +local msg_usage_del = lang.msg_usage_del or "Usage: +whitelist del " +local msg_usage_import = lang.msg_usage_import or "Usage: +whitelist import " +local msg_unknown_verb = lang.msg_unknown_verb or "Unknown verb '%s'. Try: show, add, del, count, export, import." +local msg_bad_cidr = lang.msg_bad_cidr or "Invalid CIDR / IP: %s" +local msg_bad_expires = lang.msg_bad_expires or "Invalid expires date '%s'. Expected YYYY-MM-DD." +local msg_added = lang.msg_added or "%s added whitelist entry #%d (%s, source=%s)." +local msg_add_failed = lang.msg_add_failed or "whitelist.add failed: %s" +local msg_save_failed = lang.msg_save_failed or "Failed to persist whitelist: %s" +local msg_removed = lang.msg_removed or "%s removed whitelist entry #%d (%s, source=%s)." +local msg_remove_failed = lang.msg_remove_failed or "whitelist.del failed: %s" +local msg_not_found = lang.msg_not_found or "No whitelist entry with id #%d." +local msg_hierarchy = lang.msg_hierarchy or "Cannot remove entry #%d: it was added by a higher-level operator (you=%d, by=%d)." +local msg_show_header = lang.msg_show_header or "\n=== WHITELIST ===" +local msg_show_footer = lang.msg_show_footer or "=== END ===\n" +local msg_show_empty = lang.msg_show_empty or "(no entries)" +local msg_show_filter = lang.msg_show_filter or "(filtered: source=%s)" +local msg_show_capped = lang.msg_show_capped or "(showing %d of %d entries; raise etc_whitelist_show_limit or filter by source)" +local msg_count = lang.msg_count or "whitelist: %d entries total" +local msg_no_dkjson = lang.msg_no_dkjson or "JSONL export/import requires dkjson, which is not available." +local msg_export_ok = lang.msg_export_ok or "%s exported %d whitelist entries to %s." +local msg_export_fail = lang.msg_export_fail or "whitelist.export failed: %s" +local msg_import_ok = lang.msg_import_ok or "%s imported %d entries from %s (%d skipped, %d errors)." +local msg_import_fail = lang.msg_import_fail or "whitelist.import failed: %s" +local msg_unsafe_path = lang.msg_unsafe_path or "Path '%s' is unsafe: %s" +local msg_import_level = lang.msg_import_level or "Import requires level %d or higher (you are %d)." + + +---------- +--[CODE]-- +---------- + +-- Parse `+whitelist add` args. Grammar: +-- [reason="..."] [expires=YYYY-MM-DD] +-- Returns: { cidr, reason, expires } | nil, err_msg +local function parse_add_args( parameters ) + if type( parameters ) ~= "string" or parameters == "" then + return nil, msg_usage_add + end + local cidr, rest = utf_match( parameters, "^(%S+)%s*(.*)$" ) + if not cidr or cidr == "" then + return nil, msg_usage_add + end + rest = rest or "" + + -- Extract QUOTED values first against a working copy that we strip + -- as we go, so a smuggled `expires=...` literal inside the quoted + -- reason cannot influence the parsed expires (and vice versa). + local working = rest + local reason = utf_match( working, 'reason=%"([^%"]*)%"' ) + if reason then + working = ( working:gsub( 'reason=%"[^%"]*%"', "" ) ) + end + local expires = utf_match( working, 'expires=%"([^%"]*)%"' ) + if expires then + working = ( working:gsub( 'expires=%"[^%"]*%"', "" ) ) + end + reason = reason or utf_match( working, "reason=(%S+)" ) + expires = expires or utf_match( working, "expires=(%S+)" ) + + return { cidr = cidr, reason = reason, expires = expires } +end + + +-- Parse YYYY-MM-DD into an end-of-day (23:59:59 local) unix timestamp +-- so "expires=2026-12-31" means "allowed through end of 2026". +local function parse_expires_date( s ) + if type( s ) ~= "string" or s == "" then return nil end + local y, m, d = s:match( "^(%d%d%d%d)%-(%d%d)%-(%d%d)$" ) + if not y then return nil end + return os.time{ + year = tonumber( y ), month = tonumber( m ), day = tonumber( d ), + hour = 23, min = 59, sec = 59, + } +end + + +-- The "add" action. source is always "manual" for an operator add. +-- Returns: ok=true, id, msg | ok=false, err_code, msg +local function do_add_entry( cidr, opts, actor_label, actor_level ) + opts = opts or { } + local reason = opts.reason + local expires_at = nil + if opts.expires then + expires_at = parse_expires_date( opts.expires ) + if not expires_at then + return false, "bad_expires", utf_format( msg_bad_expires, opts.expires ) + end + end + + local ok, id, err = whitelist.add( cidr, { + source = "manual", + reason = reason, + by_nick = actor_label, + by_level = actor_level, + expires_at = expires_at, + } ) + if not ok then + local err_s = tostring( err or cidr ) + if err_s:find( "^save failed" ) then + return false, "save_failed", utf_format( msg_save_failed, err_s ) + end + return false, "bad_cidr", utf_format( msg_bad_cidr, err_s ) + end + return true, id, utf_format( msg_added, actor_label or "?", id, cidr, "manual" ) +end + + +-- The "del" action. Hierarchy guard applied here. +local function do_del_entry( id, actor_label, actor_level ) + if type( id ) ~= "number" or id < 1 then + return false, "bad_id", msg_usage_del + end + local rows = whitelist.list( ) + local found + for _, e in ipairs( rows ) do + if e.id == id then found = e; break end + end + if not found then + return false, "not_found", utf_format( msg_not_found, id ) + end + local by_level = tonumber( found.by_level ) or 0 + if actor_level and actor_level < by_level then + return false, "hierarchy", utf_format( msg_hierarchy, id, actor_level, by_level ) + end + local ok, rerr = whitelist.remove( id ) + if not ok then + return false, "remove_failed", utf_format( msg_remove_failed, tostring( rerr ) ) + end + return true, found, utf_format( msg_removed, actor_label or "?", id, + found.cidr, found.source ) +end + + +local function _export_path_for( now_ts ) + return "cfg/whitelist-export-" .. os.date( "%Y%m%d-%H%M%S", now_ts ) .. ".jsonl" +end + + +-- Export operator-added (source=manual) entries to JSONL. The bundled +-- pinger seed is deliberately NOT exported: it re-seeds itself on a +-- fresh hub, and exporting it would double the seed on re-import. +-- Returns: ok=true, count, path, msg | ok=false, err_msg +local function do_export_jsonl( actor_label ) + if not json then return false, msg_no_dkjson end + local now_ts = os.time( ) + local path = _export_path_for( now_ts ) + local safe_ok, safe_err = util_safe_path( path ) + if not safe_ok then + return false, utf_format( msg_unsafe_path, path, tostring( safe_err ) ) + end + local f, ferr = io.open( path, "w" ) + if not f then + return false, "open failed: " .. tostring( ferr ) + end + local count = 0 + local rows = whitelist.list( { source = "manual" } ) + for _, e in ipairs( rows ) do + local skip = e.expires_at and e.expires_at <= now_ts + if not skip then + local line, jerr = json.encode{ + cidr = e.cidr, + source = e.source, + reason = e.reason, + by_nick = e.by_nick, + by_level = e.by_level, + expires_at = e.expires_at, + created_at = e.created_at, + } + if not line then + f:close( ) + return false, "json.encode failed: " .. tostring( jerr ) + end + f:write( line, "\n" ) + count = count + 1 + end + end + f:close( ) + return true, count, path, utf_format( msg_export_ok, actor_label or "?", count, path ) +end + + +-- Sanitize an imported row: strip control bytes off every string +-- field. Returns cidr, opts | nil, err. +local function _sanitize_import_row( row ) + if type( row ) ~= "table" then return nil, "row is not a table" end + local cidr = type( row.cidr ) == "string" and util_strip( row.cidr ) or nil + if not cidr or cidr == "" then return nil, "row.cidr missing" end + local source = "manual" + if type( row.source ) == "string" and row.source ~= "" then + source = util_strip( row.source ) + end + local opts = { + source = source, + reason = type( row.reason ) == "string" and util_strip( row.reason ) or nil, + by_nick = type( row.by_nick ) == "string" and util_strip( row.by_nick ) or nil, + by_level = tonumber( row.by_level ), + expires_at = tonumber( row.expires_at ), + } + return cidr, opts +end + + +-- Import JSONL from a file. Invalid rows are skipped + counted. +-- Every imported row is REATTRIBUTED to the importing operator's +-- nick + level (the engine's add() does not enforce hierarchy on +-- insert); `etc_whitelist_import_min_level` (default 100) gates who +-- may import so a mid-tier op cannot import + own master-tier rows. +-- Returns: ok=true, {added, skipped, errors}, msg | ok=false, err_msg +local function do_import_jsonl( path, actor_label, actor_level ) + if not json then return false, msg_no_dkjson end + if type( path ) ~= "string" or path == "" then + return false, msg_usage_import + end + if ( actor_level or 0 ) < import_min_level then + return false, utf_format( msg_import_level, import_min_level, actor_level or 0 ) + end + local safe_ok, safe_err = util_safe_path( path ) + if not safe_ok then + return false, utf_format( msg_unsafe_path, path, tostring( safe_err ) ) + end + local f, ferr = io.open( path, "r" ) + if not f then + return false, "open failed: " .. tostring( ferr ) + end + local added, skipped, errors = 0, 0, 0 + local error_log_cap = 5 + local errors_logged = 0 + local function _import_diag( msg ) + if errors_logged < error_log_cap then + hub_debug( scriptname .. " import: " .. msg ) + errors_logged = errors_logged + 1 + end + end + while true do + local line = f:read( "*l" ) + if not line then break end + line = line:gsub( "^\239\187\191", "" ):gsub( "^%s+", "" ):gsub( "%s+$", "" ) + if line ~= "" then + local row, _, jerr = json.decode( line ) + if not row then + errors = errors + 1 + _import_diag( "json.decode failed: " .. tostring( jerr ) ) + else + local cidr, opts_or_err = _sanitize_import_row( row ) + if not cidr then + skipped = skipped + 1 + _import_diag( "row skipped: " .. tostring( opts_or_err ) ) + else + local opts = opts_or_err + opts.by_nick = actor_label + opts.by_level = actor_level + local ok, _id, aerr = whitelist.add( cidr, opts ) + if ok then + added = added + 1 + else + errors = errors + 1 + _import_diag( "engine rejected " .. tostring( cidr ) .. + ": " .. tostring( aerr ) ) + end + end + end + end + end + f:close( ) + return true, { added = added, skipped = skipped, errors = errors }, + utf_format( msg_import_ok, actor_label or "?", added, path, skipped, errors ) +end + + +-- Build the `+whitelist show` body. +local function format_show( source_filter ) + local filter_spec = { } + if source_filter and source_filter ~= "" then + filter_spec.source = source_filter + end + local rows = whitelist.list( filter_spec ) + local total = #rows + local lines = { msg_show_header, "" } + if source_filter and source_filter ~= "" then + lines[ #lines + 1 ] = " " .. utf_format( msg_show_filter, source_filter ) + end + if total == 0 then + lines[ #lines + 1 ] = " " .. msg_show_empty + else + local cap = math.min( total, show_limit ) + for i = 1, cap do + local e = rows[ i ] + local reason_part = ( e.reason and e.reason ~= "" ) and ( " - " .. e.reason ) or "" + local exp_part = e.expires_at + and ( " (expires " .. tostring( os.date( "%Y-%m-%d", e.expires_at ) ) .. ")" ) + or "" + lines[ #lines + 1 ] = string_format( " #%d %s [src=%s] by=%s/L%s%s%s", + e.id, e.cidr or "?", e.source or "?", + e.by_nick or "?", tostring( e.by_level or "?" ), + exp_part, reason_part ) + end + if total > cap then + lines[ #lines + 1 ] = "" + lines[ #lines + 1 ] = " " .. utf_format( msg_show_capped, cap, total ) + end + end + lines[ #lines + 1 ] = "" + lines[ #lines + 1 ] = msg_show_footer + return table_concat( lines, "\n" ) +end + + +local function format_count( ) + local c = whitelist.count( ) + local lines = { msg_show_header, "" } + lines[ #lines + 1 ] = " " .. utf_format( msg_count, c.total ) + local sources = { } + for s in pairs( c.by_source ) do sources[ #sources + 1 ] = s end + table.sort( sources ) + for _, s in ipairs( sources ) do + lines[ #lines + 1 ] = string_format( " %s: %d", s, c.by_source[ s ] ) + end + lines[ #lines + 1 ] = "" + lines[ #lines + 1 ] = msg_show_footer + return table_concat( lines, "\n" ) +end + + +-- Seed the bundled pingers when the store file is MISSING (first run). +-- Returns the number of entries seeded. Exposed for tests. +local function seed_if_first_run( ) + if not seed_enabled then return 0 end + local probe = io.open( store_path, "r" ) + if probe then + probe:close( ) + return 0 -- store exists -> operator-owned, never re-seed + end + local seeded = 0 + for _, s in ipairs( BUNDLED_SEED ) do + local ok = whitelist.add( s.cidr, { source = "pinger", reason = s.reason } ) + if ok then seeded = seeded + 1 end + end + if seeded > 0 then + hub_debug( scriptname .. ": seeded " .. seeded .. + " bundled hublist-pinger allow entries (first run)" ) + end + return seeded +end + + +------------------ +--[ADC HANDLERS]-- +------------------ + +local function on_whitelist( user, command, parameters ) + if user:level( ) < oplevel then + user:reply( msg_denied, hub_getbot( ) ) + return PROCESSED + end + local verb, rest = utf_match( parameters or "", "^(%S+)%s*(.*)$" ) + verb = verb and verb:lower( ) or "" + rest = rest or "" + + if verb == "" then + user:reply( msg_usage, hub_getbot( ) ) + return PROCESSED + end + + local user_nick = user:nick( ) or "?" + local user_level = user:level( ) + + if verb == "show" then + local source_filter = utf_match( rest, "^(%S+)" ) + user:reply( format_show( source_filter ), hub_getbot( ), hub_getbot( ) ) + + elseif verb == "count" then + user:reply( format_count( ), hub_getbot( ), hub_getbot( ) ) + + elseif verb == "add" then + local args, perr = parse_add_args( rest ) + if not args then + user:reply( perr, hub_getbot( ) ) + return PROCESSED + end + local clean_reason = args.reason and util_strip( args.reason ) or nil + local ok, _id_or_err, msg = do_add_entry( args.cidr, { + reason = clean_reason, + expires = args.expires, + }, user_nick, user_level ) + user:reply( msg, hub_getbot( ) ) + if ok then + if report then + report.send( report_activate, report_hubbot, report_opchat, + report_llevel, msg ) + end + audit.fire( audit.build( "whitelist.add", user, nil, args.reason, { + cidr = args.cidr, + source = "manual", + expires = args.expires, + id = _id_or_err, + } ) ) + end + + elseif verb == "del" then + local id_str = utf_match( rest, "^(%S+)" ) + local id = tonumber( id_str ) + if not id then + user:reply( msg_usage_del, hub_getbot( ) ) + return PROCESSED + end + local ok, payload, msg = do_del_entry( id, user_nick, user_level ) + user:reply( msg, hub_getbot( ) ) + if ok then + local entry = payload + if report then + report.send( report_activate, report_hubbot, report_opchat, + report_llevel, msg ) + end + audit.fire( audit.build( "whitelist.remove", user, nil, + entry.reason, { + id = id, + cidr = entry.cidr, + source = entry.source, + } ) ) + end + + elseif verb == "export" then + local ok, count_or_err, _path, msg = do_export_jsonl( user_nick ) + user:reply( msg or tostring( count_or_err ), hub_getbot( ) ) + if ok then + if report then + report.send( report_activate, report_hubbot, report_opchat, + report_llevel, msg ) + end + audit.fire( audit.build( "whitelist.export", user, nil, nil, { + path = _path, + count = count_or_err, + } ) ) + end + + elseif verb == "import" then + local path = utf_match( rest, "^(%S+)" ) + if not path then + user:reply( msg_usage_import, hub_getbot( ) ) + return PROCESSED + end + local ok, payload, msg = do_import_jsonl( path, user_nick, user_level ) + user:reply( msg or tostring( payload ), hub_getbot( ) ) + if ok then + if report then + report.send( report_activate, report_hubbot, report_opchat, + report_llevel, msg ) + end + audit.fire( audit.build( "whitelist.import", user, nil, nil, { + path = path, + added = payload.added, + skipped = payload.skipped, + errors = payload.errors, + } ) ) + end + + else + user:reply( utf_format( msg_unknown_verb, verb ), hub_getbot( ) ) + end + + return PROCESSED +end + + +------------------- +--[HTTP HANDLERS]-- +------------------- +-- #78 allowlist, Phase D: HTTP API mirrors the ADC surface. Raw +-- hub.http_register (target is a CIDR / id, not a SID). HTTP +-- admin-token requests carry no operator level; they bypass the ADC +-- hierarchy guard via a synthetic by_level = 100 (the token is the +-- trust surface, per the etc_blocklist HTTP contract). + +-- Accepted `source` labels on POST. The whitelist source is a label +-- only (no priority): "manual" for an operator add, "pinger" for the +-- bundled seed. +local _SOURCE_ENUM = { "manual", "pinger" } + +-- #264 filter/sort spec for GET /v1/whitelist. All rows are pre-listed +-- via whitelist.list() so http_filter sees a stable shape. +local _list_filter_spec = { + string_fields = { + source = function( e ) return e.source or "" end, + by_nick = function( e ) return e.by_nick or "" end, + reason = function( e ) return e.reason or "" end, + cidr = function( e ) return e.cidr or "" end, + }, + integer_fields = { + by_level = function( e ) return tonumber( e.by_level ) or 0 end, + created_at = function( e ) return tonumber( e.created_at ) or 0 end, + expires_at = function( e ) return tonumber( e.expires_at ) or 0 end, + }, + sortable_fields = { + id = function( e ) return tonumber( e.id ) or 0 end, + cidr = function( e ) return e.cidr or "" end, + source = function( e ) return e.source or "" end, + created_at = function( e ) return tonumber( e.created_at ) or 0 end, + expires_at = function( e ) return tonumber( e.expires_at ) or 0 end, + }, + default_sort_field = "id", + default_sort_descending = false, +} + +local function _format_entry_for_wire( e ) + return { + id = e.id, + cidr = e.cidr, + source = e.source, + reason = e.reason or "", + by_nick = e.by_nick or "", + by_level = e.by_level, + expires_at = e.expires_at, + created_at = e.created_at, + meta = e.meta, + } +end + +local http_handler_list_entries = function( req ) + local raw = whitelist.list( ) + local entries = { } + for _, e in ipairs( raw ) do + entries[ #entries + 1 ] = _format_entry_for_wire( e ) + end + local ok, rows, code, msg = http_filter.apply( + req.query or { }, _list_filter_spec, entries ) + if not ok then + return { status = rows, error = { code = code, message = msg } } + end + local pagination = code + local wire = json.encode{ + ok = true, + data = { entries = rows }, + pagination = pagination, + } + return { status = 200, raw_body = wire, + content_type = "application/json; charset=utf-8" } +end + +local http_handler_get_counts = function( req ) + -- Force object shape on an empty by_source (dkjson emits {} not []) + -- so a typed decoder reading it as an object survives a fresh store. + local c = whitelist.count( ) + setmetatable( c.by_source, { __jsontype = "object" } ) + return { status = 200, data = c } +end + +local http_handler_create_entry = function( req ) + local body = req.body or { } + local cidr = body.cidr + if type( cidr ) ~= "string" or cidr == "" then + return { status = 400, error = { code = "E_BAD_INPUT", + message = "missing or empty `cidr` field" } } + end + cidr = util_strip( cidr ) + local reason = body.reason + if reason then reason = util_strip( reason ) end + local source = body.source or "manual" + local expires_at = body.expires_at -- schema-forced number|nil + + local actor_label = util_strip( req.token_label or "http-api" ) + + -- by_level = 100 (master) so a subsequent DELETE by the same token + -- clears the ADC hierarchy guard - the token is the trust surface. + local ok, id, engine_err = whitelist.add( cidr, { + source = source, + reason = reason, + by_nick = actor_label, + by_level = 100, + expires_at = expires_at, + } ) + if not ok then + local err_s = tostring( engine_err or "" ) + local status, code + if err_s:find( "^save failed" ) then + status, code = 500, "E_INTERNAL" + else + status, code = 400, "E_BAD_INPUT" + end + return { status = status, error = { + code = code, message = err_s ~= "" and err_s or "add failed", + } } + end + local msg = utf_format( msg_added, actor_label, id, cidr, source ) + if report then + report.send( report_activate, report_hubbot, report_opchat, + report_llevel, msg ) + end + audit.fire( audit.build( "whitelist.add", + { nick = actor_label, sid = "" }, nil, reason, { + cidr = cidr, + source = source, + id = id, + by_level = 100, + } ) ) + return { status = 201, data = { + action = "added", + id = id, + cidr = cidr, + source = source, + } } +end + +local http_handler_delete_entry = function( req ) + local id_str = req.path_vars and req.path_vars.id + local id = tonumber( id_str ) + if not id or id < 1 or id ~= math.floor( id ) then + return { status = 400, error = { code = "E_BAD_INPUT", + message = "invalid {id} - must be a positive integer" } } + end + local actor_label = util_strip( req.token_label or "http-api" ) + -- Level 100 (master) -> hierarchy guard passes for any entry. + local ok, payload, msg = do_del_entry( id, actor_label, 100 ) + if not ok then + local err_code = payload + local status = err_code == "not_found" and 404 or 400 + return { status = status, error = { + code = err_code == "not_found" and "E_NOT_FOUND" or "E_BAD_INPUT", + message = msg, + } } + end + local entry = payload + if report then + report.send( report_activate, report_hubbot, report_opchat, + report_llevel, msg ) + end + audit.fire( audit.build( "whitelist.remove", + { nick = actor_label, sid = "" }, nil, entry.reason, { + id = id, + cidr = entry.cidr, + source = entry.source, + } ) ) + return { status = 200, data = { + action = "removed", + id = id, + removed = { + cidr = util_strip( entry.cidr or "" ), + source = util_strip( entry.source or "" ), + reason = util_strip( entry.reason or "" ), + by_nick = util_strip( entry.by_nick or "" ), + }, + } } +end + + +----------------- +--[LIFECYCLE ]-- +----------------- + +hub.setlistener( "onStart", { }, + function( ) + -- Seed bundled pingers on first run (before registering the + -- command so a `+whitelist show` immediately after boot shows + -- them). + seed_if_first_run( ) + + local help = hub_import( "cmd_help" ) + if help then + help.reg( help_title, help_usage, help_desc, oplevel ) + end + + local ucmd = hub_import( "etc_usercommands" ) + if ucmd then + ucmd.add( ucmd_menu_show, cmd_main .. " show", + { }, { "CT1" }, oplevel ) + ucmd.add( ucmd_menu_add, cmd_main .. " add %[line:" .. ucmd_popup_cidr .. "] reason=%[line:" .. ucmd_popup_reason .. "]", + { }, { "CT1" }, oplevel ) + ucmd.add( ucmd_menu_del, cmd_main .. " del %[line:" .. ucmd_popup_id .. "]", + { }, { "CT1" }, oplevel ) + ucmd.add( ucmd_menu_count, cmd_main .. " count", + { }, { "CT1" }, oplevel ) + ucmd.add( ucmd_menu_export, cmd_main .. " export", + { }, { "CT1" }, oplevel ) + end + + local hubcmd = hub_import( "etc_hubcommands" ) + assert( hubcmd ) + assert( hubcmd.add( cmd_main, on_whitelist, oplevel ) ) + + -- #78 Phase D HTTP API. Raw hub.http_register (target is a + -- CIDR / id, not a SID). Schema field names use min/max. + if hub.http_register then + hub.http_register( "GET", "/v1/whitelist", "read", + http_handler_list_entries, { + plugin = scriptname, + description = "list whitelist entries (= ADC `+whitelist show`); supports filter (source, cidr_contains, reason_contains, by_nick, by_level, created_at, expires_at) + sort + pagination via the standard http_filter convention", + response_schema = { + entries = { type = "array", required = true }, + }, + } ) + hub.http_register( "GET", "/v1/whitelist/counts", "read", + http_handler_get_counts, { + plugin = scriptname, + description = "counts summary (= ADC `+whitelist count`); prometheus + WebUI dashboard friendly", + response_schema = { + total = { type = "integer", required = true }, + by_source = { type = "object", required = true }, + }, + } ) + hub.http_register( "POST", "/v1/whitelist", "admin", + http_handler_create_entry, { + plugin = scriptname, + description = "add a whitelist entry (= ADC `+whitelist add`); body `{cidr, source?, reason?, expires_at?}`. source enum {manual, pinger}; expires_at is a unix timestamp (integer). CIDRs with host bits set are rejected (`1.2.3.4/24` -> use `1.2.3.0/24`).", + request_schema = { + cidr = { type = "string", required = true, max_length = 45 }, + source = { type = "string", required = false, enum = _SOURCE_ENUM }, + reason = { type = "string", required = false, max_length = 256 }, + expires_at = { type = "integer", required = false, + min = 1, max = 2147483647 }, + }, + response_schema = { + action = { type = "string", required = true }, + id = { type = "integer", required = true }, + cidr = { type = "string", required = true }, + source = { type = "string", required = true }, + }, + } ) + hub.http_register( "DELETE", "/v1/whitelist/{id}", "admin", + http_handler_delete_entry, { + plugin = scriptname, + description = "remove a whitelist entry by numeric id (= ADC `+whitelist del `). Ids are stable across the store lifetime; a re-add gets a fresh id. HTTP admin tokens bypass the ADC-side hierarchy guard - the token is the trust surface.", + response_schema = { + action = { type = "string", required = true }, + id = { type = "integer", required = true }, + removed = { type = "object", required = true }, + }, + } ) + end + + return nil + end +) + + +hub_debug( "** Loaded " .. scriptname .. " " .. scriptversion .. " **" ) + +--// expose internals for unit tests +return { + _parse_add_args = parse_add_args, + _parse_expires_date = parse_expires_date, + _do_add_entry = do_add_entry, + _do_del_entry = do_del_entry, + _do_export_jsonl = do_export_jsonl, + _do_import_jsonl = do_import_jsonl, + _sanitize_import_row = _sanitize_import_row, + _format_show = format_show, + _format_count = format_count, + _seed_if_first_run = seed_if_first_run, + _BUNDLED_SEED = BUNDLED_SEED, + _http_handler_list_entries = http_handler_list_entries, + _http_handler_get_counts = http_handler_get_counts, + _http_handler_create_entry = http_handler_create_entry, + _http_handler_delete_entry = http_handler_delete_entry, + _list_filter_spec = _list_filter_spec, + _SOURCE_ENUM = _SOURCE_ENUM, +} diff --git a/scripts/lang/etc_whitelist.lang.de b/scripts/lang/etc_whitelist.lang.de new file mode 100644 index 00000000..2942a7c2 --- /dev/null +++ b/scripts/lang/etc_whitelist.lang.de @@ -0,0 +1,46 @@ +return { + + help_title = "etc_whitelist.lua - allowlist", + help_usage = "[+!#]whitelist show|add|del|count|export|import ...", + help_desc = "Verwaltet die globale IP/CIDR-Allowlist (nimmt vertrauenswuerdige IPs von den automatischen Blockern aus, nicht von einem manuellen Ban). `+whitelist show` zeigt die aktiven Eintraege.", + + ucmd_menu_show = { "Hub", "Whitelist", "anzeigen" }, + ucmd_menu_add = { "Hub", "Whitelist", "hinzufuegen" }, + ucmd_menu_del = { "Hub", "Whitelist", "per id entfernen" }, + ucmd_menu_count = { "Hub", "Whitelist", "Anzahl je source" }, + ucmd_menu_export = { "Hub", "Whitelist", "nach JSONL exportieren" }, + + ucmd_popup_cidr = "CIDR oder IP (z.B. 192.0.2.0/24):", + ucmd_popup_reason = "Grund (optional):", + ucmd_popup_id = "Eintrag-id aus `+whitelist show`:", + + msg_denied = "Du darfst diesen Befehl nicht benutzen.", + msg_usage = "Verwendung: +whitelist show|add|del|count|export|import ...", + msg_usage_add = "Verwendung: +whitelist add [reason=\"...\"] [expires=YYYY-MM-DD]", + msg_usage_del = "Verwendung: +whitelist del ", + msg_usage_import = "Verwendung: +whitelist import ", + msg_unknown_verb = "Unbekannter Befehl '%s'. Moeglich: show, add, del, count, export, import.", + msg_bad_cidr = "Ungueltige CIDR / IP: %s", + msg_bad_expires = "Ungueltiges expires-Datum '%s'. Erwartet YYYY-MM-DD.", + msg_added = "%s hat Whitelist-Eintrag #%d hinzugefuegt (%s, source=%s).", + msg_add_failed = "whitelist.add fehlgeschlagen: %s", + msg_save_failed = "Whitelist konnte nicht gespeichert werden: %s", + msg_removed = "%s hat Whitelist-Eintrag #%d entfernt (%s, source=%s).", + msg_remove_failed = "whitelist.del fehlgeschlagen: %s", + msg_not_found = "Kein Whitelist-Eintrag mit id #%d.", + msg_hierarchy = "Eintrag #%d nicht entfernbar: hinzugefuegt von einem hoeheren Operator (du=%d, von=%d).", + msg_show_header = "\n=== WHITELIST ===", + msg_show_footer = "=== ENDE ===\n", + msg_show_empty = "(keine Eintraege)", + msg_show_filter = "(gefiltert: source=%s)", + msg_show_capped = "(zeige %d von %d Eintraegen; etc_whitelist_show_limit erhoehen oder nach source filtern)", + msg_count = "whitelist: %d Eintraege gesamt", + msg_no_dkjson = "JSONL-Export/Import benoetigt dkjson, das nicht verfuegbar ist.", + msg_export_ok = "%s hat %d Whitelist-Eintraege nach %s exportiert.", + msg_export_fail = "whitelist.export fehlgeschlagen: %s", + msg_import_ok = "%s hat %d Eintraege aus %s importiert (%d uebersprungen, %d Fehler).", + msg_import_fail = "whitelist.import fehlgeschlagen: %s", + msg_unsafe_path = "Pfad '%s' ist unsicher: %s", + msg_import_level = "Import benoetigt Level %d oder hoeher (du bist %d).", + +} diff --git a/scripts/lang/etc_whitelist.lang.en b/scripts/lang/etc_whitelist.lang.en new file mode 100644 index 00000000..48fbb5d2 --- /dev/null +++ b/scripts/lang/etc_whitelist.lang.en @@ -0,0 +1,46 @@ +return { + + help_title = "etc_whitelist.lua - allowlist", + help_usage = "[+!#]whitelist show|add|del|count|export|import ...", + help_desc = "Manage the global IP/CIDR allowlist (exempts trusted IPs from the automated blockers, not from a manual ban). Run `+whitelist show` for the active entries.", + + ucmd_menu_show = { "Hub", "Whitelist", "show" }, + ucmd_menu_add = { "Hub", "Whitelist", "add" }, + ucmd_menu_del = { "Hub", "Whitelist", "remove by id" }, + ucmd_menu_count = { "Hub", "Whitelist", "count by source" }, + ucmd_menu_export = { "Hub", "Whitelist", "export to JSONL" }, + + ucmd_popup_cidr = "CIDR or IP (e.g. 192.0.2.0/24):", + ucmd_popup_reason = "Reason (optional):", + ucmd_popup_id = "Entry id from `+whitelist show`:", + + msg_denied = "You are not allowed to use this command.", + msg_usage = "Usage: +whitelist show|add|del|count|export|import ...", + msg_usage_add = "Usage: +whitelist add [reason=\"...\"] [expires=YYYY-MM-DD]", + msg_usage_del = "Usage: +whitelist del ", + msg_usage_import = "Usage: +whitelist import ", + msg_unknown_verb = "Unknown verb '%s'. Try: show, add, del, count, export, import.", + msg_bad_cidr = "Invalid CIDR / IP: %s", + msg_bad_expires = "Invalid expires date '%s'. Expected YYYY-MM-DD.", + msg_added = "%s added whitelist entry #%d (%s, source=%s).", + msg_add_failed = "whitelist.add failed: %s", + msg_save_failed = "Failed to persist whitelist: %s", + msg_removed = "%s removed whitelist entry #%d (%s, source=%s).", + msg_remove_failed = "whitelist.del failed: %s", + msg_not_found = "No whitelist entry with id #%d.", + msg_hierarchy = "Cannot remove entry #%d: it was added by a higher-level operator (you=%d, by=%d).", + msg_show_header = "\n=== WHITELIST ===", + msg_show_footer = "=== END ===\n", + msg_show_empty = "(no entries)", + msg_show_filter = "(filtered: source=%s)", + msg_show_capped = "(showing %d of %d entries; raise etc_whitelist_show_limit or filter by source)", + msg_count = "whitelist: %d entries total", + msg_no_dkjson = "JSONL export/import requires dkjson, which is not available.", + msg_export_ok = "%s exported %d whitelist entries to %s.", + msg_export_fail = "whitelist.export failed: %s", + msg_import_ok = "%s imported %d entries from %s (%d skipped, %d errors).", + msg_import_fail = "whitelist.import failed: %s", + msg_unsafe_path = "Path '%s' is unsafe: %s", + msg_import_level = "Import requires level %d or higher (you are %d).", + +} diff --git a/scripts/usr_hubs.lua b/scripts/usr_hubs.lua index 23e820ea..e444d8a2 100755 --- a/scripts/usr_hubs.lua +++ b/scripts/usr_hubs.lua @@ -132,6 +132,10 @@ local check = function( user ) local user_ip = user:ip() local user_cid = user:cid() local hn, hr, ho = user:hubs() + -- #78 allowlist: a whitelisted IP (trusted infra / hublist pinger) + -- is exempt from BOTH the invalid-hubcount kick and the hub-limit + -- ban - pingers legitimately report many hubs / omit the triplet. + if whitelist.is_whitelisted( user_ip ) then return nil end -- Phase 8a F-INF-1b: a client BINF without the HN/HR/HO triplet -- returns nil from user:hubs(). Pre-fix, the arithmetic on the -- next line crashed with "attempt to perform arithmetic on a nil diff --git a/tests/smoke/run.py b/tests/smoke/run.py index 03e68a16..54c3f36c 100644 --- a/tests/smoke/run.py +++ b/tests/smoke/run.py @@ -71,6 +71,14 @@ START_TIMEOUT_SEC = 20 STOP_TIMEOUT_SEC = 10 PROTOCOL_TIMEOUT_SEC = 5 +# How long to wait for a rejected second instance to exit. The launcher +# guard fires before Lua/socket init so it exits near-instantly; the +# slack is for slow CI. A timeout here means the guard did NOT fire. +SINGLE_INSTANCE_TIMEOUT_SEC = 15 +# How long to wait for a second hub (ports already held) to log the clear +# port-in-use message. The bind attempt happens during boot; slack is for +# a slow runner + cert auto-gen preceding the listener setup. +PORT_IN_USE_TIMEOUT_SEC = 25 class TestFailure(Exception): @@ -6915,6 +6923,114 @@ def _dcontains(f, substr): raise TestFailure(f"+blocklist count (post-del) did not return zero: {reply!r}") +def test_whitelist_78_phase_b(): + """#78 allowlist Phase B etc_whitelist `+whitelist` admin CLI smoke. + + End-to-end via a real ADC login as dummy (level 100 > + etc_whitelist_oplevel=80): + + 1. The bundled hublist-pinger seed lands on the FIRST boot + (store .tbl missing) - `+whitelist count` shows the seeded + entries with source=pinger. This is the seed-on-missing + validation the unit test can only mock. + 2. `+whitelist add 198.51.100.0/24 reason="smoke"` -> "added" + (TEST-NET-3, never a real host). Capture the new id. + 3. `+whitelist count` -> seed+1 total, now also "manual". + 4. `+whitelist show` -> lists BOTH the added CIDR and a seed + entry (142.54.190.133), proving seed + operator add coexist. + 5. `+whitelist del ` -> "removed". + 6. `+whitelist count` -> back to the seed-only total. + + Like the blocklist Phase B test this is registered in the initial + TESTS list (NOT the staging-runner block) so it runs in the + default-cfg phase, before the strict ratelimit_tiers overlay whose + 2-BMSG burst would starve this multi-BMSG roundtrip. + """ + def _is_chat_frame(f): + return f.startswith("EMSG ") or f.startswith("DMSG ") or f.startswith("BMSG ") + + def _dcontains(f, substr): + return substr in f.replace("\\s", " ") + + seed_n = 6 # len(BUNDLED_SEED) in scripts/etc_whitelist.lua + + with socket.create_connection( + (HUB_HOST, TEST_PORT_PLAIN), timeout=PROTOCOL_TIMEOUT_SEC + ) as sock: + sid, reader = _adc_login(sock, "dummy", "test") + + # 1. count -> the bundled pinger seed is present (source=pinger) + sock.sendall(f"BMSG {sid} +whitelist\\scount\n".encode("utf-8")) + reply = reader.recv_until( + lambda f: _is_chat_frame(f) + and _dcontains(f, f"{seed_n} entries total") + and _dcontains(f, "pinger"), + timeout=PROTOCOL_TIMEOUT_SEC, + ) + if not reply: + raise TestFailure(f"+whitelist count (seed) did not show the pinger seed: {reply!r}") + + # 2. add a TEST-NET-3 CIDR; capture the new entry id + sock.sendall( + f'BMSG {sid} +whitelist\\sadd\\s198.51.100.0/24\\sreason="smoke"\n' + .encode("utf-8") + ) + reply = reader.recv_until( + lambda f: _is_chat_frame(f) + and _dcontains(f, "added") + and _dcontains(f, "whitelist entry"), + timeout=PROTOCOL_TIMEOUT_SEC, + ) + if not reply: + raise TestFailure(f"+whitelist add did not confirm: {reply!r}") + m = re.search(r"entry #(\d+)", reply.replace("\\s", " ")) + if not m: + raise TestFailure(f"+whitelist add reply had no entry id: {reply!r}") + added_id = m.group(1) + + # 3. count -> seed + 1, now also "manual" + sock.sendall(f"BMSG {sid} +whitelist\\scount\n".encode("utf-8")) + reply = reader.recv_until( + lambda f: _is_chat_frame(f) + and _dcontains(f, f"{seed_n + 1} entries total") + and _dcontains(f, "manual"), + timeout=PROTOCOL_TIMEOUT_SEC, + ) + if not reply: + raise TestFailure(f"+whitelist count (post-add) wrong total: {reply!r}") + + # 4. show -> the added CIDR AND a seed entry both appear + sock.sendall(f"BMSG {sid} +whitelist\\sshow\n".encode("utf-8")) + reply = reader.recv_until( + lambda f: _is_chat_frame(f) + and _dcontains(f, "198.51.100.0/24") + and _dcontains(f, "142.54.190.133"), + timeout=PROTOCOL_TIMEOUT_SEC, + ) + if not reply: + raise TestFailure(f"+whitelist show missing add or seed entry: {reply!r}") + + # 5. del the entry we added + sock.sendall(f"BMSG {sid} +whitelist\\sdel\\s{added_id}\n".encode("utf-8")) + reply = reader.recv_until( + lambda f: _is_chat_frame(f) + and _dcontains(f, "removed") + and _dcontains(f, "whitelist entry"), + timeout=PROTOCOL_TIMEOUT_SEC, + ) + if not reply: + raise TestFailure(f"+whitelist del did not confirm: {reply!r}") + + # 6. count -> back to the seed-only total + sock.sendall(f"BMSG {sid} +whitelist\\scount\n".encode("utf-8")) + reply = reader.recv_until( + lambda f: _is_chat_frame(f) and _dcontains(f, f"{seed_n} entries total"), + timeout=PROTOCOL_TIMEOUT_SEC, + ) + if not reply: + raise TestFailure(f"+whitelist count (post-del) did not return to seed total: {reply!r}") + + def test_blocklist_feeds_status(): """#78 Phase E etc_blocklist_feeds `+blfeeds` read-only status smoke. @@ -9088,6 +9204,116 @@ def body_of(resp): ) +def test_http_phase_d_whitelist(staging_dir: Path, proc=None): + """#78 allowlist Phase D HTTP API for the global whitelist. + + Four endpoints on etc_whitelist.lua v0.02 (GET /v1/whitelist and + /counts, POST, DELETE /v1/whitelist/{id}) - the same shape as the + blocklist HTTP API. Seed-aware: a fresh hub already holds the bundled + pinger entries (source=pinger), so the baseline count is captured + rather than assumed. + + Coverage: anonymous POST -> 401; counts baseline carries a pinger + source; POST create -> 201 + id; GET reflects it + counts baseline+1; + POST host-bits-set cidr -> 400; DELETE created -> 200 removed; + DELETE 999 -> 404; counts back to baseline. + """ + token_path = staging_dir / "cfg" / "api_token.first" + bootstrap_token = None + for line in token_path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if line and not line.startswith("#"): + bootstrap_token = line + break + if not bootstrap_token: + raise TestFailure(f"could not parse token from {token_path}") + auth = b"Authorization: Bearer " + bootstrap_token.encode("ascii") + b"\r\n" + + def status(resp): + return resp.split("\r\n", 1)[0] + + def body_of(resp): + return resp.split("\r\n\r\n", 1)[1] if "\r\n\r\n" in resp else "" + + def post(path: bytes, body: bytes) -> str: + return _http_roundtrip( + b"POST " + path + b" HTTP/1.1\r\n" + auth + + b"Content-Type: application/json\r\n" + b"Content-Length: " + str(len(body)).encode("ascii") + b"\r\n" + b"\r\n" + body + ) + + def get(path: bytes) -> str: + return _http_roundtrip(b"GET " + path + b" HTTP/1.1\r\n" + auth + b"\r\n") + + def delete_id(id_str: str) -> str: + return _http_roundtrip( + b"DELETE /v1/whitelist/" + id_str.encode("ascii") + + b" HTTP/1.1\r\n" + auth + b"\r\n" + ) + + # 1. Anonymous POST -> 401 (admin scope requires a token). + anon_body = b'{"cidr":"203.0.113.9"}' + r = _http_roundtrip( + b"POST /v1/whitelist HTTP/1.1\r\n" + b"Content-Type: application/json\r\n" + b"Content-Length: " + str(len(anon_body)).encode("ascii") + b"\r\n" + b"\r\n" + anon_body + ) + if "401" not in status(r): + raise TestFailure(f"anonymous POST /v1/whitelist should be 401, got {status(r)!r}") + + # 2. Baseline counts: the bundled pinger seed is present. Capture the + # total (robust to any prior whitelist test in the same run). + r = get(b"/v1/whitelist/counts") + b = body_of(r).replace(" ", "") + if "200 OK" not in status(r): + raise TestFailure(f"GET counts pre-flight: expected 200, got {status(r)!r}") + m0 = re.search(r'"total":(\d+)', b) + if not m0: + raise TestFailure(f"counts had no total; body={b!r}") + base = int(m0.group(1)) + if '"pinger"' not in b: + raise TestFailure(f"seed counts missing pinger source; body={b!r}") + + # 3. POST create -> 201 + id. + r = post(b"/v1/whitelist", b'{"cidr":"203.0.113.0/24","reason":"api test"}') + if "201" not in status(r): + raise TestFailure(f"POST create: expected 201, got {status(r)!r}; body={body_of(r)!r}") + m = re.search(r'"id":(\d+)', body_of(r).replace(" ", "")) + if not m: + raise TestFailure(f"POST create response had no id; body={body_of(r)!r}") + new_id = m.group(1) + + # 4. GET reflects the addition; counts == baseline + 1. + r = get(b"/v1/whitelist") + if '"203.0.113.0/24"' not in body_of(r).replace(" ", ""): + raise TestFailure("GET /v1/whitelist did not reflect the created entry") + r = get(b"/v1/whitelist/counts") + if f'"total":{base + 1}' not in body_of(r).replace(" ", ""): + raise TestFailure(f"counts after create: expected {base + 1}; body={body_of(r)!r}") + + # 5. POST host-bits-set cidr -> 400. + r = post(b"/v1/whitelist", b'{"cidr":"1.2.3.4/24"}') + if "400" not in status(r): + raise TestFailure(f"POST host-bits-set should be 400, got {status(r)!r}") + + # 6. DELETE the created entry -> 200 removed. + r = delete_id(new_id) + if "200" not in status(r) or '"action":"removed"' not in body_of(r).replace(" ", ""): + raise TestFailure(f"DELETE id={new_id}: expected 200 removed, got {status(r)!r}; body={body_of(r)!r}") + + # 7. DELETE non-existent -> 404. + r = delete_id("999999") + if "404" not in status(r): + raise TestFailure(f"DELETE 999999 should be 404, got {status(r)!r}") + + # 8. Counts back to the baseline. + r = get(b"/v1/whitelist/counts") + if f'"total":{base}' not in body_of(r).replace(" ", ""): + raise TestFailure(f"counts after delete: expected {base}; body={body_of(r)!r}") + + def test_http_phase_c_blocklist(staging_dir: Path, proc=None): """#78 Phase C HTTP API for the unified blocklist. @@ -12153,6 +12379,7 @@ def test_select_capacity_logged(staging_dir: Path): ("+cmd routing (post-login +help)", test_command_routing), ("alias resolver fallback dispatch (#327)", test_aliases_adc_dispatch), ("blocklist admin CLI (#78 Phase B)", test_blocklist_78_phase_b), + ("whitelist admin CLI + pinger seed (#78 allowlist Phase B)", test_whitelist_78_phase_b), ("blocklist feeds status (#78 Phase E)", test_blocklist_feeds_status), ("proxydetect status (#78 Phase F)", test_proxydetect_status), ("S1: fragmented frame reassembled (phase8-io)", test_s1_fragmented_frame_reassembled), @@ -12331,6 +12558,138 @@ def run_tests(staging_dir: Path): return failed +def test_port_in_use_message(source_install: Path, proc=None, keep_staging=False): + """A second hub started with ports the running first hub already holds + must log a clear operator-facing "port ... already in use - change the + port in cfg/cfg.tbl" message (core/hub.lua add_server_handler). Uses a + SEPARATE install dir staged from the pristine source (so nothing about + the same-dir single-instance lock is involved) pointed at the SAME + test ports as the live shared hub. + + The assertion is on the NEW guidance phrase, not just "already in use": + on Linux the raw luasocket bind error ("...bind: address already in + use") is logged pre-fix too, so a naive "already in use" check would + false-pass. Provably fails pre-fix: on Windows the second hub's + pre-bind SO_REUSEADDR let it silently re-bind the port (no failure, no + message); on Linux the bind failed but only the cryptic line was + logged, never the actionable guidance this asserts.""" + if proc is None or proc.poll() is not None: + raise TestFailure("precondition failed: the first hub is not running") + + second_staging = stage_install(source_install) + second = None + second_log = None + try: + override_test_ports(second_staging) # same TEST_PORT_* as the shared hub + for stale in ("servercert.pem", "serverkey.pem", "cacert.pem", "cakey.pem"): + (second_staging / "certs" / stale).unlink(missing_ok=True) + second, second_log = start_hub(second_staging) + + # The guidance goes to error.log via out.error (log_errors=true by + # default). "change the port in cfg/cfg.tbl" is unique to the new + # message - absent from the pre-fix cryptic luasocket line. + err_log = second_staging / "log" / "error.log" + needle = "change the port in cfg/cfg.tbl" + deadline = time.monotonic() + PORT_IN_USE_TIMEOUT_SEC + found = False + while time.monotonic() < deadline: + if err_log.exists(): + text = err_log.read_text(encoding="utf-8", errors="replace") + if needle in text: + found = True + break + if second.poll() is not None: + break # process died; one more read below then fail + time.sleep(0.2) + if not found: + tail = "" + if err_log.exists(): + tail = err_log.read_text(encoding="utf-8", errors="replace")[-1000:] + raise TestFailure( + "second hub did not log the clear port-in-use guidance " + f"({needle!r}); error.log tail:\n{tail!r}" + ) + finally: + if second is not None: + stop_hub(second, second_log) + if keep_staging: + log(f"second-hub staging kept at {second_staging.parent}") + else: + shutil.rmtree(second_staging.parent, ignore_errors=True) + + +def _start_second_hub(staging_dir: Path): + """Spawn a SECOND hub against the same staging tree, capturing its + own stdout/stderr to log/second-instance.log. Deliberately does NOT + reuse start_hub(): that opens the shared log/smoke-hub.log in "wb" + mode and would truncate the first (running) hub's log.""" + binary = staging_dir / ("Luadch.exe" if sys.platform == "win32" else "luadch") + out_path = staging_dir / "log" / "second-instance.log" + out = open(out_path, "wb") + proc = subprocess.Popen( + [str(binary)], + cwd=str(staging_dir.parent), + stdout=out, + stderr=subprocess.STDOUT, + ) + return proc, out, out_path + + +def test_single_instance_lock(staging_dir: Path, proc=None): + """A second hub launched against the same install tree must refuse to + start: the single-instance lock in hub.c (flock on unix, exclusive + CreateFile on Windows) makes it exit non-zero with an "already + running" message, and the first instance keeps running untouched. + Guards against two hubs sharing cfg/user.tbl/master.key/logs and + corrupting them via interleaved saves. + + Provably fails pre-fix: without the launcher guard the second process + either boots a full second hub (on Windows SO_REUSEADDR lets it + re-bind the ports -> wait() times out) or dies on a bind error with + no single-instance message - neither matches the assertions here.""" + if proc is None or proc.poll() is not None: + raise TestFailure("precondition failed: the first hub is not running") + + second, second_out, second_log_path = _start_second_hub(staging_dir) + try: + try: + rc = second.wait(timeout=SINGLE_INSTANCE_TIMEOUT_SEC) + except subprocess.TimeoutExpired: + raise TestFailure( + f"second hub did not exit within {SINGLE_INSTANCE_TIMEOUT_SEC}s; " + "the single-instance guard is missing (two instances ran " + "concurrently)" + ) + finally: + # Never leave a stray second process behind, even on assert paths. + # Swallow a second TimeoutExpired on the reap so it cannot mask the + # real TestFailure or orphan the process. + if second.poll() is None: + second.kill() + try: + second.wait(timeout=STOP_TIMEOUT_SEC) + except subprocess.TimeoutExpired: + pass + second_out.close() + + if rc == 0: + raise TestFailure("second hub exited 0; expected a non-zero refusal") + + text = second_log_path.read_text(encoding="utf-8", errors="replace") + if "already running" not in text: + raise TestFailure( + "second hub exited non-zero but its log lacks the single-instance " + f"refusal message; log was:\n{text!r}" + ) + + # The rejected second start must not disturb the running first hub. + if proc.poll() is not None: + raise TestFailure( + f"first hub died (rc={proc.poll()}) after the second start attempt; " + "the guard must not disturb the already-running instance" + ) + + def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( @@ -12368,6 +12727,33 @@ def main(): proc, log_file = start_hub(staging_dir) failed = run_tests(staging_dir) + # Port-in-use guidance: with the shared hub still holding the test + # ports, a second hub (separate dir, same ports) must log a clear + # "change the port in cfg/cfg.tbl" message. Runs BEFORE the + # plaintext switch below stops the shared hub. + try: + test_port_in_use_message( + args.install_dir.resolve(), proc=proc, + keep_staging=args.keep_staging, + ) + except Exception as e: + log(f"FAIL port-in-use guidance on second hub: {e}") + failed.append("port-in-use guidance on second hub") + else: + log("PASS port-in-use guidance on second hub") + + # Single-instance lock: with the shared hub still running, a + # second launch against the same tree must be refused by the + # hub.c guard. Runs BEFORE the plaintext-mode switch below stops + # this hub (it needs a live first instance holding the lock). + try: + test_single_instance_lock(staging_dir, proc=proc) + except Exception as e: + log(f"FAIL single-instance lock refuses second start: {e}") + failed.append("single-instance lock refuses second start") + else: + log("PASS single-instance lock refuses second start") + # #128 plaintext-mode test runs AFTER the regular battery # because cfg_secret.init() reads encrypt_usertbl once at # startup, so we have to stop + restart with a flipped cfg. @@ -12905,6 +13291,14 @@ def main(): else: log("PASS HTTP API blocklist (#78 Phase C)") + try: + test_http_phase_d_whitelist(staging_dir, proc=proc) + except Exception as e: + log(f"FAIL HTTP API whitelist (#78 allowlist Phase D): {e}") + failed.append("HTTP API whitelist (#78 allowlist Phase D)") + else: + log("PASS HTTP API whitelist (#78 allowlist Phase D)") + # #261 plugin-management endpoints: GET /v1/plugins (read) + # PUT /v1/plugins/{name}/enabled (admin). Full toggle cycle on # the table-form etc_motd entry + negative coverage (403 on diff --git a/tests/unit/blocklist_test.lua b/tests/unit/blocklist_test.lua index 0087135d..6acad7f8 100644 --- a/tests/unit/blocklist_test.lua +++ b/tests/unit/blocklist_test.lua @@ -37,6 +37,7 @@ local function _stub_use_factory( opts ) local cfg_values = opts.cfg or { } return function( name ) if name == "type" then return type end + if name == "pcall" then return pcall end if name == "next" then return next end if name == "pairs" then return pairs end if name == "ipairs" then return ipairs end diff --git a/tests/unit/etc_geoip_test.lua b/tests/unit/etc_geoip_test.lua index b60c49ae..1a5c7190 100644 --- a/tests/unit/etc_geoip_test.lua +++ b/tests/unit/etc_geoip_test.lua @@ -157,6 +157,12 @@ _G.hub.import = function( name ) return nil end +-- #78 allowlist stub: the plugin's whitelist guard. Default: nothing +-- whitelisted (existing tests unaffected). A test flips an entry on +-- via `_wl[ip] = true`. +local _wl = { } +_G.whitelist = { is_whitelisted = function( ip ) return _wl[ ip ] == true end } + ---------------------------------------------------------------------- -- Load helper + stub user ---------------------------------------------------------------------- @@ -274,6 +280,32 @@ do eq( "allowed user: no audit", #_audit, 0 ) end +---------------------------------------------------------------------- +-- #78 allowlist: a whitelisted IP is exempt from the geoip block. +-- Provably FAILS pre-fix (without the guard the GB IP is kicked). +---------------------------------------------------------------------- + +do + _cfg.etc_geoip_action = "block" + load_plugin( ) + local check = _listeners.onConnect + + -- baseline: GB blocked when NOT whitelisted + _wl = { } + local r1 = check( make_user{ level = 20, ip = "81.2.69.160" } ) + eq( "geoip wl baseline: GB blocked", r1, "PROCESSED" ) + eq( "geoip wl baseline: kicked", #_kicks, 1 ) + + -- whitelist that IP -> exempt (no kick, no audit, allowed) + _kicks = { }; _audit = { } + _wl = { [ "81.2.69.160" ] = true } + local r2 = check( make_user{ level = 20, ip = "81.2.69.160" } ) + falsy( "geoip whitelisted -> not blocked", r2 ) + eq( "geoip whitelisted -> no kick", #_kicks, 0 ) + eq( "geoip whitelisted -> no audit", #_audit, 0 ) + _wl = { } +end + ---------------------------------------------------------------------- -- onConnect: log_only audits + reports but does NOT kick ---------------------------------------------------------------------- diff --git a/tests/unit/etc_proxydetect_test.lua b/tests/unit/etc_proxydetect_test.lua index b68b1c28..036f5e91 100644 --- a/tests/unit/etc_proxydetect_test.lua +++ b/tests/unit/etc_proxydetect_test.lua @@ -128,6 +128,10 @@ _G.blocklist = { return true, #_adds end, } +-- #78 allowlist stub: the plugin's whitelist guard. Default: nothing +-- whitelisted (existing tests unaffected). A test flips `_wl[ip] = true`. +local _wl = { } +_G.whitelist = { is_whitelisted = function( ip ) return _wl[ ip ] == true end } _G.http_client = { request = function( req ) _requests[ #_requests + 1 ] = req @@ -320,6 +324,28 @@ do eq( "exempt: no request for exempt op", #_requests, 0 ) end +---------------------------------------------------------------------- +-- #78 allowlist: a whitelisted IP skips the provider query + kick. +-- Provably FAILS pre-fix (without the guard the fresh IP fires a +-- request); the whitelisted IP must add NO new request. +---------------------------------------------------------------------- +do + load_plugin( { _resolved_key = "SECRET42" } ) + + -- baseline: a blockable-level user at a fresh IP fires one request + _wl = { } + connect( mkuser( 20, "9.9.9.9", "SIDW1", "CIDW1" ) ) + eq( "proxy wl baseline: request fired", #_requests, 1 ) + + -- whitelist a DIFFERENT fresh IP -> no new request, no kick + _wl = { [ "8.8.8.8" ] = true } + local u2 = mkuser( 20, "8.8.8.8", "SIDW2", "CIDW2" ) + connect( u2 ) + eq( "proxy whitelisted -> NO new request", #_requests, 1 ) + truthy( "proxy whitelisted -> not kicked", u2._killed == nil ) + _wl = { } +end + ---------------------------------------------------------------------- -- log_only -> no store push, no kill, but audits ---------------------------------------------------------------------- diff --git a/tests/unit/etc_whitelist_test.lua b/tests/unit/etc_whitelist_test.lua new file mode 100644 index 00000000..98fa1a73 --- /dev/null +++ b/tests/unit/etc_whitelist_test.lua @@ -0,0 +1,533 @@ +--[[ + + tests/unit/etc_whitelist_test.lua + + Unit tests for scripts/etc_whitelist.lua (#78 allowlist, Phase B). + + Coverage: + - parse_add_args: cidr-only, reason="quoted", expires=YYYY-MM-DD, + both options; the quoted/unquoted isolation + - parse_expires_date: YYYY-MM-DD -> end-of-day ts, bad -> nil + - do_add_entry: happy, bad cidr surfaces engine err, bad expires + - do_del_entry: not_found, hierarchy block, happy + - _sanitize_import_row: control-byte stripping on every field + - format_show / format_count: basic shape + - seed_if_first_run: seeds the bundled pingers on a MISSING store, + seeds nothing when the store exists, and + nothing when etc_whitelist_seed=false + - export/import JSONL round-trip (manual entries) + + The core whitelist engine is stubbed with an in-memory list so the + test isolates the plugin's mutation/dispatch logic from the Phase A + matcher (which has its own test file). + + Run: lua5.4 tests/unit/etc_whitelist_test.lua + +]]-- + +---------------------------------------------------------------------- +-- In-memory whitelist-engine stub + virtual FS for the seed probe / +-- export-import round-trip. +---------------------------------------------------------------------- + +local _entries, _next_id +local _vfs = { } -- path -> string ("exists" marker or file body) +local _vfs_active = true + +local function _reset_engine( ) + _entries, _next_id = { }, 1 +end +_reset_engine( ) + +_G.whitelist = { + add = function( cidr, opts ) + if type( cidr ) ~= "string" or cidr == "" then return false, nil, "bad cidr" end + if cidr:find( "INVALID" ) then return false, nil, "synthetic-bad" end + opts = opts or { } + local e = { + id = _next_id, cidr = cidr, source = opts.source or "manual", + reason = opts.reason or "", by_nick = opts.by_nick, + by_level = opts.by_level, expires_at = opts.expires_at, + created_at = 1000, + } + _entries[ #_entries + 1 ] = e + _next_id = _next_id + 1 + return true, e.id + end, + remove = function( id ) + for i, e in ipairs( _entries ) do + if e.id == id then table.remove( _entries, i ); return true end + end + return false, "not_found" + end, + list = function( filter ) + filter = filter or { } + local out = { } + for _, e in ipairs( _entries ) do + if ( not filter.source ) or e.source == filter.source then + out[ #out + 1 ] = { + id = e.id, cidr = e.cidr, source = e.source, + reason = e.reason, by_nick = e.by_nick, by_level = e.by_level, + expires_at = e.expires_at, created_at = e.created_at, + } + end + end + return out + end, + count = function( ) + local by_source = { } + for _, e in ipairs( _entries ) do + by_source[ e.source ] = ( by_source[ e.source ] or 0 ) + 1 + end + return { total = #_entries, by_source = by_source } + end, +} + +---------------------------------------------------------------------- +-- Sandbox-global stubs. +---------------------------------------------------------------------- + +local _cfg_overrides = { } +local _listeners = { } -- captured hub.setlistener callbacks +local _http_routes = { } -- captured hub.http_register registrations +_G.hub = { + setlistener = function( ev, _opts, fn ) _listeners[ ev ] = fn end, + debug = function( ) end, + getbot = function( ) return "stub-bot" end, + http_register = function( method, path, scope, handler, meta ) + _http_routes[ method .. " " .. path ] = { scope = scope, handler = handler, meta = meta } + end, + import = function( name ) + if name == "etc_hubcommands" then + return { add = function( ) return true end, has = function( ) return false end, + list = function( ) return { } end } + end + if name == "etc_report" then + return { send = function( ) end } + end + return nil + end, +} + +-- Minimal http_filter stub: source-filter emulation, pass-through +-- otherwise (real filter/sort/paginate is core/http_filter's own test). +_G.http_filter = { + apply = function( query, spec, rows ) + if query and query.source and query.source ~= "" then + local filtered = { } + for _, e in ipairs( rows ) do + if e.source == query.source then filtered[ #filtered + 1 ] = e end + end + rows = filtered + end + return true, rows, { total = #rows, limit = 100, offset = 0 }, nil + end, +} + +-- audit stub: capture fired events for the HTTP create/delete assertions. +local _audit_fired = { } +_G.audit = { + build = function( action, actor, target, reason, meta ) + return { action = action, actor = actor, reason = reason, meta = meta } + end, + fire = function( ev ) _audit_fired[ #_audit_fired + 1 ] = ev end, +} +_G.setmetatable = setmetatable + +_G.cfg = { + get = function( key ) + if _cfg_overrides[ key ] ~= nil then return _cfg_overrides[ key ] end + if key == "language" then return "en" end + if key == "etc_whitelist_oplevel" then return 80 end + if key == "etc_whitelist_show_limit" then return 200 end + if key == "etc_whitelist_seed" then return true end + if key == "etc_whitelist_report" then return true end + if key == "etc_whitelist_report_hubbot" then return false end + if key == "etc_whitelist_report_opchat" then return true end + if key == "etc_whitelist_llevel" then return 60 end + if key == "etc_whitelist_import_min_level" then return 100 end + if key == "whitelist_store_path" then return "scripts/data/etc_whitelist.tbl" end + return nil + end, + loadlanguage = function( ) return { }, nil end, +} + +_G.util = { + strip_control_bytes = function( s ) + if type( s ) ~= "string" then return s end + return ( s:gsub( "[%c]", "" ) ) + end, + safe_path = function( p ) + if type( p ) ~= "string" or p == "" then return false, "empty path" end + if p:find( "%.%." ) then return false, "parent-dir blocked" end + return true + end, +} + +_G.utf = { match = string.match, format = string.format } +_G.PROCESSED = 1 +_G.dkjson = require and nil -- resolved below + +-- Load the real dkjson if available; else provide a tiny stub so the +-- export/import round-trip test still exercises the plugin logic. +do + local ok, mod = pcall( dofile, "dkjson/dkjson.lua" ) + if ok and type( mod ) == "table" then + _G.dkjson = mod + else + -- Minimal JSON good enough for the flat rows we encode/decode. + _G.dkjson = { + encode = function( t ) + local parts = { } + for k, v in pairs( t ) do + local vs + if type( v ) == "string" then vs = string.format( "%q", v ) + elseif type( v ) == "number" then vs = tostring( v ) + else vs = "null" end + parts[ #parts + 1 ] = string.format( "%q:%s", k, vs ) + end + return "{" .. table.concat( parts, "," ) .. "}" + end, + decode = function( s ) + local t = { } + for k, v in s:gmatch( '"([^"]-)":"([^"]-)"' ) do t[ k ] = v end + for k, v in s:gmatch( '"([^"]-)":(%-?%d+)' ) do t[ k ] = tonumber( v ) end + if next( t ) == nil then return nil, nil, "empty" end + return t + end, + } + end +end + +-- io.open VFS: read-mode succeeds iff the path is in _vfs; write-mode +-- captures bytes into _vfs on close. Used for the seed-probe (store +-- present vs missing) and export/import round-trip. +local _real_io_open = io.open +io.open = function( path, mode ) + if not _vfs_active then return _real_io_open( path, mode ) end + mode = mode or "r" + if mode:find( "w" ) then + local buf = { } + local handle + handle = { + write = function( _self, ... ) + for _, s in ipairs{ ... } do buf[ #buf + 1 ] = tostring( s ) end + return handle + end, + close = function( ) _vfs[ path ] = table.concat( buf ) end, + } + return handle + end + local content = _vfs[ path ] + if not content then return nil, "No such file or directory" end + local pos = 1 + local handle + handle = { + read = function( _self, fmt ) + if fmt == "*l" or fmt == "l" then + if pos > #content then return nil end + local nl = content:find( "\n", pos, true ) + local line + if nl then line = content:sub( pos, nl - 1 ); pos = nl + 1 + else line = content:sub( pos ); pos = #content + 1 end + return line + end + end, + close = function( ) end, + } + return handle +end + +local function load_plugin( ) + return assert( loadfile( "scripts/etc_whitelist.lua" ) )( ) +end +local wl = load_plugin( ) + +---------------------------------------------------------------------- +-- Tiny harness +---------------------------------------------------------------------- + +local _passes, _fails = 0, 0 +local function eq( what, got, want ) + if got == want then _passes = _passes + 1 + else _fails = _fails + 1 + io.stderr:write( string.format( "FAIL: %s\n got: %s\n want: %s\n", + what, tostring( got ), tostring( want ) ) ) end +end +local function truthy( what, v ) + if v then _passes = _passes + 1 + else _fails = _fails + 1; io.stderr:write( "FAIL: " .. what .. "\n" ) end +end + +---------------------------------------------------------------------- +-- parse_add_args +---------------------------------------------------------------------- + +do + local a = wl._parse_add_args( "1.2.3.0/24" ) + truthy( "parse cidr-only", a ) + eq( "parse cidr", a and a.cidr, "1.2.3.0/24" ) + eq( "parse no reason", a and a.reason, nil ) + + local b = wl._parse_add_args( '10.0.0.0/8 reason="trusted vpn"' ) + eq( "parse quoted reason cidr", b and b.cidr, "10.0.0.0/8" ) + eq( "parse quoted reason", b and b.reason, "trusted vpn" ) + + local c = wl._parse_add_args( "9.9.9.9 expires=2027-01-01" ) + eq( "parse expires cidr", c and c.cidr, "9.9.9.9" ) + eq( "parse expires", c and c.expires, "2027-01-01" ) + + local d = wl._parse_add_args( '5.5.5.5 reason="a b" expires=2027-06-30' ) + eq( "parse both reason", d and d.reason, "a b" ) + eq( "parse both expires", d and d.expires, "2027-06-30" ) + + eq( "parse empty -> nil", wl._parse_add_args( "" ), nil ) +end + +---------------------------------------------------------------------- +-- parse_expires_date +---------------------------------------------------------------------- + +do + truthy( "expires valid -> ts", wl._parse_expires_date( "2027-12-31" ) ) + eq( "expires bad -> nil", wl._parse_expires_date( "nope" ), nil ) + eq( "expires empty -> nil", wl._parse_expires_date( "" ), nil ) +end + +---------------------------------------------------------------------- +-- do_add_entry +---------------------------------------------------------------------- + +_reset_engine( ) +do + local ok, id, msg = wl._do_add_entry( "1.2.3.0/24", { reason = "r" }, "opnick", 80 ) + eq( "add ok", ok, true ) + truthy( "add id", id ) + truthy( "add msg mentions manual", msg and msg:find( "manual" ) ) + + local ok2, code2 = wl._do_add_entry( "INVALIDcidr", { }, "opnick", 80 ) + eq( "add bad cidr false", ok2, false ) + eq( "add bad cidr code", code2, "bad_cidr" ) + + local ok3, code3 = wl._do_add_entry( "1.2.3.4", { expires = "not-a-date" }, "opnick", 80 ) + eq( "add bad expires false", ok3, false ) + eq( "add bad expires code", code3, "bad_expires" ) +end + +---------------------------------------------------------------------- +-- do_del_entry: not_found, hierarchy, happy +---------------------------------------------------------------------- + +_reset_engine( ) +do + -- entry added by a level-100 master + wl._do_add_entry( "10.10.0.0/16", { }, "master", 100 ) + local rows = wl._do_add_entry and _entries -- direct id lookup + local master_id = _entries[ 1 ].id + + -- level-80 op cannot remove the master's entry + local ok, code = wl._do_del_entry( master_id, "lowop", 80 ) + eq( "hierarchy blocks lower op", ok, false ) + eq( "hierarchy code", code, "hierarchy" ) + + -- not found + local ok2, code2 = wl._do_del_entry( 99999, "master", 100 ) + eq( "del not_found false", ok2, false ) + eq( "del not_found code", code2, "not_found" ) + + -- master can remove + local ok3 = wl._do_del_entry( master_id, "master", 100 ) + eq( "master removes own entry", ok3, true ) + eq( "count after del", wl._format_count and #_entries, 0 ) +end + +---------------------------------------------------------------------- +-- _sanitize_import_row: control-byte strip +---------------------------------------------------------------------- + +do + local cidr, opts = wl._sanitize_import_row{ + cidr = "1.2.3.0/24\r\n", reason = "clean\1reason", source = "pin\2ger", + by_nick = "op\3", by_level = 80, expires_at = 12345, + } + eq( "sanitize cidr stripped", cidr, "1.2.3.0/24" ) + eq( "sanitize reason stripped", opts.reason, "cleanreason" ) + eq( "sanitize source stripped", opts.source, "pinger" ) + eq( "sanitize by_nick stripped", opts.by_nick, "op" ) + eq( "sanitize by_level", opts.by_level, 80 ) + + local nilcidr = wl._sanitize_import_row{ reason = "x" } + eq( "sanitize missing cidr nil", nilcidr, nil ) +end + +---------------------------------------------------------------------- +-- format_show / format_count shape +---------------------------------------------------------------------- + +_reset_engine( ) +do + wl._do_add_entry( "1.2.3.0/24", { reason = "trusted" }, "op", 80 ) + local show = wl._format_show( nil ) + truthy( "show has header", show:find( "WHITELIST" ) ) + truthy( "show lists the cidr", show:find( "1.2.3.0/24" ) ) + truthy( "show shows source", show:find( "src=manual" ) ) + + local empty = wl._format_show( "nosuchsource" ) + truthy( "show filtered-empty", empty:find( "no entries" ) ) + + local cnt = wl._format_count( ) + truthy( "count total line", cnt:find( "1 entries total" ) ) + truthy( "count by-source", cnt:find( "manual: 1" ) ) +end + +---------------------------------------------------------------------- +-- seed_if_first_run: missing store -> seeds pingers; existing -> 0; +-- disabled -> 0. +---------------------------------------------------------------------- + +do + local SP = "scripts/data/etc_whitelist.tbl" + + -- 1) store MISSING -> seed the bundled pingers + _reset_engine( ) + _vfs = { } -- store file absent + wl = load_plugin( ) -- re-read cfg (seed=true) + local n = wl._seed_if_first_run( ) + eq( "seed count == bundled size", n, #wl._BUNDLED_SEED ) + eq( "engine now holds the seed", #_entries, #wl._BUNDLED_SEED ) + local all_pinger = true + for _, e in ipairs( _entries ) do if e.source ~= "pinger" then all_pinger = false end end + truthy( "all seeded entries source=pinger", all_pinger ) + + -- 2) store EXISTS -> no seed + _reset_engine( ) + _vfs = { [ SP ] = "x" } -- store present + wl = load_plugin( ) + eq( "existing store -> seed 0", wl._seed_if_first_run( ), 0 ) + eq( "existing store -> no entries added", #_entries, 0 ) + + -- 3) seed disabled -> no seed even if missing + _reset_engine( ) + _vfs = { } + _cfg_overrides = { etc_whitelist_seed = false } + wl = load_plugin( ) + eq( "seed disabled -> 0", wl._seed_if_first_run( ), 0 ) + _cfg_overrides = { } + wl = load_plugin( ) +end + +---------------------------------------------------------------------- +-- export / import JSONL round-trip (manual entries only) +---------------------------------------------------------------------- + +_reset_engine( ) +do + _vfs = { } + wl = load_plugin( ) + wl._do_add_entry( "1.2.3.0/24", { reason = "op-added" }, "op", 100 ) + wl._do_add_entry( "9.9.9.9", { }, "op", 100 ) + -- a pinger-sourced entry must NOT be exported + _G.whitelist.add( "8.8.8.8", { source = "pinger", reason = "seed" } ) + + local ok, count, path = wl._do_export_jsonl( "op" ) + eq( "export ok", ok, true ) + eq( "export count = manual only (2)", count, 2 ) + truthy( "export wrote file", _vfs[ path ] ~= nil ) + + -- wipe + re-import + _reset_engine( ) + local iok, stats = wl._do_import_jsonl( path, "master", 100 ) + eq( "import ok", iok, true ) + eq( "import added 2", stats.added, 2 ) + eq( "import 0 errors", stats.errors, 0 ) + eq( "engine holds re-imported entries", #_entries, 2 ) + + -- import level guard + local lok, lcode = wl._do_import_jsonl( path, "lowop", 80 ) + eq( "import below min_level rejected", lok, false ) + truthy( "import level err", lcode and lcode:find( "Import requires" ) ) +end + +---------------------------------------------------------------------- +-- HTTP API (Phase D): registration + scope + schema + handler shapes +---------------------------------------------------------------------- + +_reset_engine( ) +do + _vfs = { } + _cfg_overrides = { etc_whitelist_seed = false } -- boot with an empty store + wl = load_plugin( ) + assert( _listeners.onStart, "onStart not captured" ) + _listeners.onStart( ) -- register the routes + + -- registration + scope + truthy( "http GET /v1/whitelist registered", _http_routes[ "GET /v1/whitelist" ] ) + truthy( "http GET counts registered", _http_routes[ "GET /v1/whitelist/counts" ] ) + truthy( "http POST registered", _http_routes[ "POST /v1/whitelist" ] ) + truthy( "http DELETE registered", _http_routes[ "DELETE /v1/whitelist/{id}" ] ) + eq( "http GET scope read", _http_routes[ "GET /v1/whitelist" ].scope, "read" ) + eq( "http POST scope admin", _http_routes[ "POST /v1/whitelist" ].scope, "admin" ) + eq( "http DELETE scope admin", _http_routes[ "DELETE /v1/whitelist/{id}" ].scope, "admin" ) + + -- POST schema uses min/max (not minimum/maximum) + no stealth field + local ps = _http_routes[ "POST /v1/whitelist" ].meta.request_schema + eq( "http POST cidr required", ps.cidr.required, true ) + eq( "http POST cidr max_length", ps.cidr.max_length, 45 ) + truthy( "http POST source enum", ps.source.enum ) + eq( "http POST expires_at min", ps.expires_at.min, 1 ) + eq( "http POST no stealth field", ps.stealth, nil ) + + -- create (happy) + _reset_engine( ) + local rc = wl._http_handler_create_entry{ + body = { cidr = "203.0.113.0/24", source = "pinger", reason = "api", + expires_at = 2000000000 }, + token_label = "tok", + } + eq( "http create 201", rc.status, 201 ) + eq( "http create action added", rc.data.action, "added" ) + eq( "http create cidr echoed", rc.data.cidr, "203.0.113.0/24" ) + eq( "http create source echoed", rc.data.source, "pinger" ) + eq( "http create engine by_level 100", _entries[ 1 ].by_level, 100 ) + eq( "http create engine by_nick token", _entries[ 1 ].by_nick, "tok" ) + local la = _audit_fired[ #_audit_fired ] + eq( "http create audit action", la and la.action, "whitelist.add" ) + eq( "http create audit by_level 100", la and la.meta and la.meta.by_level, 100 ) + + -- create bad / missing cidr + eq( "http create bad cidr 400", + wl._http_handler_create_entry{ body = { cidr = "INVALIDx" }, token_label = "t" }.status, 400 ) + eq( "http create missing cidr 400", + wl._http_handler_create_entry{ body = { }, token_label = "t" }.status, 400 ) + + -- counts + local rct = wl._http_handler_get_counts{ } + eq( "http counts 200", rct.status, 200 ) + eq( "http counts total 1", rct.data.total, 1 ) + eq( "http counts by_source pinger 1", rct.data.by_source.pinger, 1 ) + + -- list + local rl = wl._http_handler_list_entries{ query = { } } + eq( "http list 200", rl.status, 200 ) + truthy( "http list raw_body string", type( rl.raw_body ) == "string" ) + eq( "http list content_type", rl.content_type, "application/json; charset=utf-8" ) + + -- delete (happy + not_found + bad id) + local rd = wl._http_handler_delete_entry{ path_vars = { id = "1" }, token_label = "tok" } + eq( "http delete 200", rd.status, 200 ) + eq( "http delete action removed", rd.data.action, "removed" ) + eq( "http delete not_found 404", + wl._http_handler_delete_entry{ path_vars = { id = "9999" }, token_label = "t" }.status, 404 ) + eq( "http delete bad id 400", + wl._http_handler_delete_entry{ path_vars = { id = "abc" }, token_label = "t" }.status, 400 ) + + _cfg_overrides = { } +end + +---------------------------------------------------------------------- + +if _fails > 0 then + io.stderr:write( string.format( "\nFAIL: %d/%d checks failed\n", _fails, _passes + _fails ) ) + os.exit( 1 ) +end +print( string.format( "OK: %d checks passed", _passes ) ) diff --git a/tests/unit/whitelist_test.lua b/tests/unit/whitelist_test.lua new file mode 100644 index 00000000..d7d96345 --- /dev/null +++ b/tests/unit/whitelist_test.lua @@ -0,0 +1,409 @@ +--[[ + + tests/unit/whitelist_test.lua + + Unit tests for core/whitelist.lua (#78 allowlist). Coverage: + + - add / remove / list / count round-trip + - is_whitelisted: bucket-routed match (v4 + v6) + non-match + - NO priority: any active match = allowed (source is a label) + - CIDR + exact IP; IPv6 /32 /64 /128 + - v4-mapped v6 lookup (a plain-v4 entry matches a v4-over-v6 + client - the dual-stack pinger case) + - expires_at: expired entries treated as non-matching + - reload from stub store rebuilds cache + bucket index + - fresh-install: reload with no .tbl is silent + clean + - disabled engine: is_whitelisted always false + - hex round-trip + - empty-store fast path + - INTEGRATION with core/blocklist.lua: Model-A precedence - + a whitelisted IP overrides an AUTOMATED block but NOT a + manual pin. This block provably FAILS against a blocklist + without the whitelist hook (§1a.7). + + Run: lua5.4 tests/unit/whitelist_test.lua + +]]-- + +local _disk = { } -- path -> table +local _now = 1000 +local _save_count = 0 +local _fail_save = false + +local function _stub_use_factory( opts ) + opts = opts or { } + local cfg_values = opts.cfg or { } + return function( name ) + if name == "type" then return type end + if name == "pcall" then return pcall end + if name == "next" then return next end + if name == "pairs" then return pairs end + if name == "ipairs" then return ipairs end + if name == "tostring" then return tostring end + if name == "tonumber" then return tonumber end + if name == "string" then return string end + if name == "table" then return table end + if name == "math" then return math end + if name == "socket" then return { gettime = function() return _now end } end + if name == "util" then + return { + loadtable = function( path ) return _disk[ path ] end, + savetable = function( tbl, _name, path ) + _save_count = _save_count + 1 + if _fail_save then return false, "disk full (test)" end + local copy = { } + for i, row in ipairs( tbl ) do + local rcopy = { } + for k, v in pairs( row ) do rcopy[ k ] = v end + copy[ i ] = rcopy + end + _disk[ path ] = copy + return true + end, + } + end + if name == "cfg" then + return { + get = function( k ) return cfg_values[ k ] end, + registerevent = function( ) end, + } + end + if name == "out" then + return { put = function( ) end, error = function( ) end } + end + if name == "io" then + return { + open = function( path, _mode ) + if _disk[ path ] ~= nil then + return { close = function( ) end } + end + return nil, "No such file or directory" + end, + } + end + if name == "ipmatch" then + return _G._loaded_ipmatch + end + error( "whitelist_test shim: missing dep " .. tostring( name ) ) + end +end + +_G.use = _stub_use_factory( ) +_G._loaded_ipmatch = assert( loadfile( "core/ipmatch.lua" ) )( ) +local wl = assert( loadfile( "core/whitelist.lua" ) )( ) + +---------------------------------------------------------------------- +-- Tiny harness +---------------------------------------------------------------------- + +local _passes, _fails = 0, 0 +local function eq( what, got, want ) + if got == want then _passes = _passes + 1 + else + _fails = _fails + 1 + io.stderr:write( string.format( "FAIL: %s\n got: %s\n want: %s\n", + what, tostring( got ), tostring( want ) ) ) + end +end +local function truthy( what, v ) + if v then _passes = _passes + 1 + else _fails = _fails + 1; io.stderr:write( "FAIL: " .. what .. " got=" .. tostring( v ) .. "\n" ) end +end + +local function reset_state( opts ) + _disk = { } + _now = 1000 + _save_count = 0 + _fail_save = false + _G.use = _stub_use_factory( opts ) + wl = assert( loadfile( "core/whitelist.lua" ) )( ) + wl.init( ) +end + +---------------------------------------------------------------------- +-- add + is_whitelisted basic round-trip +---------------------------------------------------------------------- + +reset_state{} +do + local ok, id = wl.add( "1.2.3.0/24", { reason = "test", source = "manual" } ) + eq( "add returns ok", ok, true ) + truthy( "add returns id", id ) + + eq( "is_whitelisted in-range true", wl.is_whitelisted( "1.2.3.50" ), true ) + eq( "is_whitelisted out-of-range false", wl.is_whitelisted( "1.2.4.50" ), false ) + + local ok2, id2 = wl.add( "9.9.9.9", { source = "manual" } ) -- bare IP -> /32 + truthy( "add bare IP ok", ok2 and id2 ) + eq( "is_whitelisted exact /32", wl.is_whitelisted( "9.9.9.9" ), true ) + eq( "is_whitelisted /32 neighbour false", wl.is_whitelisted( "9.9.9.10" ), false ) +end + +---------------------------------------------------------------------- +-- count + list + source filter +---------------------------------------------------------------------- + +do + wl.add( "10.0.0.0/8", { source = "pinger" } ) + wl.add( "172.16.0.0/12", { source = "pinger" } ) + local c = wl.count( ) + eq( "count total", c.total, 4 ) + eq( "count by_source manual", c.by_source.manual, 2 ) + eq( "count by_source pinger", c.by_source.pinger, 2 ) + + eq( "list all length", #wl.list( ), 4 ) + eq( "list filter source=pinger length", #wl.list{ source = "pinger" }, 2 ) +end + +---------------------------------------------------------------------- +-- NO priority: overlapping entries both count as a match; either +-- source label being present is enough to allow. +---------------------------------------------------------------------- + +reset_state{} +do + wl.add( "192.0.2.0/24", { source = "pinger", reason = "range" } ) + wl.add( "192.0.2.42", { source = "manual", reason = "pin" } ) + eq( "overlap: covered IP allowed", wl.is_whitelisted( "192.0.2.42" ), true ) + eq( "overlap: other in-range IP allowed", wl.is_whitelisted( "192.0.2.99" ), true ) + -- Remove the /32; the /24 still covers .42 (no priority, just any match) + local rows = wl.list{ source = "manual" } + wl.remove( rows[ 1 ].id ) + eq( "after removing /32, /24 still covers", wl.is_whitelisted( "192.0.2.42" ), true ) +end + +---------------------------------------------------------------------- +-- IPv6 /32 /64 /128 (the pinger v6 ranges) +---------------------------------------------------------------------- + +reset_state{} +do + wl.add( "2001:41d0:a:f8b3::/64", { source = "pinger" } ) -- DCpinger range + eq( "v6 /64 covers rotating host", wl.is_whitelisted( "2001:41d0:a:f8b3::1" ), true ) + eq( "v6 /64 covers other host", wl.is_whitelisted( "2001:41d0:a:f8b3::dead" ), true ) + eq( "v6 /64 outside range false", wl.is_whitelisted( "2001:41d0:a:f8b4::1" ), false ) + + wl.add( "2001:db8::/32", { source = "manual" } ) + eq( "v6 /32 in range", wl.is_whitelisted( "2001:db8:1234::5" ), true ) + eq( "v6 /32 out range", wl.is_whitelisted( "2001:db9::5" ), false ) + + wl.add( "2602:fed2:731b:25::a", { source = "pinger" } ) -- exact /128 + eq( "v6 /128 exact match", wl.is_whitelisted( "2602:fed2:731b:25::a" ), true ) + eq( "v6 /128 neighbour false", wl.is_whitelisted( "2602:fed2:731b:25::b" ), false ) +end + +---------------------------------------------------------------------- +-- v4-mapped v6: a plain-v4 whitelist entry must match a v4-over-v6 +-- client (the dual-stack pinger case, e.g. 142.54.190.133 arriving as +-- ::ffff:142.54.190.133). Provably FAILS without the normalisation. +---------------------------------------------------------------------- + +reset_state{} +do + wl.add( "142.54.190.133", { source = "pinger" } ) + eq( "v4-mapped v6 matches plain-v4 entry", + wl.is_whitelisted( "::ffff:142.54.190.133" ), true ) + + wl.add( "5.252.102.0/24", { source = "pinger" } ) + eq( "v4-mapped v6 in v4 /24 range", + wl.is_whitelisted( "::ffff:5.252.102.106" ), true ) + eq( "v4-mapped v6 outside v4 /24 range", + wl.is_whitelisted( "::ffff:5.252.103.1" ), false ) +end + +---------------------------------------------------------------------- +-- expires_at +---------------------------------------------------------------------- + +reset_state{} +do + wl.add( "11.22.33.0/24", { source = "manual", expires_at = 500 } ) -- already past + eq( "expired (past) not whitelisted", wl.is_whitelisted( "11.22.33.50" ), false ) + + wl.add( "44.55.66.0/24", { source = "manual", expires_at = 9999 } ) + eq( "future expiry still whitelisted", wl.is_whitelisted( "44.55.66.50" ), true ) + _now = 10000 + eq( "expires when now passes expires_at", wl.is_whitelisted( "44.55.66.50" ), false ) +end + +---------------------------------------------------------------------- +-- remove +---------------------------------------------------------------------- + +reset_state{} +do + local _, id1 = wl.add( "100.0.0.0/8", { source = "manual" } ) + wl.add( "101.0.0.0/8", { source = "manual" } ) + eq( "remove ok", wl.remove( id1 ), true ) + eq( "count after remove", wl.count( ).total, 1 ) + local ok, err = wl.remove( 99999 ) + eq( "remove not_found false", ok, false ) + eq( "remove not_found err", err, "not_found" ) + eq( "removed no longer whitelisted", wl.is_whitelisted( "100.9.9.9" ), false ) + eq( "remaining still whitelisted", wl.is_whitelisted( "101.9.9.9" ), true ) +end + +---------------------------------------------------------------------- +-- reload from stub disk: persistence + cache rebuild +---------------------------------------------------------------------- + +reset_state{} +do + wl.add( "55.66.77.0/24", { source = "pinger", reason = "reload test" } ) + wl.add( "2001:db8::/32", { source = "manual" } ) + + _G.use = _stub_use_factory( ) + local wl2 = assert( loadfile( "core/whitelist.lua" ) )( ) + wl2.init( ) + eq( "post-reload count", wl2.count( ).total, 2 ) + eq( "post-reload v4 match", wl2.is_whitelisted( "55.66.77.99" ), true ) + eq( "post-reload v6 match", wl2.is_whitelisted( "2001:db8::1" ), true ) + eq( "post-reload non-match", wl2.is_whitelisted( "1.2.3.4" ), false ) + eq( "post-reload source label survives", wl2.list{ source = "pinger" }[ 1 ].reason, "reload test" ) +end + +---------------------------------------------------------------------- +-- fresh-install: reload() with no .tbl is silent + clean +---------------------------------------------------------------------- + +reset_state{} +do + local err_count = 0 + _G.use = function( name ) + if name == "out" then + return { put = function( ) end, error = function( ) err_count = err_count + 1 end } + end + return _stub_use_factory( )( name ) + end + local wl_fresh = assert( loadfile( "core/whitelist.lua" ) )( ) + wl_fresh.init( ) + eq( "fresh-install reload emits no out.error", err_count, 0 ) + eq( "fresh-install reload zero entries", wl_fresh.count( ).total, 0 ) + _G.use = _stub_use_factory( ) +end + +---------------------------------------------------------------------- +-- Engine disabled: is_whitelisted false even with entries present +---------------------------------------------------------------------- + +reset_state{ cfg = { whitelist_enabled = false } } +do + wl.add( "1.2.3.0/24", { source = "manual" } ) + eq( "disabled engine is_whitelisted false", wl.is_whitelisted( "1.2.3.50" ), false ) +end + +---------------------------------------------------------------------- +-- Empty-store fast path + nil/garbage input +---------------------------------------------------------------------- + +reset_state{} +do + eq( "empty store is_whitelisted false", wl.is_whitelisted( "1.2.3.4" ), false ) + eq( "nil input false", wl.is_whitelisted( nil ), false ) + wl.add( "1.2.3.0/24", { source = "manual" } ) + eq( "garbage input false", wl.is_whitelisted( "not-an-ip" ), false ) + eq( "empty string input false", wl.is_whitelisted( "" ), false ) +end + +---------------------------------------------------------------------- +-- _resolve_match direct + hex round-trip +---------------------------------------------------------------------- + +reset_state{} +do + wl.add( "8.8.8.0/24", { source = "manual", reason = "dns" } ) + local m = wl._resolve_match( "8.8.8.8" ) + truthy( "_resolve_match returns entry", m ) + eq( "_resolve_match entry source", m and m.source, "manual" ) + eq( "_resolve_match out-of-range nil", wl._resolve_match( "9.9.9.9" ), nil ) + eq( "_resolve_match garbage nil", wl._resolve_match( "garbage" ), nil ) + eq( "_resolve_match nil nil", wl._resolve_match( nil ), nil ) + + local samples = { "\1\2\3\4", "\xff\x00\xab\xcd", string.rep( "\0", 16 ) } + for _, s in ipairs( samples ) do + eq( "hex round-trip (len " .. #s .. ")", wl._hex_decode( wl._hex_encode( s ) ), s ) + end + eq( "hex_decode odd length nil", wl._hex_decode( "abc" ), nil ) +end + +---------------------------------------------------------------------- +-- INTEGRATION: whitelist <-> blocklist Model-A precedence. +-- +-- Load BOTH engines with a shared use-stub where blocklist's +-- `use "whitelist"` resolves to the live whitelist module. Separate +-- store paths (blocklist default / whitelist default). This exercises +-- the check_ip hook: whitelist overrides an AUTOMATED block, a manual +-- pin still wins. The "automated block + whitelisted -> allowed" case +-- FAILS on a blocklist without the hook (§1a.7). +---------------------------------------------------------------------- + +do + _disk = { } + _now = 1000 + _save_count = 0 + _fail_save = false + + local BL_PATH = "scripts/data/etc_blocklist.tbl" + local WL_PATH = "scripts/data/etc_whitelist.tbl" + local wl_mod, bl_mod + + local function int_use( name ) + if name == "whitelist" then return wl_mod end + -- everything else via the standard stub + return _stub_use_factory( )( name ) + end + + _G.use = int_use + wl_mod = assert( loadfile( "core/whitelist.lua" ) )( ) + bl_mod = assert( loadfile( "core/blocklist.lua" ) )( ) + wl_mod.init( ) + bl_mod.init( ) -- captures wl_mod.is_whitelisted via `use "whitelist"` + + -- 1. AUTOMATED block (external feed) on an IP; NOT whitelisted -> blocked. + bl_mod.add( "203.0.113.10", { source = "external", reason = "tor" } ) + eq( "automated block, not whitelisted -> blocked", + bl_mod.check_ip( "203.0.113.10" ), true ) + + -- 2. Whitelist that same IP -> the automated block is overridden. + -- THIS is the assertion that fails on the unpatched blocklist. + wl_mod.add( "203.0.113.10", { source = "manual", reason = "trusted" } ) + eq( "automated block + whitelisted -> ALLOWED (hook)", + bl_mod.check_ip( "203.0.113.10" ), false ) + + -- 3. A MANUAL pin on a whitelisted IP still wins (deliberate block). + bl_mod.add( "203.0.113.20", { source = "manual", reason = "operator ban" } ) + wl_mod.add( "203.0.113.20", { source = "manual", reason = "also trusted" } ) + eq( "manual pin + whitelisted -> STILL blocked (manual wins)", + bl_mod.check_ip( "203.0.113.20" ), true ) + + -- 4. Whitelisted range covering an automated /24 feed entry. + bl_mod.add( "198.51.100.0/24", { source = "geoip", reason = "CN" } ) + eq( "automated /24, mid-range not whitelisted -> blocked", + bl_mod.check_ip( "198.51.100.5" ), true ) + wl_mod.add( "198.51.100.0/24", { source = "pinger" } ) + eq( "automated /24 fully whitelisted -> allowed", + bl_mod.check_ip( "198.51.100.5" ), false ) + + -- 5. v4-mapped-v6 client is whitelisted against a plain-v4 allow + -- entry even though the block entry is also plain v4. + bl_mod.add( "192.0.2.77", { source = "external" } ) + eq( "mapped-v6 automated block pre-whitelist -> blocked", + bl_mod.check_ip( "::ffff:192.0.2.77" ), true ) + wl_mod.add( "192.0.2.77", { source = "pinger" } ) + eq( "mapped-v6 automated block + v4 whitelist -> allowed", + bl_mod.check_ip( "::ffff:192.0.2.77" ), false ) + + -- 6. Whitelisted IP that is NOT blocked at all -> still just allowed + -- (no crash, no false-block). + eq( "whitelisted but not blocked -> allowed", + bl_mod.check_ip( "203.0.113.10" ), false ) +end + +---------------------------------------------------------------------- + +if _fails > 0 then + io.stderr:write( string.format( "\nFAIL: %d/%d checks failed\n", + _fails, _passes + _fails ) ) + os.exit( 1 ) +end +print( string.format( "OK: %d checks passed", _passes ) )