feat: optional kernel SYN rate-limiter + per-endpoint DC connect timeout

[sleep3r]

Two improvements drawn from analysing MTproxy-reanimation (a tuning wrapper for Telemt/MTProxyMax). That project is not a proxy fork — its value is OS/network-level techniques applied around a proxy. Of its five techniques, only two are worth porting; the other three we already do better or they conflict with our invariants (see "Not ported" below). Each kept technique was adversarially verified against our actual code.

---

1. mtbuddy setup syn-limit — optional kernel SYN rate-limiter (T1)

A default-OFF, per-source-IP inbound SYN rate-limiter on the proxy port via iptables -m hashlimit, run as a separate systemd oneshot (mtproto-syn-limit.service).

Why it's not redundant with our in-proxy guards: the handshake flood guard and per-/24 subnet limiter both run after accept() and both default OFF (NAT/VPN false-positives). So today every abusive SYN still costs a kernel socket + an accept() syscall. A kernel hashlimit drops excess SYNs pre-accept — a different layer, complementary not duplicate.

Design choices

• iptables hashlimit, not nftables — matches our existing stack (TCPMSS, nfqws); no new dependency.
• Separate oneshot unit so CAP_NET_ADMIN never has to be granted to mtproto-proxy.
• Default OFF behind a loud CGNAT/VPN warning (same shared-egress-IP problem that keeps the in-proxy guards off). Warns when accept_proxy_protocol is set (kernel sees the LB IP, not clients).
• Presets mirror the source: soft 2/s burst 5 (default when enabling, CGNAT-safer), medium 1/s burst 3, hard 1/s burst 1.
• Drop counter in mtbuddy status; verifies the rule actually landed (xt_hashlimit present) instead of a silent no-op.
• Idempotent apply + full uninstall cleanup (replay-delete INPUT jumps → flush → delete chain, v4 & v6). Generated script is brace-free (so std.fmt renders it) and rate/burst/port are validated before being baked in.

mtbuddy setup syn-limit --preset soft        # enable (CGNAT-safer default)
mtbuddy setup syn-limit --rate 1/second --burst 3
mtbuddy setup syn-limit --remove
mtbuddy setup syn-limit --status

2. dc_connect_timeout_sec — per-endpoint DC connect timeout (T2)

Default 10s. A filtered/black-holed DC endpoint sends no RST, so the kernel sits in SYN_SENT ~2 min. handshake_timeout_sec (15s from the client's first byte) already caps the whole handshake — so we are not exposed to an unbounded hang — but that cap is global, so a slow first endpoint starves the failover budget for the rest. This fails one dead endpoint fast and lets failover advance to the next, within the overall handshake ceiling.

Mirrors onUpstreamConnectComplete's failure path exactly (cleanupFailedUpstreamConnect → tryNextDcEndpoint for .dc, else closeSlot), driven from the timer tick. The deadline base is stamped per attempt and gated on phase == .connecting_upstream, so it can never touch an established relay; a healthy connect finishes in <1s, so working endpoints are unaffected. Deliberately does not raise handshake_timeout_sec (that would widen the active-probe/slow-loris window our DD-decision + flood guards exist to shrink).

---

Not ported (verified against our code)

• iOS keepalive sysctl (60/15/3): no-op for us — relay sockets already set SO_KEEPALIVE 60/10/3 + TCP_USER_TIMEOUT=30s (stricter), and a host-wide sysctl can't touch the pre-handshake sockets that lack SO_KEEPALIVE (already reaped by our idle/handshake timeouts).
• iOS MSS=92 + separate port DNAT: a second port = a second tg:// link (our links are immutable once distributed — hard no); the MSS half overlaps/conflicts with our existing TCPMSS=88; and it's a mechanism-free folk remedy for the iOS resume hang we already root-caused to the client MtProtoKit bad_server_salt bug (shipped client_silence_close_sec, filed Telegram-iOS#2197).
• Deployment autodetection + systemd persistence: already implemented for our own stack.

Testing

• zig build -Doptimize=ReleaseFast -Dtarget=x86_64-linux -Dcpu=x86_64_v3+aes ✅
• zig build test ✅ (new unit tests: SYN-limit preset/rate/number validation + brace-free script render; dc_connect_timeout_sec parse/default).
• Proxy core change is byte-transparent and off the happy path; the firewall feature is opt-in and isolated in its own unit.