feat(queue): offline operation queue for all panel integrations

[0kaba0hub]

Problem

When the SecondDNS API is down (outage or maintenance), zone operations fired by customers' panels (Plesk/cPanel/DirectAdmin/CyberPanel hooks) were logged and dropped — zones created during our downtime never reached SecondDNS.

Solution

Local offline queue on the panel server:

• hosting-panels/common/seconddns-queue — enqueue/flush/status tool backed by SQLite (/var/lib/seconddns/queue.db, WAL + busy_timeout; flock with a portable fallback)
• Queue-first hooks: all six bash hooks enqueue the operation and trigger an async flush — strict FIFO is preserved even when older ops are already waiting; Plesk rename enqueues delete(old)+create(new)
• Cron replay: /etc/cron.d/seconddns-queue flushes every minute; once the API is back everything drains automatically in order
• Replay semantics: 409 on create / 404 on delete = success (safe duplicate delivery); 400/422 → row marked failed, drain continues; timeout/5xx/redirect → drain stops until next tick
• CyberPanel: network errors are now caught (previously an uncaught URLError); retryable failures enqueue via the same tool
• Monitoring: nagios_plugins/check_seconddns_queue.sh (WARN/CRIT by oldest-op age, perfdata) and zabbix_templates/seconddns_queue.yaml (items + triggers via UserParameter)
• Installers deploy the tool + cron; uninstallers clean up; README documents the feature

Tests

hosting-panels/common/tests/test_queue.sh — self-contained suite with a scripted mock API (14 checks, all passing): enqueue persistence, drain stop on API-down, FIFO order, 409/404 duplicate semantics, hard-error isolation, status exit codes, 20-way concurrent enqueue.

Live E2E against the real maintenance switch (k8s_web scripts/maintenance-on.sh) is the next step per the test plan in seconddns/k8s_web#265.

Ref: seconddns/k8s_web#265

[0kaba0hub]

Reworked to the daemon model per review discussion:

• seconddns-queued — python3 (stdlib-only) systemd worker, the single delivery path: watches the queue, strict FIFO, exponential backoff (backoff_min→backoff_max) while the API is down; configurable via an optional [queue] section in /etc/seconddns.conf (poll_interval, http_timeout, backoff_min, backoff_max); --once mode backs the manual seconddns-queue flush
• Hooks are enqueue-only — a single local INSERT, so panel operations never block on API timeouts (previously up to 15 s even when healthy); no per-minute cron anywhere
• CyberPanel unified: add_zone/remove_zone enqueue-only; list/sync reads stay direct
• Installers deploy the worker + unit (systemctl enable --now seconddns-queued); uninstallers stop and remove it
• Tests: 16/16 passing, including daemon-mode pickup while the API is down and automatic drain after recovery

Cross-server migration semantics (ownership check by master IP) intentionally NOT included — tracked separately in #63.