Consolidate all hosting on Hetzner (greenfield: backend + Icecast, retire NMS + Lightsail)

[anneoneone]

Goal

Stand up a clean, greenfield Hetzner deployment that hosts everything — the backend and the audio streaming stack — on Hetzner Cloud, and retire both the AWS Lightsail backend and the NodeMediaServer relay. No legacy dependencies carried forward (no NMS, no FLV/HLS-from-NMS, no AWS). Cheaper bandwidth, one provider, co-located components, and the unpatched 960-day-uptime relay box gone.

This is the umbrella for the consolidation. Move A (lift-and-shift the backend) already shipped as tooling; this ticket covers the full greenfield target and absorbs/reframes the Phase 2–4 streaming work.

Why

• Cost: Hetzner includes ~20 TB/mo egress per server (vs. metered AWS); compute is far cheaper. Streaming + recording bandwidth becomes effectively free at our scale (~300 peak listeners ≈ 38 Mbps).
• Simplicity / SPOF: one provider, co-located backend + producer + Icecast (localhost hops, no cross-cloud RTMP). Retire the ancient relay VM (kernel 5.4, 960-day uptime, unpatched).
• iOS: validated — MP3-over-Icecast plays natively in iOS Safari (see Phase-2 harness). NMS's HLS/FLV split is no longer needed.

Target architecture (greenfield, no NMS)

Broadcaster browser ──WS /ws/stream──▶ Backend (Hetzner, ffmpeg WebM/Opus→AAC)
                                              │ localhost
                                              ▼
                                        Liquidsoap (Hetzner) ──▶ Icecast-KH (Hetzner)
                                                                      │ nginx + certbot TLS
                                                                      ▼
                                                          listeners (MP3/AAC <audio>, incl. iOS)
   recordings ─────────────────────────────────────────────────────▶ Cloudflare R2 (unchanged)

Open design decision — how the backend feeds the producer (no NMS):

• (A) Backend ffmpeg outputs directly to Icecast (-f mp3 icecast://…). Simplest; drops Liquidsoap. Loses fallback-silence/scheduling.
• (B) Backend ffmpeg → Liquidsoap harbor (Icecast SOURCE input) → Liquidsoap mksafe → Icecast. Keeps the always-up mount + room for jingles/scheduling. Preferred.
• (C) Keep a thin RTMP ingest (nginx-rtmp) so the existing backend RTMP push code is unchanged, Liquidsoap pulls localhost. Most code-compatible, but reintroduces an RTMP server.

Recommend (B); decide during implementation. The backend RTMP target is already config (RTMP_URL/RTMP_STREAM_KEY).

Scope / tasks

• Provision fresh Hetzner Cloud box(es) (Ubuntu 24.04, CPX21/CX22+). Decide single-box vs. backend+stream split. Cloud Firewall: 22/80/443 (+ Icecast/RTMP as needed). Enable snapshots.
• Backend deploy on Hetzner via backend/scripts/deploy_hetzner.sh (done as tooling) + SQLite cutover per docs/stream-rework/backend-hetzner-migration.md.
• Streaming stack on the same box: Icecast-KH + Liquidsoap (producer per decision A/B/C above), MP3/AAC mount only (iOS — never Opus/Ogg).
• nginx + certbot TLS for the public mount (e.g. radio.live.moafunk.de) and the admin domain.
• Repoint the backend producer to localhost; remove the NMS dependency end-to-end.
• Frontend: collapse VITE_STREAM_HLS_URL + VITE_STREAM_FLV_URL → a single Icecast mount URL (env flip).
• CI: add HETZNER_IP/HETZNER_SSH_KEY Bitwarden secrets; switch .github/workflows/backend.yml deploy step to deploy_hetzner.sh.
• Bandwidth/SPOF: confirm Hetzner egress headroom; decide if a CDN/relay is needed (≈38 Mbps at peak — see [spike] Phase 0: pin NMS version, peak concurrent listeners, iOS reach #168). Cheapest scale lever remains HLS-from-CDN if egress ever bites, but Icecast-on-Hetzner covers 300 directly.
• Soak both old + new in parallel through real shows, then decommission Lightsail + the NMS relay (final snapshots/backups first).

Relationship to existing issues

• Absorbs/reframes the Phase 2–4 streaming issues so they target Hetzner with no NMS:
  • Icecast-KH + Liquidsoap stack: /live + /test mounts, separate creds, TLS front #173 (Icecast-KH + Liquidsoap) → on the Hetzner box, fed by the backend (not pulling NMS).
  • Producer: configurable ffmpeg push target + Icecast output, tee kept independent #174 (configurable producer + Icecast output) → the backend→producer decision above.
  • Broadcaster preview player pointed at /test mount #175 (broadcaster preview player) → unchanged, points at the new mount.
  • Cutover: soak-test, flip push target test→live, switch public player HLS/FLV→Icecast #176 (cutover) → now also includes the backend host cutover + NMS/Lightsail teardown.
  • Observability: listener/quality telemetry → backend + /api/stream/metrics + admin display #177/Observability: Prometheus/Grafana/Alertmanager/Blackbox + alerts + systemd/SPOF hardening #178 (observability/SPOF) → on Hetzner.
• [spike] Phase 0: pin NMS version, peak concurrent listeners, iOS reach #168 (NMS-version spike) becomes moot for the target (NMS retired), but its listener-count/iOS findings still scope this.
• Parent: Stream rework: harden R2 recording + migrate relay node-media-server → Icecast-KH/Liquidsoap (audio-only) #164. Builds on the validated Phase-2 harness + runbook (docs/stream-rework/).

Acceptance

• Backend serves prod from Hetzner (admin domain green over TLS), real DB migrated, recordings still land in R2.
• Live audio flows broadcaster → backend → producer → Icecast → listeners (incl. a real iPhone), TLS, no NMS in the path.
• Lightsail + the NMS relay are powered off and deleted after a successful soak.
• CI deploys to Hetzner.

Docs

• docs/stream-rework/backend-hetzner-migration.md (Move A cutover)
• docs/stream-rework/phase-2-icecast-runbook.md (Icecast/Liquidsoap, Tier-2 relay sequence — adapt host targets to the Hetzner box)
• docs/stream-rework/local-test-harness/ (validated end-to-end, incl. iOS)