Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# be-stream-downloader
# belgarr

VRT MAX + VTM GO + GoPlay + Streamz downloader, internal-only. UI at `https://bedl.${SECRET_DOMAIN}`.
VRT MAX + VTM GO + GoPlay + Streamz downloader, internal-only. UI at `https://belgarr.${SECRET_DOMAIN}`.
Per-provider routing in `app/web/main.py:_provider_for` — host suffix decides which `*-DL.py` script runs.

Downloads run **serially** (`BEDL_MAX_CONCURRENT_DOWNLOADS=1` default; one n-m3u8dl-re saturates the household uplink alone).
Downloads run **serially** (`BELG_MAX_CONCURRENT_DOWNLOADS=1` default; one n-m3u8dl-re saturates the household uplink alone).

Auto-DL scheduler (FastAPI lifespan task) ticks every `BEDL_AUTO_DL_INTERVAL_SECONDS` (default `3600`). Per-show toggle in the UI; cold-start is baseline-from-current (only future-published episodes get queued). Quality-upgrade pass re-pulls any Plex-present episode whose video height is below `BEDL_TARGET_HEIGHT` (default `1080`). Multi-show ticks queue serially through the same semaphore as user-clicked downloads.
Auto-DL scheduler (FastAPI lifespan task) ticks every `BELG_AUTO_DL_INTERVAL_SECONDS` (default `3600`). Per-show toggle in the UI; cold-start is baseline-from-current (only future-published episodes get queued). Quality-upgrade pass re-pulls any Plex-present episode whose video height is below `BELG_TARGET_HEIGHT` (default `1080`). Multi-show ticks queue serially through the same semaphore as user-clicked downloads.

Source: <https://github.com/Varashi/be-stream-downloader>
Source: <https://github.com/Varashi/belgarr>

## Layout

Expand All @@ -16,23 +16,23 @@ ks.yaml Flux Kustomization
app/
namespace.yaml
externalsecret.yaml VRT + Streamz + GoPlay creds, VTM GO cookies, Plex token, GHCR pull secret, .wvd Secret
pvc.yaml vsan PVC be-stream-downloader-data (10 Gi)
pvc.yaml vsan PVC belgarr-data (10 Gi)
helmrelease.yaml bjw-s app-template, digest-pinned image
kustomization.yaml
```

## Image

Pinned by digest in `helmrelease.yaml` — `ghcr.io/varashi/be-stream-downloader:<git-sha>@sha256:<manifest-digest>`.
Pinned by digest in `helmrelease.yaml` — `ghcr.io/varashi/belgarr:<git-sha>@sha256:<manifest-digest>`.

To bump: pull `:latest` after a CI build, copy the digest from `podman image inspect ... --format '{{.Digest}}'` plus `git rev-parse main` from the source repo, edit the `tag:` field, push.

## Volumes

| Mount | Source |
|---|---|
| `/data` | PVC `be-stream-downloader-data` (vsan, RWO, 10Gi). Holds seeded app code + `state.json` + Streamz token cache at `/data/.config/streamz/tokens.json` |
| `/wvd/cdm.wvd` | Secret `be-stream-downloader-wvd` projected from BW SM `SECRET_BEDL_WVD` (base64 of the `.wvd` file). Same CDM serves all four providers |
| `/data` | PVC `belgarr-data` (vsan, RWO, 10Gi). Holds seeded app code + `state.json` + Streamz token cache at `/data/.config/streamz/tokens.json` |
| `/wvd/cdm.wvd` | Secret `belgarr-wvd` projected from BW SM `SECRET_BELG_WVD` (base64 of the `.wvd` file). Same CDM serves all four providers |
| `/media` | NFS `${SECRET_NFS_HOST}:/mnt/DATA/mediapool/media` — mounted at the *parent* of all libraries so the per-show `library` override can target sibling subtrees (`Series`, `Movies`, `MoviesNL`, …) |

## Secrets (BW SM via ESO)
Expand All @@ -47,18 +47,18 @@ To bump: pull `:latest` after a CI build, copy the digest from `podman image ins
| `SECRET_GOPLAY_EMAIL` | env `GOPLAY_EMAIL` | GoPlay (play.tv) login — AWS Cognito user pool. Currently mirrors VRT credentials but kept as a separate BW SM key so they can diverge. |
| `SECRET_GOPLAY_PASSWORD` | env `GOPLAY_PASSWORD` | GoPlay password |
| `SECRET_PLEX_TOKEN` | env `PLEX_TOKEN` | Plex API auth |
| `SECRET_BEDL_WVD` | Secret data `cdm.wvd` | Widevine CDM device file (base64 of binary) |
| `SECRET_BELG_WVD` | Secret data `cdm.wvd` | Widevine CDM device file (base64 of binary) |
| `SECRET_GHCR_PULL_TOKEN` | dockerconfigjson `ghcr-pull-secret` | private GHCR pull |

`PLEX_URL` and `MEDIA_LIBRARY_DEFAULT` are set as plain env in the HelmRelease, not secrets.

VRT re-logs every subprocess call. Streamz logs in once and persists the LFVP cookie envelope to `/data/.config/streamz/tokens.json`; Streamz Next.js silently rotates the inner access_token via popcorn-sdk's confidential client_secret, so the cookie is good for ~365 days. `streamz_auth.invalidate()` drops the cache on a real 401 to force re-login.

Streamz Phase 2 shipped 2026-04-28: the Quick Download flow now goes end-to-end (PSSH from MPD → DRMtoday Widevine license at `lic.drmtoday.com/license-proxy-widevine/cenc/?specConform=true` → `n-m3u8dl-re` decrypt + MKV mux → Dutch VTT subtitle → SRT mux). `STREAMZ-DL.py` ships two CDM classes: `Local_CDM` (default, uses `BEDL_WVD`) and `GetWVKeys_CDM` (cold backup via getwvkeys.cc, gated on env `WV_TOKEN` / optional `WV_BUILDINFO`/`WV_URL`). Streamz library scraper (`app/scrapers/streamz.py`) shipped 2026-04-28 too — Add Show accepts `streamz.be/streamz/<slug>~<uuid>` URLs.
Streamz Phase 2 shipped 2026-04-28: the Quick Download flow now goes end-to-end (PSSH from MPD → DRMtoday Widevine license at `lic.drmtoday.com/license-proxy-widevine/cenc/?specConform=true` → `n-m3u8dl-re` decrypt + MKV mux → Dutch VTT subtitle → SRT mux). `STREAMZ-DL.py` ships two CDM classes: `Local_CDM` (default, uses `BELG_WVD`) and `GetWVKeys_CDM` (cold backup via getwvkeys.cc, gated on env `WV_TOKEN` / optional `WV_BUILDINFO`/`WV_URL`). Streamz library scraper (`app/scrapers/streamz.py`) shipped 2026-04-28 too — Add Show accepts `streamz.be/streamz/<slug>~<uuid>` URLs.

Auto-DL scheduler shipped 2026-04-28 with two env knobs: `BEDL_AUTO_DL_INTERVAL_SECONDS` (default 3600, min 60) sets the tick cadence; `BEDL_TARGET_HEIGHT` (default 1080) caps the quality-upgrade target. Both default values are sane; the only reason to bump them is if Belgian streamers ever start serving above 1080p.
Auto-DL scheduler shipped 2026-04-28 with two env knobs: `BELG_AUTO_DL_INTERVAL_SECONDS` (default 3600, min 60) sets the tick cadence; `BELG_TARGET_HEIGHT` (default 1080) caps the quality-upgrade target. Both default values are sane; the only reason to bump them is if Belgian streamers ever start serving above 1080p.

UI hardening shipped 2026-04-28 alongside the scheduler: per-job log buffer is bounded by `BEDL_LOG_BUFFER_LINES` (default 4000) so a long-running download doesn't grow an unbounded list; n-m3u8dl-re progress lines are coalesced and benign h264 SEI/PPS warnings are filtered to keep the event loop snappy; the `/jobs/{id}/stream` SSE generator emits a 15 s `: keepalive` comment line so queued jobs don't show "stream lost" while waiting at the serial-execution semaphore. Show-page UI now ships collapsible seasons (native `<details>`/`<summary>`) plus cross-season `all/missing/none` selection shortcuts.
UI hardening shipped 2026-04-28 alongside the scheduler: per-job log buffer is bounded by `BELG_LOG_BUFFER_LINES` (default 4000) so a long-running download doesn't grow an unbounded list; n-m3u8dl-re progress lines are coalesced and benign h264 SEI/PPS warnings are filtered to keep the event loop snappy; the `/jobs/{id}/stream` SSE generator emits a 15 s `: keepalive` comment line so queued jobs don't show "stream lost" while waiting at the serial-execution semaphore. Show-page UI now ships collapsible seasons (native `<details>`/`<summary>`) plus cross-season `all/missing/none` selection shortcuts.

rc-check + orphan sweep deployed 2026-04-28 across STREAMZ-DL / VTMGO-DL / GOPLAY-DL — `n-m3u8dl-re`'s exit code is now respected and any half-muxed `<save>.mp4`/`.m4a`/`.srt`/`.ts` stems on a crashed run get cleaned up before the wrapper raises. No more orphans masquerading as legitimate Plex content.

Expand All @@ -80,20 +80,20 @@ When VTM GO scraping or DLs start returning 401/403:
(BW SM secret ID for `SECRET_VTMGO_COOKIES`.)
5. Force pod refresh — ESO polls every 1h, so for immediate effect:
```sh
kubectl rollout restart deploy/be-stream-downloader -n be-stream-downloader
kubectl rollout restart deploy/belgarr -n belgarr
```

Cookies typically last a few days. Watch for `[VTM GO] HTTP 4xx` or `cookies expired?` lines in the pod log as the trigger.

## Networking

HTTPRoute on the shared `cilium/main` gateway, hostname `bedl.${SECRET_DOMAIN}`. No `external-dns.alpha.kubernetes.io/public: "true"` annotation → internal AD DNS only, not published to Cloudflare.
HTTPRoute on the shared `cilium/main` gateway, hostname `belgarr.${SECRET_DOMAIN}`. No `external-dns.alpha.kubernetes.io/public: "true"` annotation → internal AD DNS only, not published to Cloudflare.

Homepage tile under group `Media`.

## Resources

`limits: 2 CPU / 2 GiB` (sized for serial downloads — one n-m3u8dl-re peaks ~1.5 cores + ~600 MiB during a 4 Mbps DASH decode, plus headroom for a concurrent show refresh and the FastAPI event loop). `requests: 100m / 256Mi`. If you ever raise `BEDL_MAX_CONCURRENT_DOWNLOADS` above 1, bump these accordingly — earlier 4c/4Gi was needed for parallel batches of 7-9 concurrent encoders.
`limits: 2 CPU / 2 GiB` (sized for serial downloads — one n-m3u8dl-re peaks ~1.5 cores + ~600 MiB during a 4 Mbps DASH decode, plus headroom for a concurrent show refresh and the FastAPI event loop). `requests: 100m / 256Mi`. If you ever raise `BELG_MAX_CONCURRENT_DOWNLOADS` above 1, bump these accordingly — earlier 4c/4Gi was needed for parallel batches of 7-9 concurrent encoders.

## Pod placement

Expand All @@ -104,4 +104,4 @@ No `nodeSelector` — vsan PVC binds wherever the pod schedules (WaitForFirstCon
- **Reseed `/data/app/` on image upgrade** is automatic since PR #5 (entrypoint uses `cp -rf`); no `kubectl exec rm` dance needed anymore.
- **Plex hooks** are best-effort. `/plex/status` reports connection health.
- **State migration**: state.json is auto-migrated on read (legacy entries get keys + override fields backfilled). No data loss across image upgrades.
- **GHCR pull token rotation**: the BW SM entry was revoked once via GitHub secret scanning. `SECRET_GHCR_PUSH_TOKEN` worked as a fallback for both push and pull while the read-only token was being regenerated. Generate a fine-grained PAT scoped to the `be-stream-downloader` repo with `Packages: Read` only.
- **GHCR pull token rotation**: the BW SM entry was revoked once via GitHub secret scanning. `SECRET_GHCR_PUSH_TOKEN` worked as a fallback for both push and pull while the read-only token was being regenerated. Generate a fine-grained PAT scoped to the `belgarr` repo with `Packages: Read` only.
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: be-stream-downloader-secrets
namespace: be-stream-downloader
name: belgarr-secrets
namespace: belgarr
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: bitwarden-secretsmanager
target:
name: be-stream-downloader-secrets
name: belgarr-secrets
creationPolicy: Owner
data:
# Provider creds are intentionally NOT mapped here — testing the
Expand All @@ -25,15 +25,15 @@ spec:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: be-stream-downloader-wvd
namespace: be-stream-downloader
name: belgarr-wvd
namespace: belgarr
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: bitwarden-secretsmanager
target:
name: be-stream-downloader-wvd
name: belgarr-wvd
creationPolicy: Owner
template:
type: Opaque
Expand All @@ -46,13 +46,13 @@ spec:
data:
- secretKey: wvd
remoteRef:
key: SECRET_BEDL_WVD
key: SECRET_BELG_WVD
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: ghcr-pull-secret
namespace: be-stream-downloader
namespace: belgarr
spec:
refreshInterval: 1h
secretStoreRef:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: be-stream-downloader
namespace: be-stream-downloader
name: belgarr
namespace: belgarr
spec:
interval: 30m
chart:
Expand All @@ -25,13 +25,13 @@ spec:
imagePullSecrets:
- name: ghcr-pull-secret
controllers:
be-stream-downloader:
belgarr:
containers:
app:
image:
# renovate: datasource=docker depName=ghcr.io/varashi/be-stream-downloader
repository: ghcr.io/varashi/be-stream-downloader
tag: v0.5.0@sha256:9a0159fc1f1458dea69def98f9ea3edbb139ada0ad1a367e7c251cbd018d1785
# renovate: datasource=docker depName=ghcr.io/varashi/belgarr
repository: ghcr.io/varashi/belgarr
tag: v0.6.0@sha256:1ec166fffab53fb86ddf2767345263032fb3a974bdfc8e61d4de679e7fc6b6f7
pullPolicy: IfNotPresent
env:
# Image default is /downloads; remap because we mount the
Expand All @@ -50,16 +50,16 @@ spec:
# (L1). Local L3 WVD gets HTTP 403 — route through the public
# getwvkeys.cc anonymous remote-CDM pool. Privacy note: every
# license challenge identifies playback to that operator.
BEDL_GOPLAY_USE_REMOTE_CDM: "1"
BELG_GOPLAY_USE_REMOTE_CDM: "1"
envFrom:
- secretRef:
name: be-stream-downloader-secrets
name: belgarr-secrets
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
# Serial-only execution (BEDL_MAX_CONCURRENT_DOWNLOADS=1):
# Serial-only execution (BELG_MAX_CONCURRENT_DOWNLOADS=1):
# one n-m3u8dl-re peaks ~1.5 cores + ~600 MiB during a 4 Mbps
# DASH decode; this leaves headroom for a concurrent show
# refresh scrape and the FastAPI event loop without slack.
Expand All @@ -86,21 +86,21 @@ spec:
periodSeconds: 10
service:
app:
controller: be-stream-downloader
controller: belgarr
ports:
http:
port: 8000
route:
app:
annotations:
gethomepage.dev/enabled: "true"
gethomepage.dev/name: BeStreamDL
gethomepage.dev/name: Belgarr
gethomepage.dev/description: VRT MAX + VTM GO downloader
gethomepage.dev/group: Media
gethomepage.dev/weight: "60"
gethomepage.dev/href: https://bedl.${SECRET_DOMAIN}
gethomepage.dev/href: https://belgarr.${SECRET_DOMAIN}
hostnames:
- bedl.${SECRET_DOMAIN}
- belgarr.${SECRET_DOMAIN}
parentRefs:
- name: main
namespace: cilium
Expand All @@ -119,12 +119,12 @@ spec:
# PVC pre-created in pvc.yaml (vsan); using existingClaim avoids
# bjw-s reusing the previous longhorn PVC name and trying to
# mutate its immutable spec.
existingClaim: be-stream-downloader-data
existingClaim: belgarr-data
globalMounts:
- path: /data
wvd:
type: secret
name: be-stream-downloader-wvd
name: belgarr-wvd
defaultMode: 0400
globalMounts:
- path: /wvd
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
apiVersion: v1
kind: Namespace
metadata:
name: be-stream-downloader
name: belgarr
labels:
pod-security.kubernetes.io/enforce: baseline
pod-security.kubernetes.io/warn: baseline
Expand Down
17 changes: 17 additions & 0 deletions cluster-talos/kubernetes/apps/media/belgarr/app/pvc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: belgarr-data
namespace: belgarr
spec:
# Rebind the existing PV from the old be-stream-downloader-data PVC so the
# camoufox profile + provider token caches survive the namespace rename.
# Phase C clears the old claimRef before flux applies this manifest.
volumeName: pvc-b6adea0b-f241-4915-b9ed-02e68db3f38a
accessModes:
- ReadWriteOnce
storageClassName: vsan
resources:
requests:
storage: 10Gi
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: be-stream-downloader
name: belgarr
namespace: flux-system
spec:
interval: 1h
path: ./cluster-talos/kubernetes/apps/media/be-stream-downloader/app
path: ./cluster-talos/kubernetes/apps/media/belgarr/app
prune: false
sourceRef:
kind: GitRepository
Expand Down
2 changes: 1 addition & 1 deletion cluster-talos/kubernetes/apps/media/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- be-stream-downloader/ks.yaml
- belgarr/ks.yaml
- media-toolkit/ks.yaml
- ombi/ks.yaml
- plex/ks.yaml
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ spec:
- name: RENOVATE_PLATFORM
value: github
- name: RENOVATE_REPOSITORIES
value: '["Varashi/k8s", "Varashi/scaleplex", "Varashi/be-stream-downloader", "Varashi/gpu-node-vsphere-maintenance-controller"]'
value: '["Varashi/k8s", "Varashi/scaleplex", "Varashi/belgarr", "Varashi/gpu-node-vsphere-maintenance-controller"]'
- name: RENOVATE_GIT_AUTHOR
value: "Renovate Bot <renovate@${SECRET_DOMAIN}>"
- name: LOG_LEVEL
Expand Down
Loading