Server-side source-manifest service (Lumbra's, Bundesliga) — decouple fragile URL discovery from app releases

[jozef2svrcek]

For sources whose file lists are fragile to resolve from the client — Lumbra's Gigabase (JS-rendered .7z download links) and Austrian Bundesliga (files scattered across pages, no system) — add a small server-side manifest to the existing LPDO data service (the one that already caches player normalization, normalise.lpdo.com).

Why

If scraping/URL-reconstruction lives in the shipped daemon, any source-site change breaks every installed copy until users update the app. Server-side, we fix the scraper once and all clients keep working — they just fetch a refreshed manifest. Also: politeness (we hit the source once, fan out a cached list), a place to validate URLs (HEAD-check, sizes, sha, dates), and reuse of existing infra.

Shape

• New endpoint(s): a JSON manifest per source — [{ label, url, covers:[from,to], size, sha256?, published }].
• Server runs source-specific resolvers on a schedule: Lumbra's (resolve the real wp-content/uploads/.../OTB_<era>_v<date>.7z URLs, curate to complete-year OTB), Bundesliga AT (crawl the scattered pages into a clean per-season file list).
• Client side: a generic "manifest-backed feed" driver — GET the manifest, map entries to FeedItems (with covers, reusing the B2 window file-skip), download each file straight from the origin, decompress, import. The service is a thin index, never a mirror.

Guardrails

• Thin index only — URLs + metadata, never proxy the bytes (esp. Lumbra's CC BY-NC-SA non-commercial + bandwidth).
• Not a hard dependency — ship a bundled last-known-good manifest in the app and cache the last fetched one, so a service outage only blocks discovery of new files, not import of known ones.
• Keep the high-level catalog compiled-in (defines what sources exist + how to acquire); only the volatile file list/URLs come from the manifest.
• TWIC + Lichess stay self-resolving (they publish clean indexes) — not routed through the service.

Sequencing

B3 ships Lumbra's with a bundled static manifest wired through the manifest-backed driver (proves 7z + bulk import). This issue then moves manifest generation to the service as a fast follow — the client code doesn't change, only where the manifest comes from (bundled → fetched-with-bundled-fallback). Relates to #40.

[jozef2svrcek]

Decision: deep-history sources via converted PGN on archive.org (not on-the-fly db3, not MEGA)

After working through the options, the plan for the deep-history bases:

Ajedrez Data OTB — already shipped (#96): public-domain, native PGN, plain HTTP, auto-download. The default deep-history base.

Caissabase 2024 & Lumbra's Gigabase — host converted PGN on archive.org:

1. One-off offline converter (throwaway tool, NOT shipped): decode En Croissant's .db3 → PGN. Both sources are available as En Croissant plain-HTTP db3 re-hosts (caissabase_2024.db3, LumbrasGigaBase2025-06.db3) — so this also avoids MEGA entirely for Lumbra's. The db3 move encoding is a shakmaty-legal_moves() index (we already depend on shakmaty 0.30); port En Croissant's src-tauri/src/db/encoding.rs decoder. Validate move-ordering parity on a few games.
2. Upload the compressed PGN to archive.org with metadata:
   • Lumbra's: CC BY-NC-SA 4.0 license tag + creator "Michael Jansen" + donation link + "non-commercial" note; thank-you to En Croissant for the db3.
   • Caissabase: provenance + thank-you to En Croissant (bare-scores, defunct).
3. Keep our own backup of the converted PGN (IA isn't absolutely permanent; preserves against En Croissant and IA going down).
4. App: plain-PGN catalog entries pointing at the archive.org/download/... URLs — no db3 decoder shipped, no rusqlite, no on-the-fly conversion, no MEGA, no self-hosting cost.

Why not alternatives:

• On-the-fly db3 conversion in the app — ships a fragile decoder, re-runs per user, depends on En Croissant uptime, and provides no preservation.
• Self-hosting on our own infra — cost + we'd be the only copy. archive.org is free, permanent, plain-HTTP, and a neutral host (KingBase already lives there).
• MEGA (Lumbra's official) — encrypted, quota-limited, needs a MEGA client. Sidestepped via En Croissant's plain-HTTP db3.

Policy baked in: public-domain/plain-HTTP → auto-download (Ajedrez); free-but-defunct → preserve+host (Caissabase); explicitly-licensed redistributable → host with license honoured (Lumbra's, CC BY-NC-SA); commercial/curated-protected → bring-your-own, convert your licensed copy locally, never redistributed (Megabase).

The Ajedrez Correspondence URL resolution also belongs here.

This is infra/ops + a one-off task — it does not block the Phase C GUI; the app gains trivial catalog entries once the IA items exist.

[jozef2svrcek]

Update: Caissabase is the DEFAULT deep-history base; Ajedrez is the alternative

Decision refined: the deep-history base is chosen on historical data quality + trust, not on "living/updates" — because an aggregation's recent slice is just TWIC + Lichess-broadcast games delayed (the same redundancy that disqualified Lumbra's as a live source). Currency always comes from TWIC + Lichess broadcasts directly; the base is capped at its cutoff via the date window.

• Caissabase 2024 → default (proven, accepted standard during its maintained life).
• Ajedrez OTB → alternative free option (already shipped, Multi-source Phase B3: 7z + Ajedrez Data OTB historical source (#40) #96).
• Megabase / commercial → suggested in the onboarding wizard if the user owns it (bring-your-own → convert locally → import; never redistributed).
• Lumbra's → optional (convert+host as previously noted, or bring-your-own).

Conversion route to verify: En Croissant likely has a built-in PGN export for its own .db3, so the one-off is probably no custom decoder — export db3→PGN in En Croissant → compress → upload to archive.org. (Fallback: port encoding.rs.) The archive.org upload is a maintainer step (needs an IA account).

Sequencing: Caissabase-as-default needs its hosted PGN to exist, so the convert+host here is a soft prerequisite for the onboarding default — but it runs in parallel with the Phase C GUI (which can use Ajedrez/TWIC for dev and wire Caissabase's URL once hosted). KingBase/MillionBase are not added (subsets/ingredients of Caissabase).