Skip to content

feat(cctv): add 317 operator-published webcams from the bekijkhet.nu index - #327

Closed
meijbaard wants to merge 4 commits into
simplifaisoul:masterfrom
meijbaard:feat/cctv-bekijkhetnu
Closed

feat(cctv): add 317 operator-published webcams from the bekijkhet.nu index#327
meijbaard wants to merge 4 commits into
simplifaisoul:masterfrom
meijbaard:feat/cctv-bekijkhetnu

Conversation

@meijbaard

Copy link
Copy Markdown

The Netherlands was 26 pins on the map: the Rijkswaterstaat motorway cameras that netherlands.ts fetches, and nothing else. No harbours, no coast, no city centres.

bekijkhet.nu is a link directory two people have kept by hand since 2012 — 812 links to webcams their operators publish themselves. This adds the part of it OSIRIS can actually show: 230 Dutch cameras and 87 abroad.

What the index is, and what it is not

It is a list of pages, not of feeds. Every link points at the operator's own landing page, across 383 different hosts, with no coordinates and no stream URLs anywhere. So the work here is reading those pages, and deciding honestly which ones OSIRIS can show.

Four shapes came out of it:

count how it plays
HLS 57 stream_url + stream_type: 'hls'
YouTube 210 external_url at the channel's /live
iframe 3 provider embed page
link only 47 external_url alone

The YouTube half needs no new resolver code. The index links to a channel's /streams tab — a list of past broadcasts, which parseYouTubeUrl reads as nothing at all, so those cameras would silently never play. Rewritten to /live they become live-channel, and /api/cctv/resolve already answers what is on air. Nothing is baked: these streams run for months and come back under a new video id after every restart, which is exactly the case that route's header describes.

The HLS half needs no proxy entry. These are the webcam-*.nl Wowza platform, and streamlock.net sends Access-Control-Allow-Origin: *, so they play straight from the browser. ALLOWED_HOSTS in proxy/route.ts is untouched.

47 carry a link and no stream, deliberately. Their playlists are real and live, but locked to the operator's page — a Wowza token that expires inside twenty minutes, or a Referer check the browser cannot satisfy from here. Baking either would ship a URL that is already dead when someone clicks it, so those get external_url alone and land in the viewer's external state: a way through to the camera rather than a copy of it. That is also how bekijkhet.nu asks to be read — they link to the operator's whole site so people see whose camera it is. external_url never points back at the index.

What is left out

  • 411 long-tail links across 354 one-off hosts. A sample of 30 gave 6 usable feeds; the rest are JS-rendered players, nested iframes, or hub pages. Per-site work with a high chance of breaking quietly.
  • 58 links that name a country but no place — mostly aggregator channels (Virtual Railfan, EarthCam, PTZ TV) that cycle through dozens of locations. A single pin would have to lie about where they are.
  • The 26 Rijkswaterstaat cameras the index also lists. netherlands.ts already fetches those live.

Coordinates

Resolved through Nominatim, country-filtered, cached in scratch/bekijkhetnu-places.json so regenerating never asks twice. Dutch cameras are looked up on their caption first — Sint Servaasbrug, Maastricht puts the pin on the bridge — and fall back to the village centre.

The foreign captions needed reading once by hand: Nominatim answers 3 in 16 of them raw and 9 in 9 given the place name, so Kransjka GoraKranjska Gora and Soi Buakhao, Pattaya LivePattaya live in the generator, where they can be reviewed.

Registration

Four keys in RAW_REGION_FETCHERS, split the way world-live.ts is — one worldwide key would make every bounds query fetch every camera. Bounding boxes are drawn round what the arrays actually hold, so the Canaries at 28N and Nordkapp at 71N are both European here.

Registered in RAW_REGION_FETCHERS only. Worth flagging while you are in here: netherlands is registered there and called from fetchEuropeCameras, and GET concatenates regions without deduplicating on id — so under ?region=all, which is what the frontend requests, every Rijkswaterstaat camera currently lands on the map twice. I have left that alone rather than mix a fix into this PR; happy to send it separately.

Verifying

npm test                                                        # 557 passed
RUN_LIVE_TESTS=1 npx vitest run src/app/api/cctv/bekijkhetnu.test.ts
npx tsc --noEmit
curl 'localhost:3000/api/cctv?region=bekijkhet-nl' | jq .total   # 230

Checked against a running instance: a viewport query at Harlingen picks up bekijkhet-nl and bekijkhet-europe, and /api/cctv/resolve turns @demeent7646/live into a playable embed. 40 of the 57 HLS playlists returned live segments at 22:00 CET; the other 17 are daylight-only cameras, which is why the live test accepts a Wowza 404 as a valid answer.

npm run lint runs out of heap on this repo before and after this branch, so I could not use it as a gate. eslint on the changed files reports nothing new — route.ts has the same 29 pre-existing findings on master.

Credit

With thanks to Bram and Annelies, who keep the index.

🤖 Generated with Claude Code

meijbaard and others added 4 commits September 8, 2026 22:08
bekijkhet.nu is a link directory kept by hand since 2012: 812 links to
webcams their operators publish themselves. It is a list of pages, not of
feeds — no coordinates, no stream URLs, 383 different hosts — so turning it
into cameras means reading every page it points at.

Three stages, so each can be rerun on its own: harvest reads the 16 index
pages and resolves the Wowza platform to a playlist, geocode fills the place
table, emit writes the module.

The place table is committed because the captions needed a human once.
Nominatim answers 3 in 16 of the foreign captions raw ("Kransjka Gora",
"Soi Buakhao, Pattaya Live") and 9 in 9 once the place name is pulled out of
them, so those readings live in the script and their results live here —
which also means regenerating never asks Nominatim the same question twice.

The harvest file itself is ignored. It is a copy of somebody else's index,
and it regenerates in a minute.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Netherlands was 26 pins: the Rijkswaterstaat motorway cameras, and
nothing else. No harbours, no coast, no city centres. This adds 230 Dutch
cameras and 87 abroad.

Four shapes come out of the index, and the split matters:

  57 HLS — the webcam-*.nl Wowza platform, playlists without a token.
  streamlock.net sends Access-Control-Allow-Origin: *, so these play
  straight from the browser and need no entry in the proxy allow-list.

  210 YouTube — the operator broadcasts on their own channel. The index
  links to the channel's "/streams" tab, which parseYouTubeUrl reads as
  nothing at all; rewritten to "/live" it becomes a live channel, and
  /api/cctv/resolve answers what is on air. No resolver code needed, and no
  video id baked: these streams run for months and come back under a new id
  after every restart.

  3 iframe — provider embed pages that stand on their own.

  47 link — the stream is live but locked to the operator's page, by a Wowza
  token that expires inside twenty minutes or by a Referer check the browser
  cannot satisfy from here. external_url alone puts the viewer into its
  external state: a way through to the camera rather than a copy of it.

Everything else the index lists is left out. 411 links are long-tail pages
we cannot read — a sample of 30 gave 6 usable feeds — and 58 name a country
but no place, mostly aggregator channels cycling through dozens of
locations, which a single pin would have to lie about.

external_url always points at the operator, never at bekijkhet.nu. That is
how they ask to be read, and it is the right way round anyway.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four keys rather than one, for the reason world-live.ts is split: a single
worldwide key would make every bounds query fetch every camera. The boxes
are drawn round what the arrays actually hold — the Canaries at 28N and
Nordkapp at 71N are both European here — rather than round the continents.

Registered in RAW_REGION_FETCHERS only. netherlands.ts is registered there
and called from fetchEuropeCameras as well, and GET concatenates regions
without deduplicating on id, so under ?region=all every Rijkswaterstaat
camera currently lands on the map twice. That is a separate fix; this source
just declines to repeat it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These assertions are about the generator, not about the upstream site, and
each one has already caught something: an id collision from two cameras
sharing a caption, a Wowza URL baked with a token that expires in twenty
minutes, and a "/streams" link left unrewritten — which looks fine until you
notice the camera never plays.

The YouTube check runs against parseYouTubeUrl itself rather than a copy of
its rules, because what matters is not the URL shape but whether the viewer
can read it. Writing the rules out by hand missed two legitimate forms.

The live test asks for 200 or 404 from the HLS hosts: a Wowza 404 means the
camera is not publishing right now, and plenty of these are daylight-only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Sep 8, 2026

Copy link
Copy Markdown

@meijbaard is attempting to deploy a commit to the Developing Osiris' projects Team on Vercel.

A member of the Team first needs to authorize it.

@meijbaard

Copy link
Copy Markdown
Author

Superseded by #328 — same work, renamed throughout. The cameras here are open data published by their own operators, so naming the module after the index that catalogues them overstated its role. The index is credited in the README instead.

@meijbaard meijbaard closed this Sep 8, 2026
@meijbaard
meijbaard deleted the feat/cctv-bekijkhetnu branch September 8, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant