feat(cctv): add 317 operator-published webcams from the bekijkhet.nu index - #327
Closed
meijbaard wants to merge 4 commits into
Closed
feat(cctv): add 317 operator-published webcams from the bekijkhet.nu index#327meijbaard wants to merge 4 commits into
meijbaard wants to merge 4 commits into
Conversation
bekijkhet.nu is a link directory kept by hand since 2012: 812 links to
webcams their operators publish themselves. It is a list of pages, not of
feeds — no coordinates, no stream URLs, 383 different hosts — so turning it
into cameras means reading every page it points at.
Three stages, so each can be rerun on its own: harvest reads the 16 index
pages and resolves the Wowza platform to a playlist, geocode fills the place
table, emit writes the module.
The place table is committed because the captions needed a human once.
Nominatim answers 3 in 16 of the foreign captions raw ("Kransjka Gora",
"Soi Buakhao, Pattaya Live") and 9 in 9 once the place name is pulled out of
them, so those readings live in the script and their results live here —
which also means regenerating never asks Nominatim the same question twice.
The harvest file itself is ignored. It is a copy of somebody else's index,
and it regenerates in a minute.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Netherlands was 26 pins: the Rijkswaterstaat motorway cameras, and nothing else. No harbours, no coast, no city centres. This adds 230 Dutch cameras and 87 abroad. Four shapes come out of the index, and the split matters: 57 HLS — the webcam-*.nl Wowza platform, playlists without a token. streamlock.net sends Access-Control-Allow-Origin: *, so these play straight from the browser and need no entry in the proxy allow-list. 210 YouTube — the operator broadcasts on their own channel. The index links to the channel's "/streams" tab, which parseYouTubeUrl reads as nothing at all; rewritten to "/live" it becomes a live channel, and /api/cctv/resolve answers what is on air. No resolver code needed, and no video id baked: these streams run for months and come back under a new id after every restart. 3 iframe — provider embed pages that stand on their own. 47 link — the stream is live but locked to the operator's page, by a Wowza token that expires inside twenty minutes or by a Referer check the browser cannot satisfy from here. external_url alone puts the viewer into its external state: a way through to the camera rather than a copy of it. Everything else the index lists is left out. 411 links are long-tail pages we cannot read — a sample of 30 gave 6 usable feeds — and 58 name a country but no place, mostly aggregator channels cycling through dozens of locations, which a single pin would have to lie about. external_url always points at the operator, never at bekijkhet.nu. That is how they ask to be read, and it is the right way round anyway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four keys rather than one, for the reason world-live.ts is split: a single worldwide key would make every bounds query fetch every camera. The boxes are drawn round what the arrays actually hold — the Canaries at 28N and Nordkapp at 71N are both European here — rather than round the continents. Registered in RAW_REGION_FETCHERS only. netherlands.ts is registered there and called from fetchEuropeCameras as well, and GET concatenates regions without deduplicating on id, so under ?region=all every Rijkswaterstaat camera currently lands on the map twice. That is a separate fix; this source just declines to repeat it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These assertions are about the generator, not about the upstream site, and each one has already caught something: an id collision from two cameras sharing a caption, a Wowza URL baked with a token that expires in twenty minutes, and a "/streams" link left unrewritten — which looks fine until you notice the camera never plays. The YouTube check runs against parseYouTubeUrl itself rather than a copy of its rules, because what matters is not the URL shape but whether the viewer can read it. Writing the rules out by hand missed two legitimate forms. The live test asks for 200 or 404 from the HLS hosts: a Wowza 404 means the camera is not publishing right now, and plenty of these are daylight-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@meijbaard is attempting to deploy a commit to the Developing Osiris' projects Team on Vercel. A member of the Team first needs to authorize it. |
Author
|
Superseded by #328 — same work, renamed throughout. The cameras here are open data published by their own operators, so naming the module after the index that catalogues them overstated its role. The index is credited in the README instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Netherlands was 26 pins on the map: the Rijkswaterstaat motorway cameras that
netherlands.tsfetches, and nothing else. No harbours, no coast, no city centres.bekijkhet.nu is a link directory two people have kept by hand since 2012 — 812 links to webcams their operators publish themselves. This adds the part of it OSIRIS can actually show: 230 Dutch cameras and 87 abroad.
What the index is, and what it is not
It is a list of pages, not of feeds. Every link points at the operator's own landing page, across 383 different hosts, with no coordinates and no stream URLs anywhere. So the work here is reading those pages, and deciding honestly which ones OSIRIS can show.
Four shapes came out of it:
stream_url+stream_type: 'hls'external_urlat the channel's/liveexternal_urlaloneThe YouTube half needs no new resolver code. The index links to a channel's
/streamstab — a list of past broadcasts, whichparseYouTubeUrlreads as nothing at all, so those cameras would silently never play. Rewritten to/livethey becomelive-channel, and/api/cctv/resolvealready answers what is on air. Nothing is baked: these streams run for months and come back under a new video id after every restart, which is exactly the case that route's header describes.The HLS half needs no proxy entry. These are the
webcam-*.nlWowza platform, andstreamlock.netsendsAccess-Control-Allow-Origin: *, so they play straight from the browser.ALLOWED_HOSTSinproxy/route.tsis untouched.47 carry a link and no stream, deliberately. Their playlists are real and live, but locked to the operator's page — a Wowza token that expires inside twenty minutes, or a
Referercheck the browser cannot satisfy from here. Baking either would ship a URL that is already dead when someone clicks it, so those getexternal_urlalone and land in the viewer's external state: a way through to the camera rather than a copy of it. That is also how bekijkhet.nu asks to be read — they link to the operator's whole site so people see whose camera it is.external_urlnever points back at the index.What is left out
netherlands.tsalready fetches those live.Coordinates
Resolved through Nominatim, country-filtered, cached in
scratch/bekijkhetnu-places.jsonso regenerating never asks twice. Dutch cameras are looked up on their caption first —Sint Servaasbrug, Maastrichtputs the pin on the bridge — and fall back to the village centre.The foreign captions needed reading once by hand: Nominatim answers 3 in 16 of them raw and 9 in 9 given the place name, so
Kransjka Gora→Kranjska GoraandSoi Buakhao, Pattaya Live→Pattayalive in the generator, where they can be reviewed.Registration
Four keys in
RAW_REGION_FETCHERS, split the wayworld-live.tsis — one worldwide key would make every bounds query fetch every camera. Bounding boxes are drawn round what the arrays actually hold, so the Canaries at 28N and Nordkapp at 71N are both European here.Registered in
RAW_REGION_FETCHERSonly. Worth flagging while you are in here:netherlandsis registered there and called fromfetchEuropeCameras, andGETconcatenates regions without deduplicating onid— so under?region=all, which is what the frontend requests, every Rijkswaterstaat camera currently lands on the map twice. I have left that alone rather than mix a fix into this PR; happy to send it separately.Verifying
Checked against a running instance: a viewport query at Harlingen picks up
bekijkhet-nlandbekijkhet-europe, and/api/cctv/resolveturns@demeent7646/liveinto a playable embed. 40 of the 57 HLS playlists returned live segments at 22:00 CET; the other 17 are daylight-only cameras, which is why the live test accepts a Wowza 404 as a valid answer.npm run lintruns out of heap on this repo before and after this branch, so I could not use it as a gate.eslinton the changed files reports nothing new —route.tshas the same 29 pre-existing findings onmaster.Credit
With thanks to Bram and Annelies, who keep the index.
🤖 Generated with Claude Code