Recover a rotated ZDF key by reading the one its player uses - #125
Merged
Conversation
The spike this issue asked for found the key in a far better place than feared: ZDF's homepage publishes it as apiAuthToken in its own server-rendered data, at a stable URL — not inside a hash-named JS chunk. Extracting it and calling the API with it returns 200. So a rejection now triggers a read of the current key, a verification call, and a retry. Nothing is adopted before it has answered 200, which is what keeps a failed scrape equal to today's behaviour rather than worse than it. The recovered key is in-memory only and health stays degraded while it is in use: it is a recovery, not a second place to configure the key. Closes #112 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #112.
The spike first, since the issue said the answer might be "don't"
It isn't. The fear was that the key lives in a hash-named JS chunk, making the scrape rot faster than the key rotates. Measured against the live site instead:
Server-rendered data at a fixed URL, under a property name that is unambiguous — the page carries a second, different token under
apiToken, and onlyapiAuthTokenis the one the API wants. Sub-pages don't carry it, so the homepage it is, 5.6 MB and all.What it does
On a 401/403 the client reads the current key, verifies it with a live call, and only then adopts it and retries. That verification is the whole safety argument: a scrape of an undocumented page fails by returning something plausible, and sending a garbage bearer would turn "your key is stale" into an unexplained failure inside the recovery path. A failed discovery lands exactly where today's behaviour lands — loud error, degraded health.
Other guards:
One reversal from the issue
I wrote "probably opt-in, default off, until it has been seen to work through at least one real rotation". It ships default on, because the reasoning behind that caution didn't survive the spike: the source is stable rather than brittle, and nothing is adopted without a live 200. Opt-in would mostly mean the feature is off in exactly the deployments that need it — unattended ones.
Zdf__KeyDiscovery__Enabled=falseturns it off.Tests
Eight unit tests over a stub serving both the player page and the API: reads and verifies, refuses a candidate the API also rejects, survives a redesigned page, ignores a no-op "recovery", rate-limits, stays inert when disabled, recovers a crawl that used to throw, and reports the right health.
Plus a live test that reads the key off the real page and proves it authenticates — so a ZDF redesign fails in
./build.sh TestLiverather than during someone's outage../build.sh Testgreen: 500 passing.