fix(seo): noindex the pages.dev alias, and guard the file that can deindex us

[pip-robot]

The Pages project's production alias is a fully crawlable duplicate of the whole site. Found by the CMO on technology#72; verified from outside this morning rather than taken on trust:

$ curl -sI https://pip-web-9oj.pages.dev/
HTTP/2 200
content-type: text/html; charset=utf-8
...                       # no x-robots-tag anywhere in the response

Cloudflare adds noindex to preview deployments only. The production alias does not get one. The only thing separating the two hosts today is <link rel="canonical">, which is a hint and not a directive, and on a five-week-old domain with almost no links, splitting what little signal exists across two hosts is the one avoidable own goal.

A header, not a redirect

The CMO's first preference was a 301 from any non-playpip.io host in functions/_middleware.ts. I would not do that, and the reason is not the redirect logic. A root functions/_middleware.ts runs for every request to the Pages project including static assets, so adding one to redirect a second host puts a Worker invocation in front of every chunk, image and font on playpip.io as well. That is a real latency and billing cost on the host we care about, paid permanently, to fix a duplicate on the host we do not.

A host-scoped rule in public/_headers costs nothing at runtime and Cloudflare documents this exact recipe. It leaves the duplicate reachable and linkable, which is the weaker outcome the CMO named, but removing it from the index is the whole of the risk.

Preview deployments are a different host again (<hash>.pip-web-9oj.pages.dev) and are untouched, which matters: their permanent addresses are how any past build gets audited from outside, and that is what identified the second publisher behind the account outage (#92).

The test is the more useful half of this PR

public/_headers is the one file in this repo that can take the site out of Google in a single line, and nothing was reading it. Drop the host from that rule and the same noindex applies to every path on playpip.io: the deploy is green, pnpm test:all is green, the site looks perfect to anybody who opens it, and we quietly disappear from search over the following weeks. No gate anywhere else can see that, because it is a static file Cloudflare interprets after we have shipped.

tests/headers.test.ts parses the file the way Pages does and holds three things: every rule has a pattern Pages will match and at least one header, the alias rule exists and sets noindex, and no noindex rule reaches the site itself.

Checked by mutation rather than by reading:

mutant	result
https://pip-web-9oj.pages.dev/* -> /*	2 tests fail
the rule deleted entirely	1 test fails, and the invariant test still asserts rather than passing on an empty loop

That last row is why the invariant is written as a collected list compared against [] instead of a loop of assertions: a loop with nothing to iterate is green, and green because there was nothing to check is the failure this whole file is about.

The gate, and what it does not cover

pnpm test:all green (518 tests), pnpm build green, and out/_headers confirmed to carry the rule after export.

Nothing here verifies Cloudflare actually applies it. _headers is parsed at deploy time by Pages, not by us; a syntax error is logged in their build output and the file is otherwise ignored. So the check that matters can only happen after this merges:

curl -sI https://pip-web-9oj.pages.dev/ | grep -i x-robots-tag   # expect: noindex
curl -sI https://playpip.io/            | grep -i x-robots-tag   # expect: nothing at all

The second command is the important one. If it ever returns noindex, revert this immediately. I would rather that check were automated, and it belongs in the daily canary, which cannot run on a schedule until technology#62 is granted.

[cloudflare-workers-and-pages]

Deploying pip-web with    Cloudflare Pages

Latest commit:	f16739d
Status:	✅  Deploy successful!
Preview URL:	https://63217cbd.pip-web-9oj.pages.dev
Branch Preview URL:	https://fix-pages-dev-noindex.pip-web-9oj.pages.dev

View logs