fix(seo): set Cache-Control no-cache on /robots.txt

[stysus]

Bug Description & Symptom

/robots.txt had Cache-Control: public, max-age=86400, causing Cloudflare edge CDN and intermediate proxies to cache crawling rules for 24 hours. When robots directives were updated, crawlers continued to receive stale cached directives.

Root Cause Analysis

The origin server explicitly sent Cache-Control: public, max-age=86400 from both handleRobotsTXT and the cacheControl middleware. This instructed downstream CDNs (Cloudflare) to store the response in edge caches without revalidating with origin on every request.

Proposed Fix

1. Changed Cache-Control header on /robots.txt from public, max-age=86400 to no-cache in handleRobotsTXT (handlers.go) and the cacheControl middleware (compress.go).
2. This ensures that Cloudflare and search engine crawlers always revalidate with the origin server (using ETag/304 Not Modified when unchanged), allowing any policy change to be reflected in real time without manual CDN cache purges.
3. Updated unit test expectations in TestCacheControlHeaders (handlers_test.go).

Scope Lock & Blast Radius

• Strict Scope Boundary: Fix is limited strictly to /robots.txt caching header.
• No Symptom Patching: Addressed the origin header directly to ensure proper CDN protocol compliance.
• Regression Test Added: Automated test case verifying the fix and preventing regression.

Mandatory Zero-Failure Pre-Commit Verification (AGENTS.md)

All commands below must be executed locally and pass with Exit Code 0 prior to creating the PR:

Backend Suite (if backend code changed)

• gofmt -w . && gofmt -l . (Output must be completely empty)
• go vet ./... (Static analysis clean)
• go test -count=1 -v ./... (All tests pass with cache disabled)

Frontend Suite (if frontend code changed)

• npm run format && npm run lint (Prettier code style & ESLint clean)
• npm run check (SvelteKit TypeScript validation clean)
• npm run build (Production build succeeds with static adapter)

Live Localhost End-to-End (E2E) Verification (Mandatory)

• Server booted cleanly on localhost (go run ./cmd/server / preview) with 0 startup crashes/panics.
• Live HTTP requests / UI workflows tested end-to-end against localhost matching implementation plan.
• Server logs & responses verified clean with 0 unexpected HTTP errors (no unintended 4xx/5xx responses) and proper status codes (2xx/explicit error contracts).

Local Verification Evidence

[backend]
gofmt -w . && gofmt -l . -> 0 unformatted files
go vet ./... -> exit 0
go test -count=1 -v ./... -> PASS (all packages ok)

[frontend]
npm run format && npm run lint -> All matched files use Prettier code style! 0 ESLint errors
npm run check -> 0 errors, 0 warnings
npm run build -> static site written to build, exit 0

[localhost E2E]
GET /robots.txt -> Cache-Control: no-cache, Content-Type: text/plain; charset=utf-8 (200 OK)