fix(seo): allow public api crawling in robots.txt and add X-Robots-Tag noindex

[stysus]

Bug Description & Symptom

Google Search Console reported https://neuralwire.info/ as "Halaman tidak dapat diindeks: Soft 404". Furthermore, GSC page asset inspection reported 4 assets blocked by robots.txt (/api/categories, /api/news/trending?window=week&limit=10, /api/news?page=1&page_size=16, /api/news?page_size=16), and console warnings indicated that /api/news?page_size=16 was preloaded via link preload but never consumed.

Root Cause Analysis

1. backend/internal/api/handlers.go explicitly had Disallow: /api/ in robots.txt. Because SvelteKit client renders news articles dynamically by fetching public API endpoints, Googlebot's crawler refused to fetch any /api/ endpoints due to the robots.txt directive. This caused the SPA client rendering to fail with 0 articles, rendering an empty page which Google's heuristic classified as a Soft 404.
2. In backend/internal/api/server.go, the preload link in serveIndexHTML was <link rel="preload" as="fetch" href="/api/news?page_size=16"> whereas SvelteKit's getNewsPage requested /api/news?page=1&page_size=16. The query string mismatch caused Chrome/Googlebot to flag the preload as unused.

Proposed Fix

1. Updated robots.txt in handlers.go to disallow /api/admin/ while explicitly allowing /api/ (Allow: /api/). This allows crawlers to fetch public news/category endpoints for page rendering while keeping administrative endpoints protected.
2. Added X-Robots-Tag: noindex in writeJSON() (server.go). This ensures crawlers never index raw JSON endpoints as search results, but can fetch them during page rendering.
3. Updated preload tag in server.go to /api/news?page=1&page_size=16 to match frontend SvelteKit requests.
4. Updated unit tests in handlers_test.go to verify Allow: /api/, Disallow: /api/admin/, and the updated preload link.

Scope Lock & Blast Radius

• Strict Scope Boundary: Fix is limited strictly to the bug's root cause.
• No Symptom Patching: The root data flow was addressed rather than defensive null-checks.
• Regression Test Added: Automated test case verifying the fix and preventing regression.

Mandatory Zero-Failure Pre-Commit Verification (AGENTS.md)

All commands below must be executed locally and pass with Exit Code 0 prior to creating the PR:

Backend Suite (if backend code changed)

• gofmt -w . && gofmt -l . (Output must be completely empty)
• go vet ./... (Static analysis clean)
• go test -count=1 -v ./... (All tests pass with cache disabled)

Frontend Suite (if frontend code changed)

• npm run format && npm run lint (Prettier code style & ESLint clean)
• npm run check (SvelteKit TypeScript validation clean)
• npm run build (Production build succeeds with static adapter)

Live Localhost End-to-End (E2E) Verification (Mandatory)

• Server booted cleanly on localhost (go run ./cmd/server / preview) with 0 startup crashes/panics.
• Live HTTP requests / UI workflows tested end-to-end against localhost matching implementation plan.
• Server logs & responses verified clean with 0 unexpected HTTP errors (no unintended 4xx/5xx responses) and proper status codes (2xx/explicit error contracts).

Local Verification Evidence

[backend]
gofmt -w . && gofmt -l . -> 0 unformatted files
go vet ./... -> exit 0
go test -count=1 -v ./... -> PASS (all packages ok)

[frontend]
npm run format && npm run lint -> All matched files use Prettier code style! 0 ESLint errors
npm run check -> 0 errors, 0 warnings
npm run build -> static site written to build, exit 0

[localhost E2E]
GET /robots.txt -> Allow: /api/, Disallow: /api/admin/ (200 OK)
GET /api/news -> X-Robots-Tag: noindex (200 OK)
GET / -> Preload matches /api/news?page=1&page_size=16 (200 OK)