docs site: /robots.txt and /sitemap.xml do not exist — both answer 200 with the homepage HTML

One-liner

The site has no robots.txt and no sitemap. Worse than absent: both paths answer 200 text/html with the homepage, so a crawler that fetches /robots.txt gets a web page, and a sitemap submitted to Search Console would fail to parse. 403 doc pages have nothing telling a search engine they exist.

Measured (production, 2026-08-25)

$ curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://objectstack.ai/robots.txt
200 text/html; charset=utf-8
$ curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://objectstack.ai/sitemap.xml
200 text/html; charset=utf-8
$ curl -s https://objectstack.ai/robots.txt | head -c 60
<!DOCTYPE html><html data-dpl-id="dpl_3Yii…

apps/docs/app/ contains no robots.ts and no sitemap.ts; apps/docs/public/ holds only logo.svg and hero-cover-dark.png.

Root cause of the 200-instead-of-404 half is a separate card — the [lang] catch-all. Fixing this one does not fix that one, and vice versa.

Expected

apps/docs/app/robots.ts — a real text/plain robots response that allows crawling and declares Sitemap: https://objectstack.ai/sitemap.xml.
apps/docs/app/sitemap.ts — every indexable URL: /, /docs/** (403 pages, from source.getPages()), and the blog. lastModified should come from a real signal (git mtime of the source .mdx) rather than build time, so an unchanged page does not look edited on every deploy.
Both must use the canonical origin https://objectstack.ai, taken from one shared constant, not hardcoded twice.

Acceptance

curl -sI https://objectstack.ai/robots.txt → 200 text/plain and the body names the sitemap
curl -s https://objectstack.ai/sitemap.xml | head -1 → an XML declaration, not HTML
the sitemap's URL count matches find content/docs -name '*.mdx' | wc -l plus the non-docs routes
no absolute URL in either file names a host other than objectstack.ai

Source

Found in an SEO review of the docs site (apps/docs) run on 2026-08-25, measured against the local dev server and against production. The canonical origin is https://objectstack.ai — maintainer ruling recorded in #10659:

这个仓的文档站规范 URL 是 https://objectstack.ai

Claim: PM loop round 1 (epic PM for #12243)
Session: session_f9f0958b-ab68-46cc-801c-216aa7ee2107
Branch: claude/issue-12232-robots-sitemap
Worktree: objectstack-issue-12232
Domain: domain:devx
File surface: apps/docs/app/robots.ts, apps/docs/app/sitemap.ts, and one new shared origin constant under apps/docs/lib/ (this card OWNS creating it — #12234 and #12240 import it next round) (stop on breach; explain in the report)
Container & model: M, mode:subagent, model: opus — no path-derived mandate (node scripts/pm/dispatch-gates.mjs --tier on this surface: "no path-derived mandate"); judgement call on the sitemap's source of truth and the lastModified signal
Clause-②: no
Serial constraints cleared: none in flight. Serialised AFTER this card, same round-2 queue: #12234 (imports the origin constant this card creates), then #12235 (edits the same generateMetadata block as #12234)

Claim: os-dev (developer seat), picked up from the PM claim above.

Session: session_f9f0958b-ab68-46cc-801c-216aa7ee2107 — I am the os-dev subagent dispatched by that PM session; the platform does not mint a separate session id for the subagent, so the id matches the PM comment above. What distinguishes this claim from the PM's is the worktree + base sha below.
Branch: claude/issue-12232-robots-sitemap (pushed empty at claim time as a write-routing probe)
Worktree: /Users/zhuangjianguo/Documents/GitHub/objectstack-issue-12232
Base: 20b0fdb568c648374963cc13fdab89f4f74594bf (origin/main at checkout)
File surface: apps/docs/app/robots.ts, apps/docs/app/sitemap.ts, one new shared origin constant under apps/docs/lib/.

Any other agent reading this: the branch is taken. Report a conflict rather than pushing to it.

os-dev-report

{
  "issue": 12232,
  "status": "done",
  "branch": "claude/issue-12232-robots-sitemap",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/12253",
  "premise_still_valid": true,
  "summary": "Added apps/docs/app/robots.ts (text/plain, allows crawling, declares the sitemap), apps/docs/app/sitemap.ts (408 URLs derived from source.getPages()/blog.getPages() - never a hand-maintained list), and apps/docs/lib/site.ts, the shared canonical-origin constant this card owns: it exports exactly SITE_ORIGIN = 'https://objectstack.ai' and absoluteUrl(path), which throws at build time on a path that does not start with a slash rather than emitting a URL on the wrong host. #12234 should write metadataBase: new URL(SITE_ORIGIN) (a shared mutable URL instance is deliberately not exported); #12240 should use absoluteUrl() for every JSON-LD id. lastModified is the git committer date of each source .mdx from ONE git log pass measured at 1.0s over 11,231 commits - so the cheap and the good signal turned out to be the same signal, and the per-file git log the card offered as the expensive option was never needed; a page whose date cannot be known ships with no lastmod element and the build prints a counted warning, never build time. robots.txt deliberately carries no Disallow for /api, /og, /docs/**.mdx or llms*.txt: that is #12241's card, /og must stay crawlable for #12235's OG cards, and two PRs editing the same lines is a conflict for no gain. Filed one out-of-scope finding (#12255) and left it unassigned. No changeset (docs-site only); skip-changeset label applied and read back after the size-labeler ran (labels now: documentation, size/m, skip-changeset). Worktree removed cleanly without --force; both servers I started were killed by recorded PID and verified down.",
  "premise_check": "All four PM mechanical assumptions VERIFIED, none falsified. (1) origin/main has no robots.ts and no sitemap.ts under apps/docs/app - confirmed. (2) Production re-measured today: curl -s -o /dev/null -w '%{http_code} %{content_type}' https://objectstack.ai/robots.txt -> '200 text/html; charset=utf-8', same for /sitemap.xml - unchanged from dispatch time. (3) source.getPages() is the right source; note it lists EVERY language when called with no argument (fumadocs LoaderOutput), which is left unfiltered on purpose. (4) THE ONE THE PM ASKED TO HAVE TESTED: a literal app/robots.ts DOES win over the single-segment [lang] dynamic route - confirmed in dev, in the production build route table, and against next start. So #12233 is not a blocker for this card. Converse also measured and unchanged: an unrelated dotted path still returns 200 text/html, so this PR fixes exactly two paths and does not touch #12233's defect.",
  "tests": "All at final commit e02a77cc7 (git rev-parse --short HEAD), tree clean, branch pushed. DEV SERVER (next dev -p 38412), the measurement the PM asked for: curl -s -o /dev/null -w '%{http_code} %{content_type}' http://localhost:38412/robots.txt -> '200 text/plain'; body = 'User-Agent: *' / 'Allow: /' / blank / 'Sitemap: https://objectstack.ai/sitemap.xml'. curl ... http://localhost:38412/sitemap.xml -> '200 application/xml', first line is an XML declaration, 408 url elements, 406 lastmod elements, and grep -o over every loc element yields exactly one host (objectstack.ai). 408 = 403 (find content/docs -name '*.mdx' | wc -l) + 5 non-docs routes (/, /blog, 3 blog posts); the 2 URLs without lastmod are / and /blog, which have no MDX file. Three dates spot-checked against git log -1 --format=%cI on the source file: /docs 2026-07-24T12:11:05.000Z vs 2026-07-24T20:11:05+08:00, /docs/ai 2026-08-21T10:30:53.000Z vs 2026-08-21T10:30:53Z, /blog/protocol-first-development 2026-07-17T13:17:33.000Z vs 2026-07-17T21:17:33+08:00 - all match to the second. Zero [sitemap] warnings in either the dev log or the build log. PRODUCTION BUILD: next build exit 0; route table shows both as static prerenders ('O /robots.txt', 'O /sitemap.xml' under 'O (Static) prerendered as static content'), so the git log runs in the build process and never in a request - that was the real risk in the git-mtime design and it is retired. Prerendered artifacts on disk: .next/server/app/sitemap.xml.meta declares content-type application/xml; .next/server/app/sitemap.xml.body has 408 url / 406 lastmod. PRODUCTION SERVER (next start -p 38413): /robots.txt -> 200, content-type text/plain, x-nextjs-cache HIT; /sitemap.xml -> '200 application/xml'. NEIGHBOURS unchanged: /en/docs -> 307 -> /docs (the epic asks this not break), /llms.txt -> 200 text/plain, /docs/upgrading.mdx -> 200 text/markdown (agent-reader feature intact), unrelated dotted path -> 200 text/html (#12233, untouched). GATES: the union was re-run AFTER the final commit at e02a77cc7 and every verdict below is the gate's OWN printed line, not a bare $?: check:published-files -> 'checkmark check:published-files - 69 publishable package(s) of 78 workspace member(s) ...'; check:test-source-alias -> 'check-test-source-alias OK - 72 packages with tests scanned'; check:type-source-resolution -> 'check-type-source-resolution OK - 93 tsc program(s) across 77 packages scanned'; check:nul-bytes -> 'check-nul-bytes: OK (scanned 6772 text file(s) -- 6772 tracked, 0 untracked-not-ignored; skipped 7 binary; no raw ASCII control bytes)'. The first three are exactly what node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack derives for these three paths; its first run WARNED that my tree was stale (3 commits behind origin/main, 3 gate-source files changed across the range), so the branch was fast-forwarded to origin/main f64668d3c and the derivation re-run from a current tree - the second run printed no stale warning and the same three families. pnpm --filter '@objectstack/docs^...' build (dependency closure first, per the new-worktree rule) and pnpm --filter @objectstack/docs typecheck both ended 'VERDICT command-exit 0'. REPO-WIDE LINT WAS NOT NARROWED: pnpm lint (eslint . --no-inline-config over the whole repo) ran in full, 28s, 'VERDICT command-exit 0', no findings - so no narrowing declaration is owed for it. ONE DECLARED NARROWING, stated in the PR body: scripts/pm/os-verify-lock.sh could not take the shared verify lock on this host (macOS, no usable flock; the lock is declared Linux-only), so every wrapped command ran in DECLARED UNLOCKED MODE with no serialization guarantee. NO ABLATION was run: nothing here is a new gate or guard whose ability to fail needs proving, and the reverse-verification that mattered - does a literal route actually beat [lang] - is answered by the positive measurement plus the unrelated dotted path still returning 200. NOT MEASURED, and it is the PM's half by contract: CI on PR #12253 had not reported when this was written (draft opened, report delivered immediately per the dispatch rules). The Vercel preview deployment is the one production-shaped check I could not run locally; the local next build + next start above is its closest substitute.",
  "open_questions": [],
  "out_of_scope_findings": [
    "filed as #12255: apps/docs/app/page.tsx is a shadowed route - proxy.ts rewrites / to /en so RootPage() never runs; measured (curl / -> 200 text/html, not the 307 to /docs the file would produce). Labelled 'finding', unassigned, no pm:queue. Matters to this epic: #12234 and #12240 add canonical/JSON-LD to the homepage, and an agent that edits app/page.tsx will ship a green PR that changes nothing served. Filed standalone rather than as a sub-issue of #12243 so adopting it into the epic stays the PM's call."
  ]
}

ACCEPT — PR #12253, reviewed against GitHub rather than against the report.

Checked, not taken on trust:

Shape: draft, base main, Fixes #12232 on the first line, 3 files, +217/-0 — nothing pre-existing is edited, so this cannot regress a served path by accident. skip-changeset is on the PR and Check Changeset is green with it.
Surface: apps/docs/app/robots.ts, apps/docs/app/sitemap.ts, apps/docs/lib/site.ts — exactly the declared surface, no drift.
The assumption this card existed to test: a literal app/robots.ts does outrank the single-segment [lang] route — confirmed in dev, in the production route table and against next start. The converse was measured too: an unrelated dotted path still returns 200 text/html, so this PR takes back exactly two paths and leaves docs site: any single-segment path containing a dot renders the homepage with 200 (soft-404 class) #12233's defect untouched. That is the honest result and it unblocks nothing it should not.
lastModified: git committer date per source file from one git log pass, not build time — and the shallow-clone case is handled by omitting lastmod with a counted warning rather than substituting build time. Given this repo already carries a guard for shallow clones answering window queries wrongly with exit 0, degrading loudly here is the right call. ⚠️ Expect the warning to actually fire on the Vercel build, which clones shallow: the first deploy is where we learn how many of the 408 URLs ship without a date. That is a follow-up reading, not a defect in this PR.
robots.txt carries no Disallow — deliberate, and correct: /og must stay crawlable for docs site: Open Graph images are generated for all 403 pages but no page references them #12235's cards, and the .mdx / llms*.txt directives are docs site: /docs/**.mdx serves an indexable parallel copy of every page with no robots directive #12241's card. Two PRs editing the same four lines is a conflict for no gain.
Out-of-scope finding [finding] docs site: apps/docs/app/page.tsx is a shadowed route — the proxy rewrites / to /en, so it never runs #12255 filed unassigned with the finding label and a link back — protocol-correct, and materially useful (see below).

CI at review time: 15 green, 0 red, Lint & Repo Gates and Type Check · debt ledger still converging. in_progress is an honest reading, not a pass — the PR flips ready and enters the queue only when every check on it is green, required or not.

Navigation Menu

Repository navigation

docs site: `/robots.txt` and `/sitemap.xml` do not exist — both answer 200 with the homepage HTML #12232

One-liner

Measured (production, 2026-08-25)

Expected

Acceptance

Source

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

docs site: /robots.txt and /sitemap.xml do not exist — both answer 200 with the homepage HTML #12232

Description

One-liner

Measured (production, 2026-08-25)

Expected

Acceptance

Source

Activity

os-zhuang commented on Aug 25, 2026

os-zhuang commented on Aug 25, 2026

os-zhuang commented on Aug 25, 2026

os-zhuang commented on Aug 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

docs site: `/robots.txt` and `/sitemap.xml` do not exist — both answer 200 with the homepage HTML #12232