Conversation
- 擴增 AI_SEARCH_BOTS 清單至 39 個爬蟲 - 新增 Google-CloudVertexBot, Googlebot, Bingbot, PhindBot 等 - 新增 Meta 系列: Meta-ExternalFetcher, FacebookBot - 新增 Cloudflare-AutoRAG, archive.org_bot, Timpibot 等 - 移除 robots.txt 所有註解保持簡潔 - 更新 SEO_MASTER_SSOT.md 完整 AI 爬蟲清單 測試：pnpm test -- --run（1957 個測試全過） Made-with: Cursor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. Scanned FilesNone |
|
✅ SEO 審計通過！所有 2025 標準驗證項目都符合要求。
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8b13216cef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
apps/ratewise/public/robots.txt
Outdated
| User-agent: Googlebot | ||
| Allow: / | ||
|
|
||
| User-agent: Bingbot | ||
| Allow: / |
There was a problem hiding this comment.
Adding dedicated groups User-agent: Googlebot and User-agent: Bingbot with only Allow: / makes these crawlers stop using the User-agent: * disallow set, so paths like /ratewise/?, /ratewise/theme-showcase/, and other blocked routes become crawlable again for major search engines. This reintroduces index pollution/crawl-budget regressions that the wildcard block was preventing; either remove these dedicated groups or replicate the same disallow rules inside them.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
✅ 已修復 (commit 38030f5)
感謝指出這個重要問題。已移除 Googlebot 和 Bingbot 的專門 User-agent 區塊。
修復內容：
- 移除
Googlebot和Bingbot從AI_SEARCH_BOTS清單 - 這些爬蟲現在遵循
User-agent: *的 Disallow 規則 - 開發頁面（
/ratewise/theme-showcase/、/ratewise/ui-showcase/等）不會被索引
策略說明：
- AI 爬蟲（如 GPTBot、ClaudeBot）需要明確 Allow 以確保 AI 搜尋可見度
- 傳統搜尋爬蟲（Googlebot、Bingbot）應遵循通用規則以保護 crawl budget
已更新 SEO_MASTER_SSOT.md 文檔說明此策略。
- 移除 Googlebot 和 Bingbot 的專門 User-agent 區塊 - 這些爬蟲現在遵循 User-agent: * 的 Disallow 規則 - 防止開發頁面（theme-showcase、ui-showcase 等）被索引 - 更新 SEO_MASTER_SSOT.md 說明策略變更 - AI 爬蟲清單從 39 個調整為 37 個 測試：pnpm test -- --run seo（404 個測試全過） Made-with: Cursor
|
✅ SEO 審計通過！所有 2025 標準驗證項目都符合要求。
|
2 participants
Summary
AI_SEARCH_BOTS清單從 17 個增至 39 個 AI 爬蟲變更檔案
apps/ratewise/scripts/generate-robots-txt.mjs- 擴增 AI 爬蟲清單apps/ratewise/public/robots.txt- 自動生成的 robots.txtdocs/SEO_MASTER_SSOT.md- 更新 AI 爬蟲規範文檔AI 爬蟲完整清單（39 個）
Test plan
pnpm typecheck通過pnpm test -- --run1957 個測試全過pnpm test -- --run seo404 個 SEO 測試全過References
Made with Cursor