Injects AI crawler rules (OAI-SearchBot, GPTBot, PerplexityBot, Google-Extended, ClaudeBot and more) into your Magento 2 robots.txt — without overwriting your existing configuration.
Fixes the "robots.txt — AI Bot Access" signal in angeo/module-aeo-audit.
The module intercepts the robots.txt response at render time (via plugin) and prepends a managed block of AI bot rules. No database writes. No file-system changes. Your existing admin config is untouched.
# Angeo AEO — AI Crawler Rules
# https://angeo.dev | module-robots-txt-aeo
# Do not edit this block manually — manage via Stores > Config > Angeo > Robots.txt AEO
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: anthropic-ai
Allow: /
# End Angeo AEO block
User-agent: *
Disallow: /checkout/
... (your existing rules follow unchanged)
Regenerates the full robots.txt. Preserves your custom Disallow rules from the existing wildcard block. Use only if you want this module to own the entire file.
If you prefer to manage AI bot rules yourself directly in Magento admin (Content → Design → Configuration → Edit Custom instruction of robots.txt), you can:
Option A — Disable the module entirely:
bin/magento module:disable Angeo_RobotsTxtAeo
bin/magento cache:flush
Option B — Keep the module but manage specific bots yourself: Disable individual bots in Stores → Configuration → Angeo → Robots.txt AEO → AI Crawlers. The module only injects bots that are enabled in config. Bots you disable are left entirely to your manual configuration.
Option C — Use the module as-is (recommended): The injected block is idempotent and clearly labeled. If you later add a bot manually to admin config, the module will detect the duplicate and skip re-injecting it.
composer require angeo/module-robots-txt-aeo
bin/magento setup:upgrade
bin/magento cache:flushRequirements: PHP 8.2+, Magento 2.4+. Compatible with Magento Open Source and Adobe Commerce Cloud.
Stores → Configuration → Angeo → Robots.txt AEO
| Setting | Default | Description |
|---|---|---|
| Enable AI Bot Rules | Yes | Master on/off switch |
| Injection Mode | Inject | Inject (prepend) or Replace (full file) |
| OAI-SearchBot | Yes | ChatGPT live search crawler |
| GPTBot | Yes | OpenAI training crawler |
| ChatGPT-User | Yes | ChatGPT user-triggered browsing |
| PerplexityBot | Yes | Perplexity background indexer |
| Perplexity-User | Yes | Perplexity real-time fetch |
| Google-Extended | Yes | Gemini / AI Overviews |
| ClaudeBot | Yes | Anthropic Claude citations |
| anthropic-ai | Yes | Anthropic training crawler |
Angeo → Robots.txt AEO in the Magento admin menu.
The dashboard provides:
- Validate live robots.txt — fetches
yourstore.com/robots.txtand shows pass/fail status for each enabled bot - Preview after injection — shows the full robots.txt output with syntax highlighting, without making any changes
- Per-bot status table updated in real time via AJAX
- Direct link to live robots.txt and Settings
No CLI access required — non-technical store owners can validate and preview from the browser.
# Preview what robots.txt will look like after injection (no changes made)
bin/magento angeo:robots:preview
# Preview with diff — show only lines being added
bin/magento angeo:robots:preview --diff
# Validate that AI rules are present in the live robots.txt
bin/magento angeo:robots:validate
# Validate against a specific store URL
bin/magento angeo:robots:validate --url=https://yourstore.comOn Adobe Commerce Cloud, robots.txt is served via a Fastly VCL snippet. After any configuration change:
- Save config in admin
- Purge the Fastly CDN cache
- Run
bin/magento angeo:robots:validateto confirm the live file is updated
# Fail the build if AI bot rules are missing
bin/magento angeo:robots:validate || exit 1| Module | Purpose | Signal |
|---|---|---|
angeo/module-aeo-audit |
AEO audit — detects missing signals | — |
angeo/module-robots-txt-aeo |
This module — AI bot access | Signal #1 |
angeo/module-llms-txt |
Generates llms.txt and llms.jsonl | Signal #2 |
angeo/module-rich-data |
Product / FAQ / Org JSON-LD schema | Signal #3 |
angeo/module-openai-product-feed |
ACP product feed for ChatGPT Shopping | Signal #4 |
MIT — see LICENSE