`Maho_ContentNegotiation`: serve markdown to AI agents via Accept header / .md suffix

[fballiano]

Maho_ContentNegotiation: Markdown for AI agents

Context

LLM agents that crawl ecommerce sites pay a significant token cost to consume HTML pages laden with navigation, scripts, styles, and tracking. Cloudflare, Vercel, and others have converged on a content-negotiation pattern: when a client sends Accept: text/markdown (or fetches the same URL with a .md suffix), the origin returns a stripped-down markdown rendering of the page's main content. Cloudflare reports up to ~99% payload reduction in their examples, and modern agent clients (Claude Code, OpenCode) already send the header by default.

This module implements that pattern for Maho. It does not publish llms.txt or modify sitemap.xml (deferred to a follow-up). The goal is purely content negotiation: the same URL serves HTML to browsers and markdown to agents that ask for it.

Approach

Single new module Maho_ContentNegotiation under app/code/core/Maho/. The HTTP-protocol framing leaves room for future variants (e.g. Accept: application/ld+json or other agent-facing formats) without renaming. Two observer-driven interception points:

1. controller_front_dispatch_before: runs before the router match loop. Detects whether the request is for markdown (Accept header OR .md path suffix). If so:

   • Strips the trailing .md from \$request->getPathInfo() so routing matches normally.
   • On hit, sets a request param flag _agent_markdown for the second hook.
   • Otherwise returns silently (request flows through unchanged, agent gets HTML).

2. controller_front_send_response_before: runs after the full page has rendered into \$response->getBody(). Checks the flag:

   • Validates the resolved route name against the configurable whitelist of route prefixes; on miss, returns the rendered HTML untouched.
   • Cache lookup keyed by current_store_id + path_info + locale.
   • On hit, replaces the body with the cached markdown.
   • On miss, calls Mage::app()->getLayout()->getBlock('content')->toHtml() to get just the main content area (re-render is cheap once per page-version; cache amortizes), runs it through League\HTMLToMarkdown\HtmlConverter with chrome-stripping options, prepends an H1 derived from the page title, caches the result with MAHO_CONTENT_NEG_MD tags, and replaces the body.
   • In either case, sets Content-Type: text/markdown; charset=utf-8 and Vary: Accept.

Cache is invalidated via observers on catalog_product_save_after, catalog_product_delete_after, catalog_category_save_after, catalog_category_delete_after, cms_page_save_after, and cms_page_delete_after. All call Mage::app()->getCache()->clean(['MAHO_CONTENT_NEG_MD']).

Why these interception points

• controller_front_dispatch_before (Mage_Core_Controller_Varien_Front:165) is the latest hook that still runs before route matching, so a .md strip here is invisible to all downstream code.
• controller_front_send_response_before (Mage_Core_Controller_Varien_Front:172) is the only point where the full HTML body exists in the response AND the layout block tree is still live, so we can re-grab content without parsing the body.
• Re-rendering the content block (rather than parsing the body with DOMDocument) keeps us theme-agnostic: every Maho theme uses the same <block name=\"content\"> reference from page.xml.

Files to create

```
app/etc/modules/Maho_ContentNegotiation.xml
app/code/core/Maho/ContentNegotiation/
├── Helper/
│ └── Data.php # config getters: isEnabled, getAllowedPathPrefixes, shouldRespondAsMarkdown
├── Model/
│ ├── Observer.php # 2 dispatch observers + 6 cache-invalidation observers (PHP attributes)
│ └── Converter.php # thin wrapper around HtmlConverter with Maho-specific options + H1 prepending
└── etc/
├── config.xml # module version, cache type declaration, default values
└── system.xml # admin/contentnegotiation section: enable toggle + path-prefix textarea
```

Files to modify

• `composer.json`: add `"league/html-to-markdown": "^5.1"` to `require` (already transitively present via `symfony/mime`, but declaring it directly is best practice).

Existing code to reuse

• `League\HTMLToMarkdown\HtmlConverter`: already in vendor via `symfony/mime`. Confirmed at `vendor/symfony/mime/HtmlToTextConverter/LeagueHtmlToMarkdownConverter.php:14`. Use options `remove_nodes => 'header footer nav script style aside iframe noscript form button'`, `strip_tags => true`, `hard_break => true`, `use_autolinks => true`.
• `Mage_Core_Helper_Abstract`: base class for `Helper/Data.php`. Mirror the constants pattern from `app/code/core/Maho/Captcha/Helper/Data.php:20-35`.
• `\Maho\Event\Observer`: event payload type for `#[Maho\Config\Observer]` handlers (see `app/code/core/Maho/Captcha/Model/Observer.php`).
• `Mage::app()->getCache()`: backend cache with tag invalidation. Pattern from `app/code/core/Maho/ApiPlatform/symfony/Trait/CacheTrait.php:36-66`. Tag prefix: `MAHO_CONTENT_NEG_MD`.
• `Mage::app()->getLayout()->getBlock('content')`: main content reference, established in `app/design/frontend/base/default/layout/page.xml:83`.
• `Mage::app()->getRequest()->getHeader('Accept')` and `getPathInfo()` / `setPathInfo()` for content negotiation.

Config defaults (system.xml)

• `contentnegotiation/markdown/enabled`: boolean, default `1`
• `contentnegotiation/markdown/allowed_prefixes`: textarea, default value (one per line):
  ```
  catalog/category/view
  catalog/product/view
  cms/page/view
  cms/index
  blog
  ```
• `contentnegotiation/markdown/cache_ttl`: text (integer seconds), default `86400`

The whitelist is matched against the route name (e.g. `catalog/product/view`) resolved at dispatch time, not the literal URL, so rewrites (catalog URL rewrites, slug-based URLs) work uniformly. The match is "starts with any line from the textarea".

Detection logic (Helper/Data.php)

```php
public function shouldRespondAsMarkdown(Mage_Core_Controller_Request_Http $request): bool
{
if (!$this->isEnabled()) return false;
if ($request->isPost()) return false; // never on POSTs

\$pathHasMdSuffix = str_ends_with(\$request->getPathInfo(), '.md');
\$acceptsMarkdown = str_contains((string) \$request->getHeader('Accept'), 'text/markdown');

return \$pathHasMdSuffix || \$acceptsMarkdown;

}
```

Conversion logic (Model/Converter.php)

```php
public function convert(string $contentHtml, ?string $pageTitle, ?string $metaDescription): string
{
$converter = new \League\HTMLToMarkdown\HtmlConverter([
'strip_tags' => true,
'remove_nodes' => 'header footer nav script style aside iframe noscript form button',
'hard_break' => true,
'use_autolinks' => true,
]);

\$markdown = \$converter->convert(\$contentHtml);

\$prefix = '';
if (\$pageTitle) {
    \$prefix .= '# ' . \$pageTitle . \"\n\n\";
}
if (\$metaDescription) {
    \$prefix .= '> ' . \$metaDescription . \"\n\n\";
}

return \$prefix . \$markdown;

}
```

Page title and meta description are read from `Mage::app()->getLayout()->getBlock('head')` if available.

Observer skeleton (Model/Observer.php)

```php
class Maho_ContentNegotiation_Model_Observer
{
#[Maho\Config\Observer('controller_front_dispatch_before')]
public function detectMarkdownRequest(\Maho\Event\Observer $observer): void { /* strip .md, set flag */ }

#[Maho\Config\Observer('controller_front_send_response_before')]
public function convertResponseToMarkdown(\Maho\Event\Observer \$observer): void { /* render content, convert, cache, swap body */ }

#[Maho\Config\Observer('catalog_product_save_after')]
#[Maho\Config\Observer('catalog_product_delete_after')]
#[Maho\Config\Observer('catalog_category_save_after')]
#[Maho\Config\Observer('catalog_category_delete_after')]
#[Maho\Config\Observer('cms_page_save_after')]
#[Maho\Config\Observer('cms_page_delete_after')]
public function invalidateMarkdownCache(\Maho\Event\Observer \$observer): void { /* clean cache tag */ }

}
```

The route-name match for whitelist evaluation happens inside `convertResponseToMarkdown` (after dispatch, the route name is known via `$request->getRouteName()`), not at `dispatch_before`. So the dispatch_before observer only handles `.md` stripping and Accept-header detection; the whitelist gate runs at response time. This is fine because the user gets a full HTML body anyway if not whitelisted, we just stop short of converting.

Verification

1. Tests: `composer test -- --testsuite=Frontend`. Add a Pest test under `tests/Frontend/` that:

   • Requests a product page with `Accept: text/markdown`: asserts `Content-Type: text/markdown` and a `# Product Name` first line.
   • Requests `/some-product-url.md` directly: same assertions.
   • Requests the same product as HTML: asserts unchanged behavior, `Vary: Accept` header present.
   • Requests `/customer/account` with `Accept: text/markdown`: asserts HTML returned (not in whitelist).
   • Saves a product, then re-requests its `.md` URL: asserts response differs (cache invalidated).

2. Manual smoke:
   ```bash
   curl -i -H 'Accept: text/markdown' http://maho.local/some-product
   curl -i http://maho.local/some-product.md
   curl -i http://maho.local/some-product # baseline, still HTML
   ```

3. Static analysis: `vendor/bin/phpstan analyze` and `vendor/bin/php-cs-fixer fix`.

4. Visual check: open the markdown response in any markdown viewer and confirm nav/footer are gone, product description / category text is preserved, and links are intact.

5. Cache check: hit a `.md` URL twice; the second should be measurably faster (and the cache backend should show a `MAHO_CONTENT_NEG_MD`-tagged entry).

References

• Cloudflare: Introducing Markdown for Agents
• Vercel: Making agent-friendly pages with content negotiation
• Why Sentry Serves Markdown Over HTML

[aagarwal1012]

This is the same pattern Dualmark (github) implements for JS frameworks — and there's an open spec a Maho/PHP implementation could conform to so it's interoperable.

Two pieces:

1. The AEO Spec (RFC-2119 formatted, Apache-2.0) — what Maho_ContentNegotiation could match. Shape:

Request	Response
Accept: text/markdown	text/markdown; charset=utf-8 + X-Markdown-Tokens + X-Robots-Tag: noindex + Vary: Accept
Known AI-bot UA (GPTBot, ClaudeBot, PerplexityBot, +16 more)	same as above
Direct /path.md URL	same as above
Anything else	HTML, with Link: <…>; rel=\"alternate\"; type=\"text/markdown\"

Plus a /llms.txt at the site root generated from the same routing config so it can't drift.

2. A conformance CLI — bunx @dualmark/cli verify https://store.example.com/product/sku scores 0–125 against the spec with line-item failures. Useful for catching regressions in CI once Maho_ContentNegotiation ships, regardless of language.

The PHP port should be straightforward — the heavy lifting is the `negotiateFormat` RFC 7231 §5.3.2 parser and the bot UA registry. Happy to answer spec questions if useful.

No affiliation with Maho — we built this for our own marketing site at Dodo Payments and open-sourced it.

[hirale]

https://github.com/webmachinelearning/webmcp

webMCP is more about exposing structured machine-readable interaction metadata at the frontend/runtime level, so AI agents can interact with web applications more reliably.

Probably still early, but the direction feels relevant.

[fballiano]

@hirale totally agree, but after thinking about both things it seems to me the contentnegotiation would help for agents coming "cold" on the website, where the webmcp is a deeper layer of interaction, probably complementary in my opinion.