Track llms.txt requests so we know which AIs are hitting the site

[johnnymatthews]

Track llms.txt requests so we know which AIs are hitting the site

llms.txt is a static file — crawlers fetch it directly, no JS ever runs. Plausible only sees pageviews when its script fires in a browser, so right now we have zero visibility into who's actually requesting it.

We want a count of llms.txt fetches, ideally broken down by which bot/AI made the request (GPTBot, ClaudeBot, PerplexityBot, CCBot, etc).

Tasks

We're on Cloudflare Pages, so we can intercept the request with a Pages Function instead of serving llms.txt as a static asset.

1. Move the real file content to something like llms-content.txt so it doesn't collide with the function.
2. Add functions/llms.txt.js. On each request it:
   • fetches the real content and returns it like normal
   • fires a server-side POST to Plausible's Events API with the request's User-Agent, using context.waitUntil() so it doesn't block the response

The URL a crawler hits stays /llms.txt the whole time. Moving the content file is just internal plumbing — doesn't affect how anything following the llms.txt spec resolves the file.

That being said, there are some things to watchout for:

• Plausible needs the User-Agent passed manually in the event call. The request to Plausible's API comes from our function, not the crawler, so it won't pick it up automatically.
• Breaking down by bot name needs Plausible's custom properties feature, which is a paid-plan thing. If we're not on that plan, we can fake the breakdown by mapping known UA substrings to separate event names (llms_txt_fetch_gptbot, llms_txt_fetch_claudebot, etc) instead of relying on props.
• Most bots skip referrers and JS but do send a UA string, so this should catch the majority of them.