Crawler hits on bare OAuth callback URL trigger admin error emails; add robots.txt

[mithro]

Symptom

Weekly [wafer.space] ERROR (EXTERNAL IP): Social authentication error for google: unknown emails to ADMINS (18 Jul, 25 Jul, 1 Aug, 8 Aug, 16 Aug 2026 — all ~04:00–05:00 UTC; the older three say google: str because fed2400 changed the formatting in between).

Root cause

Not a Google login failure. A crawler (Amazonbot, from AWS IPs 52.54.15.103 / 23.21.204.95) fetches the bare OAuth callback URL on a weekly recrawl:

Request URL: https://platform.wafer.space/accounts/google/login/callback/
GET: No GET data
COOKIES: No cookie data
QUERY_STRING = ''
HTTP_USER_AGENT = 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36'
Traceback (most recent call last):
  None

Code path (django-allauth 65.11.2):

1. allauth/socialaccount/providers/oauth2/views.py:180-202 — OAuth2CallbackView._get_state(): no state param → state is None → render_authentication_error(request, provider, extra_context={"state_id": None, ...}) with default error=AuthError.UNKNOWN and no exception.
2. allauth/socialaccount/helpers.py:25 → SocialAccountAdapter.on_authentication_error() (wafer_space/users/adapters.py:132).
3. adapters.py:174 only special-cases AuthError.CANCELLED; everything else is logger.error(...), and the wafer_space logger routes ERROR to RateLimitedAdminEmailHandler (config/settings/prod.py:141) → email.

A real callback always carries ?state=…&code=… (or ?error=…) plus a session cookie; a request with none of those is not an OAuth flow at all.

Fix

1. Adapter: in on_authentication_error, when exception is None and request.GET has neither code nor error (equivalently "state_id" in extra_context), log at warning instead of error (still queue the user-facing flash message). Add a RequestFactory test hitting the callback with no query string and asserting no ERROR record.
2. robots.txt: the repo has no robots route at all. Add one (config/urls.py + template) that disallows the pages crawlers have no business fetching, e.g.:

User-agent: *
Disallow: /accounts/
Disallow: /admin/
Disallow: /users/

Review the URLconf for other login/auth/redirect-only paths (SOCIALACCOUNT_LOGIN_ON_GET = True means /accounts/<provider>/login/ also acts on GET, which is how the callback link gets discovered).

Diagnosed in Claude Code session https://claude.ai/code/session_016Giui56Ajp2cwguVD3JBzy.