Summary
do_revision_fetch failed every 5 minutes from 2026-06-03 to 2026-07-09 (the gf180mcu-precheck PDK_TAG → PDK_COMMIT incident) and no admin ever got an email, even though ADMINS is configured and prod LOGGING has an AdminEmailHandler. The failures were only visible as raw sha256:... digests on the check-status pages.
Where the errors actually go
Celery workers run with --logfile=${LOGS_DIRECTORY}/worker.log (deployment/systemd/django-celery-http-ro-metadata.service:27), and Celery's default root-logger hijack sends everything there — e.g. /var/log/platform.wafer.space-celery-http-ro-metadata/worker.log. The logger.exception(...) tracebacks from wafer_space/projects/tasks_revisions.py:91 have been landing in that file every ~5 minutes, unseen.
Why no admin email (three independent gaps)
mail_admins is only wired to request loggers. In config/settings/prod.py:138-149 the AdminEmailHandler is attached to django.request and django.security.DisallowedHost only. App loggers (wafer_space.*) propagate to root, and root has just console (prod.py:137). Celery task code can never emit to django.request. No Sentry configured either.- The error path isn't even a task failure.
do_revision_fetch catches the ValueError and returns an error dict (wafer_space/projects/tasks_revisions.py:90-92), so Celery records SUCCESS. Failure-based alerting would not have fired for this bug class.
- Celery 5 has no built-in error emails.
CELERY_SEND_TASK_ERROR_EMAILS was removed in Celery 5, so exhausted-retry failures (e.g. the RequestException path after 3 retries) are also silent.
Recommended fix
- Route app errors to email: add a
wafer_space logger (or extend root) with "handlers": ["mail_admins"] at ERROR level in prod/stage LOGGING.
- Prerequisite: fix the
revisions_needs_fetching infinite requeue first (old, immutable digests still fail every 5 minutes — see the PDK incident), otherwise this change generates ~288 identical emails/day. Options: save partial metadata instead of discarding on missing pdk_version, parse PDK_COMMIT as a Makefile fallback, and/or set a failure marker so permanently-broken digests stop being requeued.
- Complement: a Celery
task_failure signal handler that mails admins (rate-limited), covering genuine task crashes that logging-based routing can miss.
- Consider whether tasks should stop swallowing fatal errors into success-shaped return dicts — a repeated, permanent failure that reports SUCCESS defeats both alerting layers.
Acceptance criteria
- An unhandled exception or
logger.error/logger.exception in any Celery task in prod results in an ADMINS email (deduplicated/rate-limited to something sane).
- A permanently-failing periodic fetch is surfaced once, not re-attempted every 5 minutes forever.
Context: the June incident itself is mitigated (GHCR images relabeled 2026-07-09; wafer-space/gf180mcu-precheck#45 prevents recurrence), but old check digests still fail metadata fetch every 5 minutes until the requeue fix lands.
🤖 Generated with Claude Code
https://claude.ai/code/session_01KMV2xQm2RP5JiTzU22U1sJ
Summary
do_revision_fetchfailed every 5 minutes from 2026-06-03 to 2026-07-09 (the gf180mcu-precheckPDK_TAG→PDK_COMMITincident) and no admin ever got an email, even thoughADMINSis configured and prod LOGGING has anAdminEmailHandler. The failures were only visible as rawsha256:...digests on the check-status pages.Where the errors actually go
Celery workers run with
--logfile=${LOGS_DIRECTORY}/worker.log(deployment/systemd/django-celery-http-ro-metadata.service:27), and Celery's default root-logger hijack sends everything there — e.g./var/log/platform.wafer.space-celery-http-ro-metadata/worker.log. Thelogger.exception(...)tracebacks fromwafer_space/projects/tasks_revisions.py:91have been landing in that file every ~5 minutes, unseen.Why no admin email (three independent gaps)
mail_adminsis only wired to request loggers. Inconfig/settings/prod.py:138-149theAdminEmailHandleris attached todjango.requestanddjango.security.DisallowedHostonly. App loggers (wafer_space.*) propagate to root, and root has justconsole(prod.py:137). Celery task code can never emit todjango.request. No Sentry configured either.do_revision_fetchcatches theValueErrorand returns an error dict (wafer_space/projects/tasks_revisions.py:90-92), so Celery records SUCCESS. Failure-based alerting would not have fired for this bug class.CELERY_SEND_TASK_ERROR_EMAILSwas removed in Celery 5, so exhausted-retry failures (e.g. theRequestExceptionpath after 3 retries) are also silent.Recommended fix
wafer_spacelogger (or extend root) with"handlers": ["mail_admins"]atERRORlevel in prod/stage LOGGING.revisions_needs_fetchinginfinite requeue first (old, immutable digests still fail every 5 minutes — see the PDK incident), otherwise this change generates ~288 identical emails/day. Options: save partial metadata instead of discarding on missing pdk_version, parsePDK_COMMITas a Makefile fallback, and/or set a failure marker so permanently-broken digests stop being requeued.task_failuresignal handler that mails admins (rate-limited), covering genuine task crashes that logging-based routing can miss.Acceptance criteria
logger.error/logger.exceptionin any Celery task in prod results in an ADMINS email (deduplicated/rate-limited to something sane).Context: the June incident itself is mitigated (GHCR images relabeled 2026-07-09; wafer-space/gf180mcu-precheck#45 prevents recurrence), but old check digests still fail metadata fetch every 5 minutes until the requeue fix lands.
🤖 Generated with Claude Code
https://claude.ai/code/session_01KMV2xQm2RP5JiTzU22U1sJ