LO-062 [Intermediate]: Celery Beat Monitoring Health Check Endpoint

[Kuldeeep18]

Description

Our platform relies on Celery Beat to trigger lead processing every minute. If Celery Beat hangs or crashes, outreach sequences stop running silently. We need a health-check endpoint that verifies if the cron runner is alive.

User & Contributor Value

• Contributors: Caching state checks, background heartbeat updates, and creating systems monitoring APIs.
• Users: Enables devops teams to monitor schedule health and auto-restart stuck worker containers.

Code Locations

• backend/backend/urls.py
• backend/campaigns/tasks.py

Implementation Guide

1. Heartbeat Task: Write a simple periodic task celery_heartbeat() that runs every minute, updating a Redis/Cache key celery_beat_heartbeat with the current timestamp.
2. API View: Add a route /api/v1/health/celery-beat/. It should check if the cache key exists and is less than 3 minutes old, returning HTTP 200 (healthy) or HTTP 503 (stuck).

[manikanta-tamminana]

Hi @Kuldeep18,

I would like to work on this issue under GSSoC.

My approach would be:

• Implement a periodic celery_heartbeat() task that runs every minute via Celery Beat and updates a cache/Redis key with the current timestamp.

• Create a dedicated health-check API endpoint at /api/v1/health/celery-beat/.

• Validate the heartbeat timestamp and return:

  • HTTP 200 when the heartbeat is present and less than 3 minutes old.
  • HTTP 503 when the heartbeat is missing or stale, indicating a potential Celery Beat failure.

• Ensure the implementation is lightweight, cache-backed, and suitable for monitoring systems and container orchestration platforms.

• Add clear response messages to aid debugging and operational monitoring.

This feature will improve platform reliability by providing a simple way to detect Celery Beat failures and trigger automated recovery mechanisms. Could you please assign this issue to me?

Thank you! 🚀

[NikithaDineshKumar]

Hi @Kuldeeep18, I would love to implement this Celery Beat health check endpoint! Here is my proposed approach:

1. Background Heartbeat Task (backend/campaigns/tasks.py)
   Task Definition: I will create a periodic Celery task named celery_heartbeat() configured to execute every minute.

Cache Update: The task will fetch the current timestamp and write it to the Redis/Cache using a dedicated key, such as celery_beat_heartbeat.

2. Health Check API (backend/backend/urls.py)
   API Route & View: I will register a new endpoint at /api/v1/health/celery-beat/.

Timestamp Evaluation: The view will look up the celery_beat_heartbeat cache key and calculate the time difference.

Response Logic:

If the key exists and the timestamp is less than 3 minutes old, it will return an HTTP 200 OK status indicating a healthy scheduler.

If the key is missing or older than 3 minutes, it will return an HTTP 503 Service Unavailable status to signal that Celery Beat is stuck or down.

Could you please assign this issue to me? Thanks!

/assign

[aeraa1725-bot]

Hi @Kuldeeep18,

I would like to work on this issue. I have reviewed the requirements and would be happy to implement the Celery Beat heartbeat task, health-check endpoint, and corresponding tests.

Could you please assign this issue to me?

Thank you!