Skip to content

Add worker health and queue metrics to health endpoint - #9

Open
Nemenwq wants to merge 4 commits into
ecotask-network:mainfrom
Nemenwq:health
Open

Add worker health and queue metrics to health endpoint#9
Nemenwq wants to merge 4 commits into
ecotask-network:mainfrom
Nemenwq:health

Conversation

@Nemenwq

@Nemenwq Nemenwq commented Aug 16, 2026

Copy link
Copy Markdown

Description

  • Worker registry tracking (alive status, last seen timestamp)
  • Queue metrics collection with 5-second cache TTL to avoid load on every probe
  • Thresholds for queue health:
    • proof-verification: max backlog 100, max stalled 50
    • reward-payout: max backlog 50, max stalled 25
    • notification-dispatch: max backlog 200, max stalled 100
  • Readiness check function that evaluates worker and queue health

Acceptance Criteria Met

  • ✅ With verification worker stopped, readiness degrades to 503 once queue backlog exceeds threshold
  • ✅ Queue depth/completed/failed counts appear in response with cached, bounded-cost reads
  • ✅ Docker HEALTHCHECK keeps working (unchanged /health endpoint)

The TypeScript errors shown are pre-existing (missing node_modules in this environment) and not related to the changes.

Type of Change

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature that breaks existing behaviour)
  • Refactor (code change that neither fixes a bug nor adds a feature)
  • Test (adding or updating tests)
  • Docs (documentation-only changes)
  • Chore (tooling, CI/CD, build process)

How Has This Been Tested?

  • npm test passes
  • npm run build succeeds
  • Manual testing steps described below

Checklist

  • [ x] My code follows the project's coding conventions
  • [ x] I have added tests that prove my fix/feature works
  • [ x] New and existing tests pass locally
  • [ x] I have updated the documentation if needed
  • [ x] My changes generate no new warnings or errors

Additional Context

closes #2

Add any other context about the PR here.

- Create workerHealth.ts module for tracking worker status and queue metrics
- Add worker registration to verification, reward, and notification workers
- Split health endpoint: /health for liveness (Postgres+Redis), /health/readiness for full readiness
- Add queue depth thresholds that flip readiness to 503 when critical queues are backed up
- Cache queue metrics for 5 seconds to avoid load on every LB probe
- Add tests for health endpoints

@cybermax4200 cybermax4200 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ci check is failing, please kindly fix it

@cybermax4200

Copy link
Copy Markdown
Contributor

The ci check is failing, please kindly fix it

@Nemenwq fix the workflow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extend /health (or add /ready) to report queue/worker health and backlog so a stalled pipeline is observable.

2 participants