Skip to content

fix(ci): add production background job E2E verification - #799

Merged
manavgup merged 3 commits into
mainfrom
fix/588-prod-job-e2e-verification
May 23, 2026
Merged

fix(ci): add production background job E2E verification#799
manavgup merged 3 commits into
mainfrom
fix/588-prod-job-e2e-verification

Conversation

@manavgup

Copy link
Copy Markdown
Owner

Summary

  • Add a lightweight ping ARQ job that returns "pong" immediately, proving the full background pipeline works (API -> Redis -> ARQ worker -> result)
  • Add /health/job-ping endpoint that enqueues the ping, polls for completion with a 30s timeout, and returns structured status with latency
  • Update both staging smoke tests and production verification in deploy.yml to call this endpoint and fail the deploy if the job does not complete

Test plan

  • make verify passes (2088 tests pass, lint clean, types clean)
  • New /health/job-ping endpoint appears in regenerated OpenAPI spec
  • Staging deploy exercises the new smoke test step
  • Production deploy exercises the new verification step
  • Endpoint returns {"status": "skipped"} in dev mode (no Redis)
  • Endpoint returns {"status": "ok", "result": "pong"} in prod mode with healthy worker

Closes #588

🤖 Generated with Claude Code

Comment thread src/wikimind/api/routes/health.py
@github-actions
github-actions Bot force-pushed the fix/588-prod-job-e2e-verification branch from 81d1b32 to 84121e8 Compare May 23, 2026 19:48
@manavgup
manavgup force-pushed the fix/588-prod-job-e2e-verification branch from 84121e8 to 5b53a37 Compare May 23, 2026 19:55
manavgup and others added 2 commits May 23, 2026 20:03
Production deploy verification previously only checked that
background_mode == "arq" in the health response, which proves
config is set but not that jobs actually execute. Add a lightweight
ping job that proves the full pipeline: API -> Redis -> ARQ worker ->
result.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove /health/job-ping from auth EXEMPT_PATHS so it requires
  authentication (prevents unauthenticated abuse of side-effecting endpoint)
- Add Depends(get_current_user_id) to the job_ping route handler
- Move job-ping check after auth token exchange in both staging and
  production verification steps so the Bearer token is available
- Remove acceptance of status=="skipped" — contradicts the background_mode
  == "arq" check that runs first (if mode is arq, skipped is an error)
- Add retry with backoff (5 attempts, 5/10/15/20/25s delays) to handle
  worker startup delays after deploy

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions
github-actions Bot force-pushed the fix/588-prod-job-e2e-verification branch from 5b53a37 to 1c10395 Compare May 23, 2026 20:03
@manavgup
manavgup merged commit 21ddd39 into main May 23, 2026
16 checks passed
@manavgup
manavgup deleted the fix/588-prod-job-e2e-verification branch May 23, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: add real background job E2E verification for production deploys

2 participants