Skip to content

fix: avoid gevent signal shutdown error - #274

Merged
suguanYang merged 1 commit into
mainfrom
fix/wangbinqi/fix-gevent-shutdown
Aug 13, 2026
Merged

fix: avoid gevent signal shutdown error#274
suguanYang merged 1 commit into
mainfrom
fix/wangbinqi/fix-gevent-shutdown

Conversation

@suguanYang

Copy link
Copy Markdown
Contributor

Summary

  • remove REMAP_SIGTERM=SIGQUIT from the staging worker task definition
  • keep ECS stopTimeout=120 and Celery soft-shutdown settings
  • update the worker shutdown contract to require the normal SIGTERM path

Root cause

ECS sends SIGTERM. The remap routed it into Celery 5.5.3's cold-shutdown handler, which calls its soft-shutdown sleep from a gevent signal/event-loop callback. The staging drill reproduced gevent.exceptions.BlockingSwitchOutError.

The normal SIGTERM warm-shutdown path avoids that incompatible callback while retaining late acknowledgements and ECS task replacement behavior.

Verification

  • uv run --directory /home/suguan/github.com/ontosAI/knowhere pytest -q apps/worker/tests/contract/test_worker_shutdown_contract.py apps/worker/tests/contract/test_worker_bootstrap_contract.py
  • Result: 3 passed

Staging deployment and interruption re-test are intentionally deferred until this PR is reviewed and merged.

@suguanYang
suguanYang merged commit 7630d02 into main Aug 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant