Fix/worker oom kill - #48
Merged
Merged
Conversation
The Celery worker was being SIGKILL'd (signal 9) during pipeline execution because the ONNX segmentation model + image processing exceeded the container's memory limit. Since SIGKILL cannot be caught, the job's exception handler never ran, leaving it stuck in 'processing' forever. Changes: - docker-compose.yml: Add mem_limit (4g) and memswap_limit (6g) to the worker service, with env overrides (WORKER_MEM_LIMIT/WORKER_MEMSWAP_LIMIT) - celery_app.py: Add task_acks_late + task_reject_on_worker_lost so tasks are re-queued when a worker dies; add task_time_limit (600s) and task_soft_time_limit (540s); add worker_max_memory_per_child (3 GiB) to recycle worker processes before OOM - segmenter.py: Process ONNX inference one page at a time instead of batching all pages, reducing peak memory from N*page to 1*page - cleanup.py: Add cleanup_stale_jobs Beat task (every 5 min) that marks jobs stuck in 'processing' longer than JOB_STALE The Celery worker was being SIGKILL'd (signal 9) during pipeline executs -because the ONNX segmentation model + image processing exceeded the contaxacontainer's memory limit. Since SIGKILL cannot be caught, the job'ERexception handler never ran, leaving it sment new env vars, stale-jo Changes: - docker-compose.yml: Add mem_limit (4g) and memswap_limit Upd- dockeke worker service, with env overrides (WORKER_MEM_LIMIT/WORKER_MEMSWAPal- celery_app.py: Add task_a
- Add typings/onnxruntime/__init__.pyi stub for InferenceSession, get_inputs, and run methods to satisfy strict Pylance/pyright type checking - Add ellipsis body to SemanticSegmenter Protocol method - Cast HoughLinesP result to np.ndarray | None since cv2 stubs claim it never returns None but it can at runtime - Cast _require_image_array return value to fully-typed np.ndarray - Add pyright: ignore[reportPrivateUsage] comments to test file for accessing the private _cleanup_stale_jobs_async helper
The task_acks_late=True setting caused the Redis broker to re-queue un-acked tasks when a worker process was SIGKILL'd, resulting in an infinite retry loop (17+ attempts). autoretry_for's max_retries did not limit these broker-level re-queues. Changes: - Set task_acks_late=False: tasks are acknowledged immediately, so the broker does NOT re-queue them when the worker process dies. Task-level exceptions are still retried via autoretry_for (max_retries=1). - Remove task_reject_on_worker_lost (only relevant with task_acks_late=True). - Keep the task_failure signal handler as immediate back-stop: it fires in the main process when a worker is lost, marking the job as 'failed' in the DB within seconds instead of waiting for the 5-min stale sweeper. - Lower worker_max_memory_per_child from 2 GiB to 1 GiB. - Remove ONNX model preload at worker startup (lazy load via lru_cache). - Lower SEGMENTER_MODEL_INPUT_SIZE from 256 to 128 (75% less input tensor memory per page). - Lower task_time_limit from 600 to 120s and task_soft_time_limit from 540 to 100s (pipeline completes in ~22s, 120s is generous ceiling). - Reduce max_retries from 3 to 1 (OOM retries just OOM again). - Add SEGMENTER_MODEL_INPUT_SIZE to .env.example and docs/operations.md. - Add task_failure to celery signals py.typed stub.
Josh-Uvi
added
bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: stop infinite OOM retry loop and harden job recovery
The task_acks_late=True setting caused the Redis broker to re-queue
un-acked tasks when a worker process was SIGKILL'd, resulting in an
infinite retry loop (17+ attempts). autoretry_for's max_retries did
not limit these broker-level re-queues.
Changes:
broker does NOT re-queue them when the worker process dies. Task-level
exceptions are still retried via autoretry_for (max_retries=1).
in the main process when a worker is lost, marking the job as 'failed'
in the DB within seconds instead of waiting for the 5-min stale sweeper.
memory per page).
540 to 100s (pipeline completes in ~22s, 120s is generous ceiling).