Fix/worker oom kill

[Josh-Uvi]

Fix: Celery Worker OOM Kill and Stale Job Recovery

Root Cause

The Celery worker was being killed by SIGKILL (signal 9) during pipeline execution because the ONNX segmentation model + image processing exceeded the container's memory limit. Since SIGKILL cannot be caught, the job's exception handler never ran, leaving jobs stuck in processing forever.

Changes (2 commits on fix/worker-oom-kill)

Commit 1 (db87429): OOM prevention and stale job recovery

1. docker-compose.yml — Added mem_limit: 4g and memswap_limit: 6g to the worker service with env overrides (WORKER_MEM_LIMIT/WORKER_MEMSWAP_LIMIT). Added JOB_STALE_TIMEOUT_SECONDS to worker and beat environments.

2. backend/app/tasks/celery_app.py — Added Celery safety settings:

   • task_acks_late=True + task_reject_on_worker_lost=True — re-queue tasks when a worker dies
   • task_time_limit=600 / task_soft_time_limit=540 — prevent hung tasks
   • worker_max_memory_per_child=3 GiB — recycle worker processes before OOM
   • Added cleanup-stale-jobs beat schedule (every 5 minutes)

3. backend/app/pipeline/steps/segmenter.py — Refactored OnnxSemanticSegmenter.segment() to process pages one at a time instead of batching all pages into a single np.stack. Peak memory reduced from N_pages × page_tensor_size to 1 × page_tensor_size.

4. backend/app/tasks/cleanup.py — Added cleanup_stale_jobs task that finds jobs stuck in processing for longer than JOB_STALE_TIMEOUT_SECONDS and marks them as failed.

5. backend/app/core/config.py — Added JOB_STALE_TIMEOUT_SECONDS setting (default 300s).

6. .env.example — Documented new env vars.

7. docs/operations.md & docs/risks-and-edge-cases.md — Documented new settings, cleanup tasks, and OOM troubleshooting.

8. backend/tests/test_stale_job_recovery.py — 3 new tests for stale job recovery.

Commit 2 (63ef6c1): Pylance type fixes

1. typings/onnxruntime/__init__.pyi — New type stub for InferenceSession, get_inputs, run methods to satisfy strict Pylance/pyright type checking.

2. backend/app/pipeline/steps/segmenter.py:

   • Added ellipsis body to SemanticSegmenter Protocol method
   • Cast HoughLinesP result to np.ndarray | None since cv2 stubs claim it never returns None but it can at runtime
   • Cast _require_image_array return value to fully-typed np.ndarray

3. backend/tests/test_stale_job_recovery.py — Added # pyright: ignore[reportPrivateUsage] comments for accessing the private _cleanup_stale_jobs_async helper.

Defense in Depth

1. Prevent OOM: Memory limit (4g) + worker_max_memory_per_child (3 GiB) + per-page segmenter processing
2. Recover from OOM: task_acks_late re-queues the task on a fresh worker
3. Recover orphaned jobs: Stale-job sweeper marks stuck processing jobs as failed after 5 minutes
4. Prevent hung tasks: Hard/soft time limits (600s/540s) ensure workers don't block indefinitely