Skip to content

fix: re-raise exception in ingest_document so Celery retry mechanism fires (#561)#582

Open
ionfwsrijan wants to merge 10 commits into
param20h:devfrom
ionfwsrijan:fix/issue-561
Open

fix: re-raise exception in ingest_document so Celery retry mechanism fires (#561)#582
ionfwsrijan wants to merge 10 commits into
param20h:devfrom
ionfwsrijan:fix/issue-561

Conversation

@ionfwsrijan

Copy link
Copy Markdown
Contributor

Description

ingest_document() wraps its entire pipeline in a try/except block. The except Exception handler logs the error, rolls back the DB session, marks the document as failed, and commits. However, the exception is completely consumed — never re-raised.

Meanwhile, the Celery task process_document is decorated with autoretry_for=(Exception,), max_retries=3, acks_late=True — all designed to handle transient failures. But because ingest_document never propagates an exception, none of them ever fire.

Changes

backend/app/services/document_ingestion.py

  • Added raise after the except block so the original exception propagates to Celery's autoretry_for after DB state is cleaned up (rollback, mark failed, commit).

backend/app/tasks.py

  • Changed the return value to query the document's actual DB status after ingest_document returns, instead of hardcoding "completed". This ensures the task result accurately reflects success or failure.

Impact

  • Transient failures (API timeout, DB hiccup) now trigger automatic Celery retries
  • Task result status matches DB status consistently
  • Framework-level retry configuration (max_retries=3, default_retry_delay=30) is now functional

Closes #561

@ionfwsrijan ionfwsrijan requested a review from param20h as a code owner June 13, 2026 10:27
@ionfwsrijan

Copy link
Copy Markdown
Contributor Author

@param20h Please review this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Silent Exception Swallowing in ingest_document Breaks Celery Retry

1 participant