feat(imports): recover an import whose background job was killed

[Rl0007]

Three small changes so a killed worker doesn't leave an Import unrecoverable. A worker
died three times mid-job on a 398-page import last night; each time the Import was left
in Remediating at 0% with no way back except hand-editing status through the API.

1. The stale error is cleared when a run starts

error is written in jobs/parse.py, jobs/remediate.py and jobs/generate.py (each in
an except block) and was never cleared again — a traceback from a failed run weeks ago
stayed on the import and was displayed while a fresh run was in progress. Each job now
clears it alongside the status flip that starts the run.

2. retry_import — a whitelisted way back

trigger_remediation only runs from Review, and generate_wiki only from
Graphed/Completed, so an import in Failed or stuck in a running status had no
whitelisted entry point at all. wikify.api.imports.retry_import(import_name):

frappe.call("wikify.api.imports.retry_import", { import_name: "IMP-2026-00001" })

• runs from Failed, or from a running status whose progress has gone stale — a live job
  is never interrupted (frappe.throw otherwise);
• clears the error;
• re-enqueues the parse job when nothing was parsed yet, otherwise hands the import back
  to the stage before the one that died (Graphed for a lost wiki generation, else
  Review) so remediation or generation can be re-run from the UI. The parse result and
  any approved tree are left intact.

3. fail_stuck_imports — stuck jobs stop being invisible

New scheduled task (wikify/tasks.py), wired as a cron every 15 minutes, that marks an
import Failed when it has been in Queued / Parsing / Remediating /
Generating Wiki with no progress written for 30 minutes, and logs the reason to the
import's log.

It keys off time-since-last-progress, not total runtime: publish_progress writes on
every page, so the real 398-page remediation (~50 minutes wall clock, a write every few
seconds) is never touched, while a worker that died 30 minutes ago is.

Registered on the dev site after bench migrate:

[{"name": "0su62l04bh", "method": "wikify.tasks.fail_stuck_imports", "frequency": "Cron", "cron_format": "*/15 * * * *"}]

Tests

wikify/tests/test_job_recovery.py — 10 tests over the real DB covering all three
changes: both jobs clearing the earlier error, each retry_import branch (resume,
re-parse, refuse a live job, refuse a completed import, recover a stale one), and the
scheduled task failing a stuck import while leaving a slow-but-reporting job and an
import at rest alone.

Both error-clearing tests were verified red before the change:

 x  test_generate_job_clears_the_error_of_an_earlier_run
 x  test_remediate_job_clears_the_error_of_an_earlier_run
AssertionError: 'Traceback (most recent call last):\n  RuntimeError: from a run in August' is not None
FAILED (failures=2)

Full suite (bench --site wikify.localhost run-tests --app wikify):

 x  test_renders_with_wiki_renderer
 FAIL  test_renders_with_wiki_renderer (wikify.tests.test_wiki_preview.TestWikiPreview.test_renders_with_wiki_renderer)
AssertionError: '<pre class="mermaid">' not found in ...
Ran 191 tests in 58.967s
FAILED (failures=1)

test_renders_with_wiki_renderer also fails on main (the installed wiki app no longer
emits <pre class="mermaid">) and is unrelated to this change. Everything else passes.

No UI is wired to retry_import in this PR — it is the API half only.

🤖 Generated with Claude Code

https://claude.ai/code/session_0126A9jCM6LTxnmCPH3hWbTR

[greptile-apps]

Greptile Summary

This PR adds recovery for imports abandoned by killed background workers, clears stale job errors, and schedules detection of imports with no recent progress.

• Adds a whitelisted retry endpoint that either re-enqueues parsing or returns an import to a prior workflow gate.
• Adds a 15-minute scheduler task that marks running imports Failed after 30 minutes without progress.
• Clears previous error text when parse, remediation, and generation jobs begin.
• Adds database-backed recovery tests for retry branches, stale detection, and error clearing.

Confidence Score: 2/5

The recovery paths need correction before merging because they can restore imports to the wrong workflow gate, expose incomplete parses as review-ready, and race with workers that resume reporting progress.

The scheduler discards the stage needed for generation recovery, retry relies on source_document before the parse pipeline is necessarily complete, and both stale-recovery paths mutate state after non-atomic liveness checks.

Files Needing Attention: wikify/api/imports.py, wikify/tasks.py

Important Files Changed

Filename	Overview
wikify/api/imports.py	Adds retry orchestration, but loses generation-stage provenance after scheduled failure and mistakes a partially completed parse for a review-ready artifact.
wikify/tasks.py	Adds stale-import detection, but replaces stage-specific state with generic Failed and performs the failure write without atomically revalidating staleness.
wikify/jobs/parse.py	Clears stale errors at job start; its existing ordering demonstrates that source_document can be present before tree construction completes.
wikify/jobs/remediate.py	Clears a previous error when remediation starts, with no independent issue identified.
wikify/jobs/generate.py	Clears a previous error when generation starts, with no independent issue identified.
wikify/hooks.py	Registers the new stale-import task on a 15-minute cron schedule.
wikify/tests/test_job_recovery.py	Covers individual retry and scheduler branches but misses scheduler-to-retry generation recovery, partial parse completion, and concurrent progress races.

Prompt To Fix All With AI

### Issue 1
wikify/api/imports.py:125-127
**Generation stage is lost**

When the scheduler marks a stale `Generating Wiki` import as `Failed`, this branch no longer knows which stage failed and returns the import to `Review` instead of `Graphed`, causing `generate_wiki` to reject the retry until the tree is manually approved again.

### Issue 2
wikify/api/imports.py:112-127
**Partial parses become review-ready**

When parsing dies after storing `source_document` but before remediation, tree rebuilding, or classification finishes, this check skips parse enqueueing and moves the import directly to `Review`, exposing an incomplete or empty section tree as ready for review.

### Issue 3
wikify/tasks.py:37-47
**Staleness check races progress**

If a delayed worker publishes progress after this query selects its import but before the unconditional failure write, the refreshed import is still marked `Failed`; the same read-check-write race in `retry_import` can reset or re-enqueue work while the original worker continues, causing conflicting statuses or concurrent parse jobs.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "feat(imports): recover an import whose b..." | Re-trigger Greptile}

[greptile-apps]

Generation stage is lost

When the scheduler marks a stale Generating Wiki import as Failed, this branch no longer knows which stage failed and returns the import to Review instead of Graphed, causing generate_wiki to reject the retry until the tree is manually approved again.

Knowledge Base Used: Document processing pipeline

Prompt To Fix With AI

This is a comment left during a code review.
Path: wikify/api/imports.py
Line: 125-127

Comment:
**Generation stage is lost**

When the scheduler marks a stale `Generating Wiki` import as `Failed`, this branch no longer knows which stage failed and returns the import to `Review` instead of `Graphed`, causing `generate_wiki` to reject the retry until the tree is manually approved again.

**Knowledge Base Used:** [Document processing pipeline](https://app.greptile.com/bwh-tech/-/custom-context/knowledge-base/bwhtech/wikify/-/docs/document-processing-pipeline.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Partial parses become review-ready

When parsing dies after storing source_document but before remediation, tree rebuilding, or classification finishes, this check skips parse enqueueing and moves the import directly to Review, exposing an incomplete or empty section tree as ready for review.

Knowledge Base Used:

• Import and source parsing
• Document processing pipeline

Prompt To Fix With AI

This is a comment left during a code review.
Path: wikify/api/imports.py
Line: 112-127

Comment:
**Partial parses become review-ready**

When parsing dies after storing `source_document` but before remediation, tree rebuilding, or classification finishes, this check skips parse enqueueing and moves the import directly to `Review`, exposing an incomplete or empty section tree as ready for review.

**Knowledge Base Used:**
- [Import and source parsing](https://app.greptile.com/bwh-tech/-/custom-context/knowledge-base/bwhtech/wikify/-/docs/import-and-parsing.md)
- [Document processing pipeline](https://app.greptile.com/bwh-tech/-/custom-context/knowledge-base/bwhtech/wikify/-/docs/document-processing-pipeline.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Staleness check races progress

If a delayed worker publishes progress after this query selects its import but before the unconditional failure write, the refreshed import is still marked Failed; the same read-check-write race in retry_import can reset or re-enqueue work while the original worker continues, causing conflicting statuses or concurrent parse jobs.

Knowledge Base Used: Backend API and realtime services

Prompt To Fix With AI

This is a comment left during a code review.
Path: wikify/tasks.py
Line: 37-47

Comment:
**Staleness check races progress**

If a delayed worker publishes progress after this query selects its import but before the unconditional failure write, the refreshed import is still marked `Failed`; the same read-check-write race in `retry_import` can reset or re-enqueue work while the original worker continues, causing conflicting statuses or concurrent parse jobs.

**Knowledge Base Used:** [Backend API and realtime services](https://app.greptile.com/bwh-tech/-/custom-context/knowledge-base/bwhtech/wikify/-/docs/backend-api-and-realtime-services.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.