Skip to content

Refactor: split process_attachments.py into focused modules #74

Open

Description

@splaice

Context

`src/maildb/ingest/process_attachments.py` is ~750 lines and houses five distinct concerns:

  1. The extraction claim/process loop
  2. Embed-chunk pipeline
  3. Empty-extraction sweep + reembed
  4. Subprocess worker entrypoint
  5. The supervisor

Grew organically across PRs #54, #56, #58, #60. Now it's hard to hold the whole thing in head while reading any single function.

Proposal

Split into:

  • `process_attachments/init.py` — re-exports for back-compat
  • `process_attachments/claim.py` — `_claim_row`, `_load_attachment`, `_set_status`, `ensure_pending_rows`, `_reclaim_stale`, `_count_selected`
  • `process_attachments/embed.py` — `_embed_chunks`, `EmbedFailedError`, `_build_embedding_client`, `reembed_zero_vectors`, `sweep_empty_extractions`
  • `process_attachments/pipeline.py` — `process_one`, `_claim_and_process_loop`
  • `process_attachments/supervisor.py` — `_run_supervised_single_worker`, `_find_stuck_extracting`, `_killpg_quietly`, `_mp_context`
  • `process_attachments/run.py` — `run`, dispatch logic, `_subprocess_worker`

Each module < 200 lines, each tested in isolation. No behavior change.

Acceptance criteria

  • Module split per the layout above.
  • All existing tests pass without modification (back-compat exports).
  • No public API change to callers (CLI, MCP).

🤖 Generated with Claude Code

Activity

  1. added
    P4Low: someday/maybe
    refactorCode restructuring without behavior change
    on Apr 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P4Low: someday/mayberefactorCode restructuring without behavior change

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions