IMAP Thread Dedupe

[prvnsmpth]

Cross-source IMAP thread dedup

Problem

Two IMAP accounts subscribed to the same mailing list produce separate documents rows because today's external_id bakes source_id in. Search returns near-duplicates with different scores - bad UX.

Proposal

Connector (connectors/imap/src/models.rs:164): drop source_id from make_thread_document_id.

pub fn make_thread_document_id(folder: &str, thread_id: &str) -> String {
    format!("imap-thread:{}:{}", urlenc(folder), encode(thread_id))
}

Folder stays (keeps Gmail's INBOX vs [Gmail]/All Mail distinct). Source goes — same thread across two accounts now produces the same external_id but distinct rows (different source_id). The (source_id, external_id) UNIQUE constraint handles the rest. No schema change.

Searcher (services/searcher/src/search.rs): replace deduplicate_cross_source with:

1. Over-fetch from BM25 (3 × limit, capped).
2. Apply existing permission filter.
3. Group by external_id - we will have to decide for which rows we do this, can't do for all or we risk losing legitimate results
4. Keep highest score per group (tiebreaker: updated_at). Attribution list via also_in: [{source_id, document_id, score}, …]. Don't mutate permissions - kept row is permitted by step 2.
5. Trim to limit.

Out of scope

Content-blob dedup (byte-identical, simhash) and cross-connector entity dedup are separate concerns.

Caveats

• Different folder names per account → no collapse. I think this is acceptable, since we're mostly looking to get rid of dupes from mailing list in the inbox. But we can consider alternatives as well if needed.
• Existing rows keep old external_ids; collapse on next full resync.

Related PR: #149

[prvnsmpth]

@ramteid Please let me know if this will solve the issue, or if I'm missing something.

[ramteid]

I agree — external_id is the right approach here. You're correct that content_fingerprint doesn't generalize. After reviewing the implementation, the fingerprint is actually SHA256("imap-thread:" + thread_id) — it's an identity hash, not a content hash. It keys on the exact same data (thread_id) that external_id would use, just with extra indirection: a new column, a migration, an index, and changes across 15 files vs ~4.

A few things worth noting from my code review:

Your proposal is sound as-is. Keeping the folder in the external_id is the right call — removing it would cause UNIQUE(source_id, external_id) violations when the same thread appears in both INBOX and Archive within a single source (the IMAP connector syncs folders independently, so this produces separate thread documents). The folder-name-mismatch caveat is acceptable for the mailing list case.

The content_type gate for step 3 matters. You noted "we will have to decide for which rows we do this." The current branch gates on content_type == "email_thread" AND attributes.thread_id. I'd suggest gating on external_id prefix instead — external_id.starts_with("imap-thread:") — since that's the field you're collapsing on and it's self-documenting. This avoids coupling to attributes and makes the collapse criterion match the collapse key.

Skip permission mutation entirely. The current branch merges permissions from all duplicates into the winner. This is dead code — both fulltext (search_repository.rs:658) and semantic (embedding.rs:190) paths apply permission filtering as a SQL WHERE clause before results reach the dedup function. The user only sees docs they're authorized for. The merged permissions are written to the response JSON but nothing reads them (frontend doesn't use document.permissions from search results, admin scope intentionally omits user_email to see everything).

The embedding cloning question remains. Removing source_id from external_id gives you search-time dedup, but the second account still generates duplicate embeddings (vector space pollution, double API cost). The AI service already selects external_id from documents — find_embedded_duplicate could simply query WHERE external_id = $1 AND id != $2 AND embedding_status = 'completed' using the existing idx_documents_external_id index. No new column needed. This is ~2 lines changed in documents.py and batch_processor.py.

One race condition to accept: if two sources sync simultaneously, both workers may check find_embedded_duplicate before either completes → both generate embeddings independently. Survivable (identical vectors, no corruption), and the window is narrow.

Regarding the Rust find_embedded_duplicate in shared — it's dead code on the current branch (zero callers). Can be dropped.

Suggested implementation scope:

File	Change
models.rs	Remove source_id from make_thread_document_id signature + format
sync.rs	Update call sites (3 places)
search.rs	Collapse by external_id prefix, over-fetch, no permission merging
documents.py	find_embedded_duplicate queries by external_id
batch_processor.py	Use doc.external_id instead of doc.content_fingerprint

That's 5 files, no migration, no schema change, no SDK change.