Cluster embeddings with DBSCAN and collapse duplicates

[ionfwsrijan]

Description

Use DBSCAN clustering on the embeddings to group near-duplicate findings and collapse each cluster into a single representative finding with metadata about the duplicates.

What to implement

Create backend/app/ml/deduplicator.py:

• deduplicate(findings: list[dict], epsilon: float = 0.15) -> list[dict]
• Embeds findings via embedder.py
• Runs sklearn.cluster.DBSCAN(eps=epsilon, min_samples=2, metric='cosine')
• For each cluster, keep the finding with the highest raw_severity as representative
• Attach duplicate_count: int and related_files: list[str] to representative
• Noise points (label == -1) are returned as-is with duplicate_count: 0

Acceptance criteria

• A set of 10 identical SQL injection findings across different files collapses to 1
• duplicate_count accurately reflects the cluster size
• related_files lists all files in the cluster except the representative's own file

[Trishanthsai]

Hi @ionfwsrijan i would like to work on this issue if possible could u assign it to me

[ionfwsrijan]

Hey @Trishanthsai !

You've been assigned to this issue.

Before you start, a couple of quick things:

⭐ Star the repo — it helps the project grow and supports the roadmap
🔀 Fork it — work on your fork and open a PR from there when ready

Feel free to drop a comment here if you run into anything or have questions while implementing. Happy building!

[Trishanthsai]

Hi! @ionfwsrijan Quick clarification before I start implementing this issue.

The acceptance criteria mention selecting the representative finding using the highest raw_severity, but the current Finding model only contains severity and doesn't persist raw_severity.

Should the deduplicator:

1. Use the normalized severity field already present in Finding, or
2. Introduce/store a separate raw_severity value?

Just wanted to confirm the intended behavior before implementation. Thanks!

[ionfwsrijan]

@Trishanthsai it means the same

[Trishanthsai]

Hi @ionfwsrijan I pulled the latest main and don't see backend/app/ml/ or embedder.py in the repository yet.

For this issue, should I:

1. Create both app/ml/embedder.py and app/ml/deduplicator.py, or
2. Is there another issue/PR that introduces embedder.py first?

Just want to make sure I'm building on the intended structure.

[ionfwsrijan]

@Trishanthsai There is already an issue made for it

[Trishanthsai]

Thanks for the clarification @ionfwsrijan Since #84 depends on the embedder from #83 and #83 is currently assigned, I'll wait for that to be merged before starting implementation on this issue.

[ionfwsrijan]

@Trishanthsai Sure. Thanks for understanding