fix: Add unique IDs for RRF deduplication in hybrid retrieval#644
Open
ananyasingh7052258502-coder wants to merge 1 commit into
Open
fix: Add unique IDs for RRF deduplication in hybrid retrieval#644ananyasingh7052258502-coder wants to merge 1 commit into
ananyasingh7052258502-coder wants to merge 1 commit into
Conversation
Owner
|
mergeee conflict |
Contributor
Author
|
@param20h Sir, mobile app se merge conflict resolve nahi ho raha. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📋 PR Checklist
🔗 Related Issue
Closes #634
📝 What does this PR do?
Adds unique chunk IDs in hybrid retrieval to fix RRF deduplication.
Problem:
RRF was failing to deduplicate chunks because BM25 retriever wasn't assigning unique IDs to chunks. Vector retriever had IDs but BM25 chunks had none, so duplicates from both retrievers weren't merged.
Solution:
chunk['id']generation inCustomBM25Retriever-retriever.pychunk['id']generation in_query_single_index-bm25.pybm25_{document_id}_{page}_{index}Now RRF can properly identify and merge duplicate chunks from Vector + BM25.
🗂️ Type of Change
🧪 How was this tested?
uvicorn app.main:app --reload)npm run devinsidefrontend/)📸 Screenshots (if UI change)
N/A - Simple bug fix. Added unique ID generation for BM25 chunks to enable RRF deduplication. No breaking changes.
✅ Self-Review Checklist
dev, notmainmainbranch or any HuggingFace deployment config