Skip to content

Add Gemini reranking for video search results#79

Merged
ssrajadh merged 4 commits into
ssrajadh:masterfrom
Binary67:gemini-rerank
Jul 4, 2026
Merged

Add Gemini reranking for video search results#79
ssrajadh merged 4 commits into
ssrajadh:masterfrom
Binary67:gemini-rerank

Conversation

@Binary67

@Binary67 Binary67 commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add optional --rerank support for search to re-score embedding candidates with Gemini 2.5 Flash.
  • Extract candidate clips, request structured Gemini JSON scores, and sort visual matches ahead of embedding-only results.
  • Use rerank match/confidence in low-confidence prompts while falling back to embedding scores when rerank output is missing or invalid.
  • Document the new search rerank behavior and add unit coverage for CLI, reranker parsing/sorting, and Gemini reranker integration.

Tests

  • uv run pytest tests/test_cli.py tests/test_reranker.py tests/test_gemini_reranker.py
  • Result: 88 passed

Related Issue

Relates to #71

@ssrajadh

ssrajadh commented Jul 3, 2026

Copy link
Copy Markdown
Owner

Looks good for the most part, some issues before we can merge:

1. A single trim/API failure aborts the whole search and drops all results. rerank_results runs before _cache_last_search, so if trim_clip raises for one candidate (source file moved, ffmpeg missing, a bad window) or the Gemini call exhausts _retry / hits GeminiQuotaError, the exception propagates to _handle_error and exits 1. The user loses the full embedding ranking they'd have gotten without --rerank. The graceful fallback advertised in the README only covers unparseable JSON (returns None, keeps original rank), the more common trim/API failures aren't handled. Could we wrap each candidate's trim+score in a per-candidate try/except that falls back to score=None, so the feature degrades consistently instead of nuking the search?

2. --rerank requires GEMINI_API_KEY regardless of index backend. GeminiReranker() is constructed unconditionally, so a local (or qwen-cloud) index with no Gemini key raises GeminiAPIKeyError and exits with no results. Same root cause as (1), the rerank pass isn't isolated from the primary search path. Worth at least a clear upfront message, or ideally the same graceful fallback.

3. Minor efficiency: each candidate is trimmed once for scoring and then the winner is re-trimmed by trim_top_results at save time, and the Gemini uploads run sequentially. With --rerank --results 10 that's ~10 ffmpeg extractions + 10 sequential uploads before any output. Users opt in, so this is fine for now, but the redundant second trim of the top clip is avoidable.

The sort logic and the has_rerank_score gate look correct, since the sort guarantees results[0] is the best-bucket item, keying the low-confidence check off it is sound. Nothing here is a correctness bug; (1) is the main one I'd want addressed.

@Binary67

Binary67 commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

Hi, thanks for the comments. I pushed two follow-up improvements to the rerank flow:

  • Made --rerank fail gracefully: if Gemini reranking cannot initialize, or if an individual candidate cannot be trimmed/scored, search now falls back to the embedding-ranked results instead of failing the whole command. I also updated the README wording and added tests for the missing-key and candidate-failure paths.

  • Reused the temporary clips created during reranking when saving/trimming final results, avoiding duplicate clip extraction. Internal rerank-only fields are stripped before caching search results, and tests were added for clip reuse plus fallback behavior when the temporary clip is unavailable.

Also rebased the branch on the latest upstream master.

@ssrajadh
ssrajadh merged commit 8e94ac1 into ssrajadh:master Jul 4, 2026
9 checks passed
@Binary67
Binary67 deleted the gemini-rerank branch July 7, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants