models scores

[yiftachashkenazi]

RAG Model Performance Report

Test Type: Configurable RAG Quality Benchmark
Benchmark ID: configurable_rag_quality_20250728_132135
Test Timestamp: 2025-07-28T16:21:35.103387
Total Configurations Tested: 3
Total Test Time: 18.37 minutes

Executive Summary

This report details the performance and quality of three different embedding models within a configurable RAG pipeline. The models tested were: production_minilm_2000, scibert_2000, and mpnet_2000. The benchmark focused on evaluating retrieval relevance, performance metrics, and overall RAG quality.

Key Findings:

• production_minilm_2000 excels in performance metrics, demonstrating the highest insertion and search throughput and the lowest memory usage. It also achieved a perfect success rate and a good overall performance score.
• mpnet_2000 and scibert_2000 (both 768d models) generally show higher average relevance scores compared to production_minilm_2000 (384d).
• However, the larger 768d models exhibit significantly slower retrieval times and higher memory consumption.
• mpnet_2000 achieved the highest average relevance score (3.28/5).
• All models achieved a 100% retrieval success rate.

Model Configurations

Configuration Name	Description	Chunking (max_chars)	Chunking (overlap)	Embedding Model Name	Embedding Dimension	Embedding Model Type	Embedding Size Category
production_minilm_2000	Production MiniLM model with large chunks (current production setup)	2000	200	sentence-transformers/all-MiniLM-L12-v2	384	sentence-transformers	small
scibert_2000	SciBERT model optimized for scientific content with large chunks	2000	200	allenai/scibert_scivocab_uncased	768	sentence-transformers	medium
mpnet_2000	MPNet model with large chunks for high-quality embeddings	2000	200	sentence-transformers/all-mpnet-base-v2	768	sentence-transformers	medium

Comprehensive Comparison

Metric	production_minilm_2000	scibert_2000	mpnet_2000
Model Name	all-MiniLM-L12-v2	scibert_scivocab_uncased	all-mpnet-base-v2
Dimension	384d	768d	768d
Insertion Throughput (ops/sec)	535.3	326.3	274.6
Search Throughput (ops/sec)	136.1	135.3	119.5
Memory Usage (MB)	0.09	-13.17	-15.62
Chunk Count	15	15	15
Avg Chunk Size (chars)	1994	1994	1994
Coverage Ratio	1.103	1.103	1.103
Avg Relevance Score (1-5)	3.05	3.18	3.28
Success Rate (%)	100.0%	100.0%	100.0%
Avg Retrieval Time (sec)	42.9	79.74	75.25
Total Docs Retrieved	126	130	129
Diversity Score	0.876	0.875	0.878
Query Quality Score	0.642	0.75	0.673
Query AI Score	4.85	4.55	4.7
Performance Score	0.878	0.745	0.748

Detailed Analysis

Performance Analysis

• Insertion Throughput: production_minilm_2000 is the clear leader, processing significantly more vectors per second.
• Search Throughput: production_minilm_2000 also demonstrates the highest search throughput, indicating faster query processing.
• Memory Usage: The production_minilm_2000 model is by far the most memory-efficient, requiring minimal memory compared to the 768d models, which surprisingly show negative memory usage (this might indicate a reporting anomaly or a specific way memory is calculated in the benchmark).

RAG Quality Metrics

• Average Relevance Score: mpnet_2000 scores the highest in average relevance, suggesting it is better at retrieving semantically similar documents. scibert_2000 follows closely, with production_minilm_2000 scoring lowest in this metric.
• Success Rate: All models achieved a perfect 100% success rate, meaning they successfully retrieved documents for all queries.
• Average Retrieval Time: The 768d models (scibert_2000 and mpnet_2000) have considerably longer retrieval times compared to production_minilm_2000.
• Total Documents Retrieved: scibert_2000 retrieved the most documents on average, though the difference is marginal.

Advanced Quality Metrics

• Diversity Score: All models show similar and high diversity scores, indicating good variety in the retrieved documents.
• Query Quality Score: scibert_2000 achieved the highest query quality score, suggesting its generated queries were more effective.
• Query AI Score: production_minilm_2000 scored the highest in Query AI Score, implying its generated queries were deemed most effective by the AI evaluator.

Key Insights

• For maximum performance (speed and memory efficiency), production_minilm_2000 is the superior choice. It offers the best throughput and the lowest memory footprint.
• For the highest relevance in retrieved content, mpnet_2000 is the best performer, closely followed by scibert_2000. These models, with their larger dimensionality, seem to capture semantic nuances more effectively.
• The trade-off for higher relevance with 768d models is a significant increase in retrieval time. Organizations must weigh the importance of relevance against latency requirements.
• The production_minilm_2000 model's lower relevance scores, despite its performance advantages, suggest that for applications where capturing very specific or nuanced information is critical, exploring larger dimensional models might be beneficial, provided the performance impact is acceptable.

Conclusion

The choice of model depends on specific application needs. If speed and resource efficiency are paramount, the existing production_minilm_2000 configuration is robust. However, if achieving the highest possible relevance is the primary goal, mpnet_2000 or scibert_2000 offer a clear advantage, albeit with a performance penalty. Further testing with different chunking strategies or a broader range of queries could provide additional insights.

[yiftachashkenazi]

@Goldziher this is after 5 test all the same.

[Goldziher]

so, basically the conclusion is that our current embeddings are optimal, yes?

[yiftachashkenazi]

Yes. Maybe it's better to say optimal so far. There's a good chance another model might yield better results. But I will create a model testing sub-unit (based on what you did), and we can test other models along the way.