t5-small performance :( tf-idf+e5-base

[yiftachashkenazi]

T5-Small Final Test Results

Test Date: July 28, 2025
Status: FAILED - Not suitable for production RAG

Summary

T5-small was tested for RAG performance and showed excellent speed characteristics but critical quality issues that make it unsuitable for production use.

Performance Results ✅

Metric	T5-small	Production MiniLM	Result
Search Throughput	177.48 ops/sec	175.49 ops/sec	+1.1% faster ✅
Insertion Throughput	68.18 ops/sec	74.28 ops/sec	-8.2% slower
Memory Usage	0.03 MB	0.06 MB	50% more efficient ✅
Embedding Speed	100.63ms	65.13ms	54% slower
Retrieval Speed	13.46ms	77.13ms	473% faster ✅

Quality Results ❌ CRITICAL ISSUES

Metric	T5-small	Production MiniLM Baseline	Result
Relevance Score	2.17/5	3.18/5	-32% worse ❌
Success Rate	0%	100%	Complete failure ❌
Similarity Score	0.045	High quality	Extremely poor ❌
AI Assessment	N/A	4.9/5	Not tested

Key Findings

✅ Strengths

• Fastest search throughput (177.48 ops/sec)
• Fastest retrieval time (13.46ms)
• Most memory efficient (0.03MB)
• Higher dimensional embeddings (512d vs 384d)

❌ Critical Issues

• Zero successful retrievals (0% success rate)
• Poor semantic understanding (2.17/5 relevance)
• Extremely low similarity scores (0.045)
• Unsuitable for user-facing RAG applications

Final Recommendation

❌ T5-small is NOT RECOMMENDED for production RAG due to critical quality issues that outweigh performance benefits.

Alternative Recommendation

✅ E5-base emerges as best upgrade option:

• Superior quality: 3.69/5 relevance with 80% success rate
• Excellent semantic understanding: 0.802 similarity scores
• Clear improvement over current baseline

---

Test Data Sources:

• Performance: performance_only_extended_20250728_193643.json
• Quality: extended_rag_quality_working_20250728_190251.json
• Baseline: configurable_rag_quality_report_configurable_rag_quality_20250728_101901.json

[yiftachashkenazi]

@Goldziher, so we will have data of what we tried.
anyway keeping the same model as now.