A production-oriented GenAI / Agentic RAG reliability investigation platform that combines structured operating data, historical incident retrieval, hybrid search, statistical tools, local Qwen3 reasoning, claim/citation verification, observability, and explicit human approval.
Status: Portfolio-ready, evaluated, GitHub CI validated, and live portfolio application deployed
Live application: Open the ReliabilityOps Hugging Face Space
Repository: unit-mole/reliabilityops-agentic-rag
Primary stack: Python · PyTorch · Qwen3 · BGE-M3 · BM25 · RRF · LangGraph · Qdrant · PostgreSQL · FastAPI · Gradio · Phoenix · Docker · Hugging Face Spaces
Default external paid LLM/API dependency: $0.
The validated local runtime uses local open-source models. Hardware, electricity, and internet access are not claimed to be free.
ReliabilityOps is intended for technical learning, experimentation, research, and portfolio demonstration.
- Generated hypotheses can be incomplete or incorrect and must be reviewed against the cited evidence.
- The confidence score is an evidence-ranking heuristic, not a universally calibrated probability.
- Corrective actions are recommendations until explicitly reviewed and approved by a human.
- Synthetic incidents and procedures are demonstration data derived from structured truth.
- The system is not validated for autonomous production, safety-critical, regulatory, or operational decision-making.
- Authentication/RBAC and enterprise hardening are future work.
Industrial reliability investigations rarely depend on one signal or one document. A useful investigation may require historical similarity, structured sensor conditions, technical procedures, statistical comparisons, competing hypotheses, contradiction handling, evidence sufficiency checks, traceable citations, and human review.
A naive RAG system that simply retrieves text and generates an answer is not enough for that workflow.
This project answers:
Given an equipment-failure investigation request and structured operating conditions, can an agentic RAG system retrieve relevant historical evidence, use bounded analytical tools, rank root-cause hypotheses, verify its claims, and present a traceable result for human review?
The system returns:
- Ranked root-cause hypotheses
- Confidence and support status
- Stable evidence/citation IDs
- Structured analytical findings
- Evidence-sufficiency decisions
- Verification status
- Recommended next checks
- Corrective-action recommendations
- Investigation timeline
- Human approve / reject / request-more-evidence state
Build a professional reliability-investigation system that can:
- Ingest and validate the UCI AI4I 2020 predictive-maintenance dataset.
- Generate deterministic synthetic incident narratives and technical procedures from canonical structured truth.
- Compare dense-only, section-aware, hybrid, and reranked retrieval strategies.
- Select the final retrieval architecture strictly from development evaluation.
- Combine vector retrieval, BM25, structured database tools, and statistical analysis.
- Use LangGraph to manage a bounded multi-step investigation workflow.
- Use local Qwen3 reasoning without mandatory paid external LLM/API calls.
- Detect evidence gaps and perform bounded retrieval retries.
- Rank root-cause hypotheses using retrieved and analytical evidence.
- Independently verify claims, citations, numeric support, and canonical root-cause evidence.
- Keep corrective actions behind a human approval boundary.
- Measure retrieval, RCA, grounding, agent behavior, latency, ablations, and failure modes.
- Expose the system through FastAPI and Gradio.
- Trace application activity through OpenTelemetry and Phoenix.
- Run a final untouched frozen test exactly once after development is complete.
- Publish a professional GitHub repository and free Hugging Face Static Space.
| Item | Implementation |
|---|---|
| Project name | reliabilityops-agentic-rag |
| Application | Evidence-grounded industrial root-cause investigation |
| Core reasoning model | Qwen3-14B locally |
| Faster local alternative | Qwen3-8B |
| Dense embeddings | BAAI/bge-m3 |
| Sparse retrieval | BM25 |
| Fusion | Reciprocal Rank Fusion |
| Selected retrieval runtime | v2_hybrid_dense_bm25_rrf |
| Agent orchestration | LangGraph |
| Structured DB | PostgreSQL with SQLite fallback |
| Vector DB | Qdrant |
| Verification | Citation, claim, canonical metadata, numeric support |
| Observability | OpenTelemetry + Phoenix |
| API / UI | FastAPI + Gradio |
| Deployment | GitHub + Hugging Face Static Space |
| Final evaluation | 200-case development split + untouched 100-case frozen test |
| Area | Technology |
|---|---|
| Language | Python, HTML, CSS, JavaScript |
| Deep learning | PyTorch |
| Reasoning model | Qwen3-14B / Qwen3-8B |
| Embeddings | BAAI/bge-m3 |
| Sparse retrieval | rank-bm25 |
| Fusion | Reciprocal Rank Fusion |
| Reranker ablation | BAAI/bge-reranker-v2-m3 |
| Vector database | Qdrant |
| Structured database | PostgreSQL / SQLite |
| Agent orchestration | LangGraph |
| Analytics | pandas, NumPy, SciPy / bounded Python analytics |
| API | FastAPI |
| Local UI | Gradio |
| Observability | OpenTelemetry + Arize Phoenix |
| Local LLM server | Ollama OpenAI-compatible endpoint |
| Infrastructure | Docker Compose |
| Testing | pytest |
| Code quality | Ruff |
| Automation | GitHub Actions |
| Public application | Hugging Face Static Spaces |
- Dense retrieval with
BAAI/bge-m3 - BM25 sparse retrieval
- Reciprocal Rank Fusion hybrid retrieval
- Section-aware chunking and metadata
- Qdrant vector retrieval
- PostgreSQL development mode and SQLite self-contained fallback
- Bounded SQL/database tools
- Statistical Python tools
- Local Qwen3 reasoning through an OpenAI-compatible endpoint
- LangGraph stateful orchestration
- Stable evidence and citation IDs
- Claim, citation, and numeric verification
- Bounded evidence-gap retry behavior
- Human approve/reject/request-more-evidence state
- Retrieval, RCA, grounding, agent, and latency evaluation
- V0-V6 ablation framework
- Failure-analysis taxonomy
- OpenTelemetry tracing with Phoenix
- FastAPI API
- Gradio UI
- Docker Compose
- pytest and Ruff quality gates
- Zero mandatory paid external LLM/API dependency in the default configuration
BAAI/bge-reranker-v2-m3 is implemented and evaluated in V3 as an ablation. It was not selected for the final V6 runtime because development evaluation showed worse ranking quality and substantially higher retrieval latency than the V2 hybrid retriever.
A useful reliability investigation cannot be reduced to "upload a PDF, embed it, and ask a question."
A root-cause investigation may require historical similarity, structured sensor conditions, statistical comparison, troubleshooting procedures, competing hypotheses, contradiction handling, citations, evidence sufficiency checks, verification, and an explicit decision boundary between an AI recommendation and a human-approved action.
ReliabilityOps models that workflow end to end.
The structured source is the public UCI AI4I 2020 Predictive Maintenance Dataset. It contains 10,000 synthetic industrial-machine instances and failure indicators.
ReliabilityOps creates deterministic synthetic incident narratives and technical procedures from structured truth. Synthetic content is explicitly marked.
| Property | Value |
|---|---|
| Validated AI4I source rows | 10,000 |
| Unique source UIDs | 10,000 |
| Machine-failure rows | 339 |
| Synthetic incident cases | 2,000 |
| Synthetic technical procedures | 6 |
| V0 naive/source-level chunks | 2,006 |
| V1+ section-aware chunks | 14,030 |
| Development evaluation cases | 200 |
| Frozen test cases | 100 |
| Evaluation seed | 42 |
See DATA_SOURCES.md for source and licensing details.
UCI AI4I structured source
│
▼
Canonical truth validation
│
├──────────────► Structured PostgreSQL / SQLite data
│
▼
Deterministic synthetic incidents + procedures
│
▼
Section-aware document processing
│
├──────────────► BGE-M3 / Qdrant
└──────────────► BM25
│
▼
Reciprocal Rank Fusion
│
▼
Selected V2 hybrid evidence
│
▼
LangGraph investigation
│
┌────────────────────┼────────────────────┐
▼ ▼ ▼
Retrieval tools Database tools Statistical tools
└────────────────────┼────────────────────┘
▼
Evidence bundle
│
▼
Local Qwen3
│
▼
Ranked root-cause hypotheses
│
▼
Citation / claim / numeric verifier
│
┌──────────┴──────────┐
▼ ▼
Evidence sufficient Evidence gap
│ │
│ bounded retry
└──────────┬──────────┘
▼
Structured RCA report
│
┌────────────────────┼────────────────────┐
▼ ▼ ▼
FastAPI Gradio Human review
│
Approve / Reject /
Request more evidence
flowchart TD
A[UCI AI4I] --> B[Canonical truth]
B --> C[Synthetic incidents]
B --> DB[Structured DB]
C --> CH[Section-aware chunks]
P[Synthetic procedures] --> CH
CH --> D[BGE-M3 / Qdrant]
CH --> S[BM25]
D --> H[RRF Hybrid]
S --> H
Q[User investigation] --> G[LangGraph investigation]
G --> H
G --> DB
G --> PY[Bounded Python analytics]
H --> E[Evidence bundle]
DB --> E
PY --> E
E --> L[Local Qwen3 synthesis]
L --> V[Citation / claim / numeric verifier]
V -->|bounded retry when needed| H
V --> O[Structured report]
O --> HITL[Human approval]
O --> API[FastAPI]
O --> UI[Gradio]
H -. V3 ablation only .-> R[BGE reranker]
v2_hybrid_dense_bm25_rrf
The V3 BGE reranker remains in the repository as an evaluated ablation, not as the selected final runtime.
| Version | Purpose | Development outcome |
|---|---|---|
| V0 | Dense retrieval baseline | Baseline |
| V1 | Section-aware chunks + metadata | Large retrieval improvement |
| V2 | Dense + BM25 + RRF | Selected retrieval architecture |
| V3 | Hybrid + BGE reranker | Rejected as final runtime; retained as ablation |
| V4 | Tool-augmented investigation | Strong tool/citation baseline |
| V5 | Stateful agentic investigation | Bounded evidence-gap loop |
| V6 | Agentic investigation + verifier | Final candidate |
| Version | Hit@10 | Hit@5 | MRR | nDCG@10 | Recall@10 | Mean latency |
|---|---|---|---|---|---|---|
| V0 | 9.5% | 6.0% | 0.0462 | 0.0353 | 4.75% | 28.77 ms |
| V1 | 41.5% | 37.0% | 0.3040 | 0.2025 | 20.75% | 28.02 ms |
| V2 | 97.5% | 90.5% | 0.7203 | 0.5300 | 59.75% | 61.25 ms |
| V3 | 97.5% | 92.0% | 0.6564 | 0.4802 | 56.0% | 152.58 ms |
V2 was selected because V3 did not improve Hit@10, reduced MRR/nDCG/Recall@10, and increased mean retrieval latency from approximately 61 ms to approximately 153 ms.
| Version | RCA Top-1 | RCA Top-3 | Hypothesis MRR | Citation completeness | Verification |
|---|---|---|---|---|---|
| V4 | 78.5% | 94.5% | 0.8600 | 100% | Not a V4 feature |
| V5 | 91.5% | 100% | 0.9567 | 100% | Not a V5 feature |
| V6 | 93.0% | 100% | 0.9642 | 100% | 100% |
Development metrics are retained in reports/experiments/ and reports/ablations/.
The final V6 candidate was evaluated once on the untouched frozen test split after development was complete.
| Metric | Frozen test |
|---|---|
| Cases | 100 |
| RCA Top-1 | 93.0% |
| RCA Top-3 | 100.0% |
| Hypothesis MRR | 0.9633 |
| Citation completeness | 100.0% |
| Citation precision | 100.0% |
| Verification pass rate | 100.0% |
| Unsupported primary hypothesis rate | 0.0% |
| Task completion | 100.0% |
| Tool-selection accuracy | 100.0% |
| Mean latency | 12.88 s |
| P50 latency | 11.96 s |
| P95 latency | 20.20 s |
Additional frozen-test agent behavior:
- Evidence-gap detection rate: 28.0%
- Evidence sufficient at final decision: 80.0%
- Agent-loop rate: 28.0%
- Mean agent iterations: 1.49
- Mean retrieval retries: 0.49
- Mean tool calls: 7.98
- Verification repairs: 0.0
Source of record:
reports/experiments/v6_test.json
The frozen test is treated as final and is not used for subsequent model, retrieval, prompt, verifier, or threshold tuning.
The evaluation framework measures:
- Hit@5 and Hit@10
- Precision@5 and Precision@10
- Recall@5 and Recall@10
- Mean Reciprocal Rank
- nDCG@10
- RCA Top-1
- RCA Top-3
- Hypothesis MRR
- Citation precision
- Citation completeness
- Unsupported primary hypothesis rate
- Task completion
- Tool-selection accuracy
- Evidence sufficiency
- Evidence-gap detection
- Agent-loop rate
- Retrieval retries
- Verification pass
- Verification repairs
- Mean, P50, and P95 latency
- Stage-aware failure taxonomy
The experiment design keeps the 100-case test split frozen until development decisions are complete.
| Role | Default |
|---|---|
| Reasoning LLM | Qwen3-14B locally |
| Faster local alternative | Qwen3-8B |
| LLM server | Ollama OpenAI-compatible local endpoint |
| Dense embeddings | BAAI/bge-m3 |
| Sparse retrieval | rank-bm25 |
| Reranker ablation | BAAI/bge-reranker-v2-m3 |
| Vector DB | Qdrant |
| Agent orchestration | LangGraph |
| Structured DB | PostgreSQL; SQLite fallback |
| Observability | OpenTelemetry + self-hosted Phoenix |
| API | FastAPI |
| UI | Gradio |
torch is intentionally installed separately so compatible CUDA-enabled builds can be selected for the target GPU.
Final local validation was performed on:
| Component | Validated configuration |
|---|---|
| Operating system | Windows 11 |
| Python | 3.12.10 |
| GPU | NVIDIA GeForce RTX 5090 |
| NVIDIA driver | 610.60 |
| PyTorch | 2.12.1+cu132 |
| CUDA runtime | 13.2 |
| BF16 | Supported |
| Ollama | 0.33.1 |
| Local LLMs | Qwen3-14B and Qwen3-8B |
| Database | PostgreSQL 17 |
| Vector DB | Qdrant |
| Observability | Phoenix |
The final environment manifest is generated at:
reports/final/final_manifest.json
The final local validation confirmed:
- FastAPI startup and health endpoint
- Swagger/OpenAPI rendering
- Real investigation submission through the API
- Investigation retrieval by ID
- Human approval endpoint
- Gradio investigation workflow
- Evidence/citation rendering
- Verification status rendering
- Local Qwen3 execution
- Selected V2 hybrid retrieval strategy
- Phoenix OTLP trace export and trace visibility
- PostgreSQL/Qdrant/Phoenix Docker services
- Clean timeline encoding
ruff check .passingpytestpassing with 30 tests
uvicorn reliabilityops.api.main:app --app-dir src --host 127.0.0.1 --port 8000- Health:
http://127.0.0.1:8000/health - Swagger/OpenAPI:
http://127.0.0.1:8000/docs
python .\app\gradio_app.pyOpen:
http://127.0.0.1:7860
With Docker mode running:
http://127.0.0.1:6006
The public application is available at:
| Component | Validated local project | Public Static Space |
|---|---|---|
| Qwen3 inference | Live local inference | Precomputed validated output |
| BGE-M3 retrieval | Live | Not executed server-side |
| BM25 / RRF | Live | Demonstrated through saved validated results |
| LangGraph | Live | Workflow replay |
| Qdrant / PostgreSQL | Live | Not required |
| FastAPI / Gradio | Live local interfaces | Static HTML/CSS/JavaScript |
| Phoenix | Live local observability | Not required |
| Purpose | Engineering implementation and evaluation | Free public portfolio showcase |
The Static Space does not claim to perform live Qwen3 or RAG inference. This separation keeps the public demonstration technically honest while the GitHub repository preserves the full reproducible implementation.
The Hugging Face deployment is a static interactive portfolio demonstration that replays genuine precomputed outputs from the validated local V6 pipeline. The full engineering implementation runs locally with Qwen3, BGE-M3, hybrid retrieval, LangGraph, Qdrant/PostgreSQL, FastAPI, Gradio, verification, and Phoenix observability.
ReliabilityOps live portfolio interface showing the evidence-grounded RCA positioning, final frozen-test metrics, architecture, and deployment links.
Validated investigation example showing the ranked primary hypothesis, support status, verification result, investigation timeline, citations, and analytical findings.
The selected hybrid retrieval architecture, bounded LangGraph investigation workflow, independent verifier, and measured V0-V6 evolution.
Retrieved text is treated as untrusted data. Embedded prompt-injection instructions are marked and ignored by policy.
The model has:
- No shell tool
- No arbitrary file-edit tool
- No unrestricted network tool
- No arbitrary Python execution tool
- No unrestricted SQL executor
Security smoke checks and dedicated security tests passed during final local validation.
See SECURITY.md.
The AI may investigate and recommend actions. Approval is a separate recorded state exposed through API endpoints:
POST /investigations/{id}/approve
POST /investigations/{id}/reject
POST /investigations/{id}/request-more-evidence
Corrective actions remain recommendations until explicitly approved.
OpenTelemetry spans cover the investigation workflow. Phoenix receives OTLP traces when enabled.
Final local validation confirmed successful trace export to:
http://localhost:6006/v1/traces
and recent traces were visible in the Phoenix UI.
The stage-aware failure taxonomy separates retrieval misses, RCA errors, overreach, and model-fallback resilience events.
For V6 development evaluation:
| Item | Count |
|---|---|
| Evaluated rows | 200 |
| Root-cause Top-1 errors | 14 |
| Model-fallback events | 1 |
| Total V6 issue rows | 15 |
Agent loops and retries are treated as bounded control-flow behavior rather than failures.
Detailed analysis:
reports/failure_analysis/summary.json
Important artifacts after a completed run:
reports/baseline/data_validation.json
reports/baseline/synthetic_validation.json
reports/baseline/index_build.json
reports/experiments/v0_dev.json
reports/experiments/v1_dev.json
reports/experiments/v2_dev.json
reports/experiments/v3_dev.json
reports/experiments/v4_dev.json
reports/experiments/v5_dev.json
reports/experiments/v6_dev.json
reports/experiments/v6_test.json
reports/experiments/v6_test.md
reports/ablations/ablation_results.json
reports/ablations/ablation_results.csv
reports/ablations/ablation_results.md
reports/failure_analysis/summary.json
reports/failure_analysis/failed_cases.json
reports/failure_analysis/v6_failed_cases.json
reports/final/final_manifest.json
reports/final/final_report.md
Demo-safe exports are generated under:
data/demo/
reliabilityops-agentic-rag/
├── .github/
│ └── workflows/
├── app/
├── assets/
│ ├── architecture/
│ └── screenshots/
├── configs/
├── data/
├── docs/
├── notebooks/
├── reports/
├── scripts/
├── src/
│ └── reliabilityops/
│ ├── agents/
│ ├── analytics/
│ ├── api/
│ ├── core/
│ ├── data/
│ ├── database/
│ ├── embeddings/
│ ├── evaluation/
│ ├── ingestion/
│ ├── llm/
│ ├── observability/
│ ├── reranking/
│ ├── retrieval/
│ ├── schemas/
│ ├── security/
│ ├── synthetic/
│ ├── tools/
│ └── verification/
├── tests/
├── .env.example
├── .gitignore
├── docker-compose.yml
├── LICENSE
├── pyproject.toml
├── README.md
├── RUNBOOK_WINDOWS.md
├── SECURITY.md
└── VALIDATION_REPORT.md
Read RUNBOOK_WINDOWS.md for the complete ordered run.
py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pippython -m pip install torch torchvision
python -m pip install -e ".[models,postgres,dev]"
Copy-Item .env.example .envollama pull qwen3:14b
ollama pull qwen3:8bdocker compose up -d
powershell -ExecutionPolicy Bypass -File .\scripts\configure_docker_mode.ps1powershell -ExecutionPolicy Bypass -File .\scripts\configure_embedded_mode.ps1The self-contained fallback is intended for software smoke tests. Portfolio performance claims are based on the validated full local-model configuration.
For a fresh development/evaluation run:
python -m scripts.check_environment
python -m scripts.download_ai4i
python -m scripts.generate_incidents --count 2000 --seed 42
python -m scripts.generate_procedures
python -m scripts.build_database
python -m scripts.process_documents
python -m scripts.build_index --strict-models
python -m scripts.generate_evaluation --dev 200 --test 100 --seed 42
python -m scripts.run_experiment --version v0 --split dev --strict-models
python -m scripts.run_experiment --version v1 --split dev --strict-models
python -m scripts.run_experiment --version v2 --split dev --strict-models
python -m scripts.run_experiment --version v3 --split dev --strict-models
python -m scripts.run_experiment --version v4 --split dev --strict-models
python -m scripts.run_experiment --version v5 --split dev --strict-models
python -m scripts.run_experiment --version v6 --split dev --strict-models
python -m scripts.run_ablation
python -m scripts.export_failure_analysis
python -m scripts.security_smoke
python -m scripts.smoke_investigation --strict-models
python -m scripts.export_demo_data
ruff check .
pytestAfter all development choices are complete, run the frozen final test once:
python -m scripts.run_experiment --version v6 --split test --strict-models
python -m scripts.export_final_resultsDo not tune the system using frozen-test results.
- Synthetic incident narratives and procedures are demonstration data derived from structured truth.
- The verifier is deterministic and evidence-based; it is not a formal proof system.
- Confidence is an evidence-ranking heuristic, not a universally calibrated probability.
- The CPU hashing/no-op fallback path exists for software smoke testing only and must not be used for portfolio metrics.
- Industrial corrective actions require appropriate domain and safety review.
- Authentication/RBAC and enterprise hardening are future work.
- The validated deployment is local rather than production cloud infrastructure.
- The frozen test is deliberately bounded and synthetic; external industrial validation is required before operational deployment.
- The public Hugging Face Static Space replays validated precomputed results rather than executing the full Qwen3/RAG stack live.
Potential extensions include:
- NASA C-MAPSS
- richer time-series tools
- GraphRAG when evaluation justifies it
- multimodal technical manuals and VLMs
- LoRA/domain adaptation when retrieval/prompting proves insufficient
- optional frontier commercial-model benchmarking
- streaming
- authentication/RBAC
- persistent enterprise investigation history
- production cloud deployment
- MCP interfaces
- live hosted inference when suitable free GPU capacity is available
- Generative AI
- Retrieval-Augmented Generation
- Agentic AI
- Hybrid retrieval
- Dense embeddings
- BM25
- Reciprocal Rank Fusion
- Vector databases
- Qdrant
- PostgreSQL
- LangGraph
- Qwen3 local inference
- Evidence-grounded generation
- Citation verification
- Numeric verification
- Root-cause analysis
- Statistical analysis
- Bounded tool use
- Agent loop and stopping logic
- Failure analysis
- Retrieval evaluation
- RCA evaluation
- Latency benchmarking
- Frozen-test methodology
- FastAPI
- Gradio
- Docker
- OpenTelemetry
- Phoenix observability
- Security testing
- pytest
- Ruff
- GitHub Actions
- Hugging Face Static Spaces
- Portfolio-focused AI engineering
Performance claims in this README are taken only from locally generated evaluation artifacts.
The V6 frozen test was run once after development was complete and is treated as final. It is not used for subsequent model, retrieval, prompt, verifier, or threshold tuning.
The public Static Space is explicitly disclosed as a precomputed demonstration and does not claim to reproduce the full local runtime.
One-line description: Evidence-grounded agentic RAG platform for industrial root-cause investigation using hybrid retrieval, LangGraph, local Qwen3 reasoning, analytical tools, independent verification, human approval, and measured V0-V6 evaluation.
Pinned repository description: Production-oriented Agentic RAG project with BGE-M3 + BM25/RRF hybrid retrieval, LangGraph investigation workflows, Qwen3, Qdrant/PostgreSQL, evidence verification, FastAPI, Gradio, Phoenix observability, 93% frozen-test RCA Top-1, and a live Hugging Face portfolio demo.
Project code: MIT.
External datasets and models retain their own licenses. See DATA_SOURCES.md and CITATIONS.md.
Anmol Tripathi
Quality Data Scientist building production-oriented portfolio projects in Data Science, Machine Learning, Applied AI, Generative AI, Agentic RAG, Natural Language Processing, Analytics Engineering, and Quality Analytics.


