A production-grade RAG system for oil market analysis. Combines real-time EIA inventory data, WTI/Brent price feeds, and financial news with a fine-tuned Llama 3.2 model to generate analyst-grade investment memos, market briefs, and deal risk assessments.
Built as a self-directed research project — not affiliated with any institution.
Live market indicators (WTI-Brent spread, returns, volatility, inventory) above a 5-tab interface.
Generates structured 5-section investment memos with cited sources.
Every answer shows the underlying documents used, with similarity scores and links.
Scenario-based risk assessment across 5 dimensions: force majeure, sanctions, chokepoints, refinery outages, OPEC supply.
Automated eval harness scoring retrieval precision, numeric accuracy, source-groundedness, and hallucination resistance.
| Mode | Description |
|---|---|
| News Brief | Sentiment-tagged summary of recent oil market news (BULLISH / BEARISH / NEUTRAL) |
| Data Q&A | Question answering grounded in EIA inventory data, price history, and news |
| Memo Generator | 5-section institutional investment memos (Executive Summary, Market Conditions, Drivers, Outlook, Risks) |
| Deal Risk | Scenario-based risk assessment for trades, monitors 5 risk categories continuously |
| Evaluation | Automated test suite measuring system quality across 4 dimensions |
┌──────────────────────────────────────────────────────────────┐
│ Data Pipeline (day1) │
│ EIA API ──┐ │
│ NewsAPI ──┼──→ Scrape (full text) ──→ corpus.json │
│ Y!Finance─┘ │
└──────────────────────────────────────────────────────────────┘
↓
┌──────────────────────────────────────────────────────────────┐
│ Embedding & Index (day2) │
│ corpus.json ──→ all-MiniLM-L6-v2 ──→ FAISS index │
└──────────────────────────────────────────────────────────────┘
↓
┌──────────────────────────────────────────────────────────────┐
│ Fine-Tuning (Colab, T4 GPU) │
│ Llama 3.2 3B + QLoRA + synthetic analyst-style outputs │
│ ──→ LoRA adapter (~50MB) │
└──────────────────────────────────────────────────────────────┘
↓
┌──────────────────────────────────────────────────────────────┐
│ Inference (app.py) │
│ Query → FAISS retrieval → fine-tuned LLM → cited answer │
│ ↓ │
│ Indicators · Risk scan · Evaluation harness │
└──────────────────────────────────────────────────────────────┘
- LLM: Llama 3.2 3B Instruct, QLoRA fine-tuned (Groq API fallback)
- Embeddings: sentence-transformers
all-MiniLM-L6-v2 - Vector DB: FAISS (IndexFlatIP, cosine similarity)
- Fine-tuning: Hugging Face
transformers,peft,trl,bitsandbytes - UI: Streamlit with custom CSS
- Data: EIA Open Data API, NewsAPI, Yahoo Finance
# 1. Install
pip install -r requirements.txt
# 2. Set API keys (free tiers)
export EIA_API_KEY="your_key" # eia.gov/opendata
export NEWS_API_KEY="your_key" # newsapi.org
export GROQ_API_KEY="your_key" # console.groq.com
# 3. Build the data pipeline
python day1_data_pipeline.py # pulls prices, EIA, news
python scraper.py # enriches with full article text
python day2_rag.py # builds FAISS index
# 4. (Optional) Fine-tune the LLM
# Upload finetune.ipynb to Google Colab, run top-to-bottom on T4 GPU.
# Download lora_adapter/ folder back to this directory.
# 5. Launch
streamlit run app.pyoil-intelligence/
├── day1_data_pipeline.py # Pulls prices, EIA, news → corpus.json
├── scraper.py # Full article text extraction
├── day2_rag.py # FAISS index + LLM wrapper + source-tracking
├── indicators.py # WTI-Brent spread, returns, volatility, etc
├── risk_and_eval.py # Risk signal detection + RAG eval harness
├── app.py # Streamlit UI (5 tabs)
├── finetune.ipynb # QLoRA fine-tuning on Colab
├── data/ # Generated by pipeline
│ ├── oil_prices.csv
│ ├── eia_inventories.csv
│ ├── oil_news.json
│ └── corpus.json
├── index/ # FAISS vector index
└── lora_adapter/ # Fine-tuned LoRA weights (from Colab)
Run from the Evaluation tab. Current scores against the 6-question test set:
| Metric | Score |
|---|---|
| Retrieval Precision | Run the harness |
| Keyword Coverage | Run the harness |
| Numeric Accuracy | Run the harness |
| Hallucination Resistance | Run the harness |
The eval set covers 4 dimensions:
- Retrieval precision — did the right source types come back?
- Numeric grounding — when asked for numbers, does the answer contain them?
- Source groundedness — does the answer use retrieved content?
- Hallucination resistance — does the system refuse impossible queries?
Why QLoRA over full fine-tuning? 4-bit quantization + low-rank adapters fit Llama 3.2 3B training on a free Colab T4. Full fine-tuning would need an A100 (40GB+) or multi-GPU setup. QLoRA achieves ~95% of full fine-tuning quality at <5% of the cost.
Why FAISS over a managed vector DB (Pinecone, Weaviate)? For a corpus of ~500 documents, FAISS local index is faster, free, and removes a network dependency. The system fits entirely on a laptop. Trade-off is no built-in metadata filtering, which we handle in post-retrieval.
Why synthetic training data via Groq? Hand-labeling analyst-style outputs at scale isn't realistic for a side project. Self-instruct via a stronger teacher model (Groq's Llama 3.1 8B) generates plausible analyst-grade outputs that the smaller 3B student learns to replicate. Standard technique from Alpaca / Vicuna research.
Why hard-coded risk categories? Force majeure, sanctions, chokepoints, refinery outages, and OPEC dynamics cover the dominant risk dimensions in physical oil trading. Keyword-driven detection is interpretable and auditable in a way that "ask the LLM" isn't.
- News coverage is limited by NewsAPI free tier (100 req/day, ~1 month lookback)
- Several major paywalled sources (FT, WSJ, Bloomberg) are unscrapable
- Fine-tuning corpus is small (~200 examples) — quality improves linearly with more data
- Risk detection is keyword-based, not semantic — misses creative phrasings
- No real-time intraday data — daily granularity only
MIT