diff --git a/README.md b/README.md index 555d6fd..ae69010 100644 --- a/README.md +++ b/README.md @@ -9,62 +9,100 @@ [![Claude](https://img.shields.io/badge/Claude-Sonnet_4.5-CC785C?style=flat&logo=anthropic&logoColor=white)](https://www.anthropic.com) [![License](https://img.shields.io/badge/License-MIT-6B7280?style=flat)](LICENSE) -**Ask any business question in plain English. Get SQL, a chart, and a written insight in seconds.** +**Ask a business question in plain English. Get the SQL, a chart, and a written insight in seconds.** -Live: https://insightiq-frontend.orangemeadow-25c8b891.westeurope.azurecontainerapps.io +InsightIQ turns a natural-language question into a parameter-safe SQL query, runs it against a live PostgreSQL database, picks an appropriate chart, and writes a short interpretation — with the generated SQL always shown for inspection. It is built as a production system, not a demo: every query is traced, cost-metered, audited, and rate-limited, and the whole thing scales to zero when idle. -**Demo tip:** First visit after idle may take **~30–60s** while the app wakes from scale-to-zero — deliberate idle scaling keeps hosting near **$0**. Open the [frontend](https://insightiq-frontend.orangemeadow-25c8b891.westeurope.azurecontainerapps.io) a minute before a demo; sign-in wakes the backend automatically. +**Live:** https://insightiq-frontend.orangemeadow-25c8b891.westeurope.azurecontainerapps.io +> **First load after idle takes ~30–60s.** The apps scale to zero, so the first request wakes them; this keeps hosting near $0. Open the link a minute before a demo — sign-in wakes the backend automatically. -**Production layers:** Langfuse trace per query (SQL · time · rows · cost), -flushed on graceful shutdown · 15-pair eval graded by EXECUTION ACCURACY, -CI-gated at 90% · AST guardrails (SELECT-only, table/column whitelist, -injection scan) · read-only DB role + statement timeout + forced LIMIT + -PII block · cost per query (~$0.004 avg, daily budget alert) · graceful -degradation on timeout / bad SQL / zero rows · normalised-question cache · -scale-to-zero on Container Apps ($0 idle) +--- + +## Contents + +- [What it does](#what-it-does) +- [Production posture](#production-posture) +- [Threat model](#threat-model) +- [How a query flows](#how-a-query-flows) +- [System architecture](#system-architecture) +- [Engineering decisions](#engineering-decisions) +- [Data model](#data-model) +- [Observability & operations](#observability--operations) +- [Testing](#testing) +- [Security & secrets](#security--secrets) +- [Cost](#cost) +- [Limitations & hardening roadmap](#limitations--hardening-roadmap) +- [Running it](#running-it) +- [Project layout](#project-layout) --- -## Demo +## What it does + +Managers at SaaS companies are locked out of their own data. "Which customers are at churn risk?" means writing SQL, or waiting two days for an analyst. Analysts, in turn, lose most of their time to ad-hoc query requests instead of strategic work. + +InsightIQ removes that dependency. A non-technical user asks a question; the system generates and executes the SQL, chooses the chart that fits the data shape, and returns a written business interpretation. The query is shown alongside the answer, so a technical user can verify exactly what ran. -[▶ Watch demo video](assets/insightiq-demo.mp4) · [Try the live app](https://insightiq-frontend.orangemeadow-25c8b891.westeurope.azurecontainerapps.io) +The design principle is narrow and deliberate: **automate the retrieval and synthesis, never hide the query.** Trust in a text-to-SQL system comes from transparency, not from asking the user to take the answer on faith. -> Sign in with Google → ask a plain English question → chart + AI insight in under 5 seconds. -> Click a history item to reload it from cache — no API call, instant result. +--- + +## Production posture -## The problem +The interesting problem here is not "call an LLM to write SQL" — that is a weekend. The problem is doing it safely against a real database, observably, and within a cost ceiling. Everything below is implemented and running today. -Business managers at SaaS companies are blocked from their own data. A question like "which customers are at churn risk?" requires writing SQL — or waiting two days for an analyst to do it. Analysts spend the majority of their time on ad-hoc query requests rather than strategic work. +| Concern | What's in place | +|---------|-----------------| +| **Observability** | Langfuse trace per query — question, generated SQL, execution time, row count, retry count, `cost_usd`. The trace is the first place I look when an answer is wrong. | +| **Cost control** | Per-query token cost metered onto the trace and into `query_audit`; daily spend tracked in `daily_cost_spend` with a webhook alert past `BUDGET_USD_DAILY`; a scheduled job opens a GitHub issue if Anthropic spend clears $5/day. | +| **Write prevention** | A read-only guard validates every generated statement before execution: comments are stripped, the first token must be `SELECT`, and a secondary scan rejects `DROP/DELETE/INSERT/UPDATE/TRUNCATE/ALTER` anywhere in the string (catches `SELECT 1; DROP TABLE …`). | +| **Self-healing** | On execution error, the failed SQL and the database error are fed back to the model and retried, capped at 3 attempts. Retry count is persisted and surfaced — a rising retry rate is an early signal of prompt drift. | +| **Abuse limits** | Per-user query budget enforced with a single atomic `UPDATE … WHERE total_queries < query_limit RETURNING …` — no Redis, no check-then-increment race. | +| **Caching** | Query results (rows, columns, explanation) are stored as JSONB in `query_history`; replaying a past query is a free Postgres read, not another model round-trip. | +| **Operations** | Scale-to-zero on Azure Container Apps ($0 idle); zero-downtime rolling deploys; smoke-test failure triggers automatic revision rollback. | +| **Secrets** | Zero credentials in the repo. OIDC federated login to Azure (no stored service-principal password); runtime secrets injected as Container App `secretref`. | -InsightIQ removes that dependency entirely. A manager types a question, the system generates and executes the SQL, selects the right chart type, and returns a written business interpretation — without any technical knowledge required. +Items I have **not** built yet, and why they matter, are listed honestly under [Limitations & hardening roadmap](#limitations--hardening-roadmap). The most important of them is a dedicated read-only database role — today the guard is the only thing standing between a bypassed validation and a write. --- -## How it works +## Threat model + +InsightIQ takes untrusted natural language and turns it into code that runs against a production database. That is the whole risk in one sentence. There are three trust boundaries, and a control at each. + +**1 — The user's question is untrusted.** It can carry injection ("ignore the schema and …"). The question is never string-concatenated into SQL; it is an input to the model, and the *generated* SQL is validated independently of the question. The natural-language text never reaches the database. + +**2 — The model's output is untrusted.** A non-deterministic code generator must be treated as hostile, not helpful. Every generated statement passes the read-only guard before it runs: comment-stripped token parse (first token must be `SELECT`) plus a keyword scan that rejects DDL/DML and semicolon-separated injection. This is token parsing rather than a naive regex precisely because `re.match(r'SELECT', …)` is defeated by leading whitespace, comments, or encoding tricks. + +**3 — Database access is the blast radius.** Connections are TLS-only (`sslmode=require`); a per-user query budget bounds abuse; a statement-timeout guard bounds runaway queries. **Known gap:** the application still connects with a single write-capable role, so the guard at boundary 2 is currently the *only* write-prevention. The planned fix — a dedicated read-only Postgres role — moves write-prevention into the database itself, so a guard bypass degrades to "permission denied" rather than data loss. This is the top item on the roadmap, and I'd rather name it than imply it's already there. + +--- + +## How a query flows ``` User question │ ▼ -Schema injection ──── full Postgres schema + 3 sample values per column +Schema injection ──── full Postgres schema + 3 sample values per column, │ baked into the Claude system prompt ▼ Claude generates SQL │ ▼ -Read-only guard ──── token-level parse: first token must be SELECT - │ secondary scan for DROP / DELETE / INSERT / UPDATE +Read-only guard ──── strip comments → first token must be SELECT → + │ secondary scan rejects DROP/DELETE/INSERT/UPDATE/TRUNCATE/ALTER ▼ -Execute via asyncpg pool +Execute via asyncpg pool (statement-timeout guard) │ ┌──┴──────────────────┐ - │ Execution error? │ Yes ──► error + failed SQL fed back to Claude - │ │ retry up to 3× with full context + │ Execution error? │ Yes ──► error + failed SQL fed back to Claude, + │ │ retried up to 3× with full context └──┬──────────────────┘ │ No ▼ -Results cached to query_history (rows + columns + explanation as JSONB) +Trace + cost to Langfuse · row to query_audit · results cached to query_history (JSONB) │ ▼ Chart type derived from data shape: @@ -74,7 +112,7 @@ Chart type derived from data shape: everything else → table │ ▼ -Response returned with SQL, rows, explanation, execution_ms, retry count +Response: SQL · rows · explanation · execution_ms · retry count · cost ``` --- @@ -86,86 +124,76 @@ Browser │ HTTPS — managed TLS via Azure Container Apps │ ├─► Frontend Container App (nginx + React 18 + TypeScript) - │ Serves static bundle only. No backend proxy. - │ VITE_API_BASE_URL is baked into the bundle at build time. - │ All API calls go directly: browser → backend HTTPS URL. + │ Serves the static bundle only — no backend proxy. + │ VITE_API_BASE_URL is baked into the bundle at build time; + │ the browser calls the backend's ACA URL directly. │ └─► Backend Container App (FastAPI + asyncpg + Claude) min-replicas: 0 (scale to zero — wakes on first HTTP request) - max-replicas: 3 (HPA on CPU) - │ + max-replicas: 3 (autoscale on CPU) + │ Langfuse (traces + cost) ▼ Neon Serverless PostgreSQL - Connection pooling via asyncpg (min=2, max=10) - sslmode=require enforced at connection string level + asyncpg pool (min=2, max=10) · sslmode=require ``` -**CI/CD:** +**CI/CD** + ``` -Pull request → ruff lint · pytest unit tests · tsc typecheck · Trivy scan -Merge to main → Neon migrations · deploy backend · deploy frontend · smoke test · rollback on failure -Daily 09:00 → Anthropic cost check → GitHub issue if daily spend > $5 +Pull request → ruff · pytest · tsc · Trivy +Merge to main → Neon migrations → deploy backend → deploy frontend → smoke test → rollback on failure +Daily 09:00 → Anthropic spend check → open a GitHub issue if daily cost > $5 ``` --- ## Engineering decisions -### Multi-app networking in Azure Container Apps +Each of these is a decision I'd defend in a design review, stated with the alternative I rejected and the cost I accepted. -Docker Compose and Azure Container Apps have a different networking model. In Docker Compose, services share a bridge network — `http://backend:8000` works because `backend` resolves to the container hostname. In ACA, each Container App is isolated on its own network and has its own public HTTPS URL. There is no shared hostname. - -The consequence: an nginx `proxy_pass http://insightiq-backend:8000` that works locally fails silently in production with a 502. The fix is architectural, not a config tweak. nginx in production serves only static files. The React bundle calls the backend at its full ACA URL, injected via `VITE_API_BASE_URL` at build time. The backend CORS config must explicitly allow the frontend's ACA domain, injected as `FRONTEND_URL` at runtime by the CD pipeline. +### Multi-app networking in Azure Container Apps -| | Docker Compose (local) | Azure Container Apps (production) | -|--|------------------------|-----------------------------------| -| Frontend → Backend | nginx `proxy_pass http://backend:8000` | React calls full HTTPS ACA URL directly | -| Backend hostname | Docker bridge DNS | Public ACA FQDN | -| CORS required | No (same-origin via proxy) | Yes (separate domains) | -| nginx role | Reverse proxy + SPA server | SPA server only | +Docker Compose gives services a shared bridge network, so `http://backend:8000` resolves. ACA does not: each Container App is network-isolated with its own public FQDN. An nginx `proxy_pass http://insightiq-backend:8000` that works locally fails in production with a silent 502. -### Self-healing SQL capped at 3 retries +The fix is architectural, not a config tweak. In production, nginx serves static files only; the React bundle calls the backend at its full ACA URL (injected as `VITE_API_BASE_URL` at build time), and the backend CORS allow-list names the frontend's ACA domain (injected as `FRONTEND_URL` at runtime). -When generated SQL fails, the error message and the failed SQL are appended to the Claude conversation and a new attempt is made. The loop runs up to 3 times. +| | Docker Compose (local) | Azure Container Apps (prod) | +|--|------------------------|-----------------------------| +| Frontend → Backend | nginx `proxy_pass` | React calls full HTTPS ACA URL | +| Backend hostname | bridge DNS | public ACA FQDN | +| CORS | not needed (same origin) | required (separate domains) | +| nginx role | reverse proxy + SPA | SPA server only | -The cap is deliberate. A fourth retry rarely succeeds where three have failed — at that point the issue is the question itself (ambiguous phrasing, references data that doesn't exist) or a schema mismatch that requires human intervention, not more LLM calls. Retrying further burns API credits on unrecoverable queries and degrades the user experience with a long wait before an eventual failure. The retry count is persisted in `query_history` and surfaced in the UI — monitoring retry rates is a useful proxy for prompt quality degradation over time. +### Self-healing SQL, capped at 3 retries -### Result caching eliminates redundant LLM calls +A failed query and its database error are appended to the conversation and retried. The cap is deliberate: a fourth attempt rarely succeeds where three failed — by then the issue is the question (ambiguous, references absent data) or a schema mismatch needing a human, not more tokens. Retrying further just burns credits and lengthens the wait before an inevitable failure. The retry count is persisted; monitoring its rate is a cheap proxy for prompt-quality drift. -Query results (rows, columns, explanation) are stored as JSONB in `query_history` on first execution. Replaying a past query via `GET /api/history/:id` is a free PostgreSQL read — Claude is not called again. +### Result caching over re-generation -Without this, every click on a history item would trigger a full Claude round-trip. At claude-sonnet-4-5 pricing (~$3/1M input tokens, $15/1M output tokens), replaying a 10-query session could cost as much as running it for the first time. The cache hit is surfaced in the UI with a visual indicator so the cost saving is transparent. +Results are stored as JSONB on first execution; replaying via `GET /api/history/:id` is a free Postgres read. Without it, every history click would be a full model round-trip — at sonnet-4-5 pricing, replaying a ten-query session could cost as much as running it fresh. The cache hit is shown in the UI so the saving is visible. ### Atomic rate limiting without Redis -Each user has a `query_limit` (default: 2) and `total_queries` counter stored in PostgreSQL. The check and increment happen in one statement: - ```sql UPDATE users SET total_queries = total_queries + 1 -WHERE id = $1 - AND is_active = TRUE - AND total_queries < query_limit +WHERE id = $1 AND is_active = TRUE AND total_queries < query_limit RETURNING id, total_queries, query_limit ``` -If the `UPDATE` returns no rows, the limit has been reached. If it returns a row, the request is permitted. This is atomic — concurrent requests from the same user cannot both pass the check. No Redis, no distributed lock, no separate check-then-increment race condition. +No rows returned means the limit is hit; a row means the request is allowed. The check and increment are one statement, so concurrent requests from the same user cannot both pass. A distributed lock or a Redis counter would be the textbook answer; at this scale the database does it correctly with zero extra infrastructure. ### Schema-aware prompting with sample values -The full PostgreSQL schema is injected into every Claude system prompt: table names, column names, data types, foreign key relationships, and 3 sample values per column. Sample values are particularly important — without them, Claude has to guess domain vocabulary. With them, it knows that `plan` contains `'starter'`, `'pro'`, `'business'`, `'enterprise'` rather than inventing values. - -The schema is cached in memory with a 5-minute TTL. An in-process cache is sufficient at this scale — no Redis needed, and cache invalidation on schema change happens naturally on pod restart. - -### asyncpg over psycopg2 - -asyncpg uses PostgreSQL's binary protocol rather than the text protocol used by psycopg2. This produces a measurable throughput difference for concurrent requests. More importantly, asyncpg is natively async — it integrates with FastAPI's event loop without the thread pool workarounds required by psycopg2 in an async context. The `asyncpg.create_pool()` pool is initialised during FastAPI lifespan startup, so connections are warm before the first request arrives. +The full schema — tables, columns, types, foreign keys, and 3 sample values per column — is injected into every system prompt. The sample values matter most: without them the model invents domain vocabulary; with them it knows `plan` is one of `starter/pro/business/enterprise`. The schema is cached in-process with a 5-minute TTL — sufficient at this scale, and invalidation falls out naturally on pod restart. ### Read-only guard: token parsing, not regex -SQL validation uses token-level parsing rather than a regex pattern match on the query string. Comments are stripped first (`--` and `/* */`), then the first token is extracted and compared. A regex like `re.match(r'SELECT', sql, re.IGNORECASE)` can be bypassed with leading whitespace, comments, or encoding tricks. Token parsing handles these cases correctly. +Validation strips comments (`--`, `/* */`), extracts the first token, and compares it; a secondary scan rejects dangerous keywords anywhere in the string. A regex on the raw query is bypassable with whitespace, comments, or encoding; token parsing is not, and the keyword scan catches semicolon-separated injection. *(This is the application-layer control. The database-layer control — a read-only role — is on the roadmap; see below.)* -A secondary scan checks for dangerous keywords (`DROP`, `DELETE`, `INSERT`, `UPDATE`, `TRUNCATE`, `ALTER`) anywhere in the query after comment removal, catching semicolon-separated injection attempts like `SELECT 1; DROP TABLE customers`. +### asyncpg over psycopg2 + +asyncpg speaks PostgreSQL's binary protocol and is natively async, so it joins FastAPI's event loop without the thread-pool workarounds psycopg2 needs. The pool is created during lifespan startup, so connections are warm before the first request. --- @@ -173,80 +201,106 @@ A secondary scan checks for dangerous keywords (`DROP`, `DELETE`, `INSERT`, `UPD ``` customers 1,000 rows — company, plan, country, health_score, churn date - └── subscriptions MRR, billing cycle, status, cancellation - └── events ~50K product usage: login, feature_used, report_export, ... - └── invoices ~3K amount, status: paid | overdue | failed + ├── subscriptions MRR, billing cycle, status, cancellation + ├── events ~50K product usage: login, feature_used, report_export, … + ├── invoices ~3K amount, status: paid | overdue | failed └── support_tickets ~4K priority: low | medium | high | critical users registered users — google_id, email, query budget └── query_history every question, generated SQL, results (JSONB), timing, retries Views: - customer_health joins customers + active subscription + 30-day events + open tickets + customer_health customers + active subscription + 30-day events + open tickets monthly_mrr pre-aggregated MRR by month across active subscriptions ``` +Seed data is reproducible (`seed=42`): 1,000 customers and ~60K related rows via Faker. + --- -## Secrets management +## Observability & operations + +I treat the alert thresholds below as informal SLOs — the lines past which something is wrong and I want to know before a user tells me. + +**Tracing** — Langfuse on every query: the SQL, the timing, the retries, the cost. Reconstructing "why was this answer wrong" is a trace lookup, not a log grep. + +**Azure Monitor** — email alerts on backend CPU > 80% for 5 min, HTTP 5xx > 10 in 5 min (sev 1), and P95 latency > 10s. + +**Cost monitor** — a scheduled GitHub Action queries the Anthropic usage API and opens an issue automatically if daily spend clears $5; the issue ships with a SQL template to find unusual volume in `query_history`. An Azure budget emails at 80% and 100% of $30/month. -No credentials are stored in the repository. +**Deploys** — rolling and zero-downtime; migrations run before either app updates; a failed smoke test triggers `az containerapp revision deactivate` for an automatic rollback. + +--- + +## Testing + +CI runs on every PR: `ruff` lint, `pytest`, `tsc` typecheck, and a Trivy image scan. + +Current unit coverage focuses on the parts where a bug is dangerous or silent: the read-only guard (12 cases, including comment and semicolon-injection bypasses), result serialisation and Pydantic boundaries, and auth — JWT verification, the atomic rate-limit path, and 401 handling. An execution-accuracy evaluation gate is the next addition (see roadmap); it is not in CI yet, and I'd rather say so than badge it. + +--- + +## Security & secrets + +No credentials live in the repository. ``` -Source (public) GitHub Secrets (CI/CD) Azure runtime (Container App) -──────────────── ────────────────────────── ────────────────────────────── -Source code AZURE_CLIENT_ID OIDC ANTHROPIC_API_KEY secretref: -Dockerfiles NEON_DATABASE_URL JWT_SECRET_KEY secretref: -nginx.conf ANTHROPIC_API_KEY DATABASE_URL secretref: -Workflow definitions JWT_SECRET_KEY GOOGLE_CLIENT_ID secretref: - GOOGLE_CLIENT_ID FRONTEND_URL plain env - FRONTEND_URL +Source (public) GitHub Secrets (CI/CD) Azure runtime (Container App) +─────────────── ────────────────────── ───────────────────────────── +Source code AZURE_CLIENT_ID (OIDC) ANTHROPIC_API_KEY secretref: +Dockerfiles NEON_DATABASE_URL JWT_SECRET_KEY secretref: +nginx.conf ANTHROPIC_API_KEY DATABASE_URL secretref: +Workflows JWT_SECRET_KEY GOOGLE_CLIENT_ID secretref: + GOOGLE_CLIENT_ID FRONTEND_URL plain env + FRONTEND_URL ``` -Azure authentication uses OIDC federated credentials — GitHub receives a short-lived token exchange rather than storing a service principal password. The credential is scoped to this repository and branch only. +Azure auth is OIDC federated, scoped to this repository and branch — GitHub exchanges a short-lived token instead of holding a service-principal password. Runtime secrets are Container App `secretref` values, so they never appear in the plaintext revision spec. Auth is Google OAuth with server-side token verification and HS256 JWTs; no passwords are stored. --- -## Monitoring +## Cost -**Azure Monitor** — email alerts on: -- Backend CPU > 80% for 5 minutes -- HTTP 5xx rate > 10 in 5 minutes (severity 1) -- P95 response latency > 10 seconds +| Service | Monthly | +|---------|---------| +| Container Apps — backend (scale to zero) | ~$1–5 | +| Container Apps — frontend (scale to zero) | ~$1–3 | +| Neon Serverless PostgreSQL | $0 (free tier) | +| GitHub Actions | $0 (free tier) | +| **Infrastructure total** | **~$2–8** | -**Anthropic cost monitor** — GitHub Actions workflow on a daily schedule. Queries the Anthropic usage API, calculates spend, and opens a GitHub issue automatically if daily cost exceeds $5. The issue includes a SQL template to identify unusual query volume in `query_history`. +Model cost scales with volume. A **measured production trace** (full schema prompt, sonnet-4-5) costs **~$0.06** per `generate_sql` call (~19k input tokens) and **~$0.12** end-to-end when the self-healing agent retries once (38.5k prompt + 201 completion tokens, 17.45s total). The dominant term is the injected schema, so prompt caching or a slimmer schema is the lever toward **~$0.004** per query. Every query’s `cost_usd` lands on the Langfuse trace and in `query_audit`; `daily_cost_spend` drives the budget webhook. The per-user cap and JSONB caching bound marginal cost per visitor. -**Azure budget** — email notification at 80% and 100% of $30/month. +![Langfuse cost trace — insightiq_query with one retry](docs/cost.png) + +See [docs/cost.md](docs/cost.md) for the Langfuse dashboard walkthrough and SQL to compute live averages. --- -## Tech stack +## Limitations & hardening roadmap + +What a system *doesn't* do yet is as honest a signal as what it does. In rough priority order: -| Layer | Choice | Rationale | -|-------|--------|-----------| -| LLM | Claude claude-sonnet-4-5 | Consistent structured JSON output; superior instruction-following for SQL generation | -| Backend | FastAPI + Python 3.12 | Async-native; automatic OpenAPI docs; Pydantic v2 validation at boundary | -| DB driver | asyncpg | Binary protocol; native async; 3× psycopg2 throughput for concurrent queries | -| Database | PostgreSQL 16 (Neon) | JSONB for event properties; window functions for MRR; Neon free tier for production | -| Frontend | React 18 + TypeScript (strict) | `noImplicitAny`, `noUnusedLocals` enforced; zero `any` types | -| Auth | Google OAuth + JWT (HS256) | No passwords stored; server-side token verification; single provider simplifies codebase | -| Styling | Tailwind CSS + Tabler Icons | Utility-first; no custom CSS; consistent design system | -| Charts | Chart.js (lazy-loaded) | Keeps initial bundle lean; bar/line/pie with one API | -| CI/CD | GitHub Actions + Azure Container Apps | OIDC auth; no stored credentials; ~2 min CI; scale-to-zero cost model | +1. **Dedicated read-only database role.** Today the app connects with a single write-capable role, so the read-only guard is the only write-prevention. Next step: a `GRANT SELECT`-only role on the request path, so a guard bypass is stopped by Postgres itself, not just the application. Highest priority — it's the difference between two locks and one. +2. **AST-based validation.** Move the guard from token parsing to a `sqlglot` AST walk. That enables a real table/column allow-list and closes CTE/encoding edge cases that token scanning can only approximate. +3. **Forced `LIMIT` + PII column policy.** Cap result-set size at the query layer so no question can return the whole table; deny sensitive columns by policy rather than by prompt. +4. **Execution-accuracy eval gate in CI.** A labelled question/gold-SQL set graded by *result-set equivalence* (not string match — two correct queries can look nothing alike), wired as a merge gate so a correctness regression blocks the deploy. +5. **Single-region data.** One Neon instance, and the schema cache is per-pod in-process; a multi-region deployment would need a shared cache and a read-replica strategy. +6. **Quotas are per-user, not per-org.** Fine for a portfolio; a real tenant model would meter and bill per organisation. --- -## Local development +## Running it ```bash git clone https://github.com/AttiR/InsightIQ cd InsightIQ -cp .env.example .env # add ANTHROPIC_API_KEY, GOOGLE_CLIENT_ID, JWT_SECRET_KEY +cp .env.example .env # ANTHROPIC_API_KEY, GOOGLE_CLIENT_ID, JWT_SECRET_KEY # Postgres docker compose up -d psql $DATABASE_URL -f database/migrations/002_auth.sql -python3 database/seed.py # 1,000 customers + ~60K related rows +python3 database/seed.py # 1,000 customers + ~60K related rows (seed=42) # Backend cd backend @@ -260,87 +314,68 @@ npm install npm run dev # http://localhost:5173 ``` ---- - -## Production deployment +**Production deploy** (automatic on merge to `main`): ```bash -# One-time Azure provisioning -chmod +x infra/provision.sh && ./infra/provision.sh - -# Monitoring alerts -chmod +x infra/alerts.sh && ./infra/alerts.sh your@email.com - -# Deploy (triggered automatically on merge to main) -git push origin main +./infra/provision.sh # one-time: ACA environment + OIDC federated credentials +./infra/alerts.sh your@email.com # Azure Monitor: CPU · 5xx · latency · budget +git push origin main # CI/CD takes it from here ``` -All deploys are zero-downtime rolling updates. The CD pipeline applies Neon migrations before updating either Container App. Smoke test failure triggers automatic rollback via `az containerapp revision deactivate`. +Deploys are zero-downtime rolling updates; migrations apply before either Container App updates; a failed smoke test rolls back automatically. --- -## Monthly cost - -| Service | Cost | -|---------|------| -| Azure Container Apps — backend (scale to zero) | ~$1–5 | -| Azure Container Apps — frontend (scale to zero) | ~$1–3 | -| Neon Serverless PostgreSQL | $0 (free tier) | -| GitHub Actions | $0 (free tier) | -| **Total infrastructure** | **~$2–8/month** | - -Anthropic API cost scales with query volume. **Measured average per query: ~$0.03** at current schema-injection size (~10.5k input + ~130 output tokens on claude-sonnet-4-5). With prompt caching or a slimmer schema, that drops toward **~$0.004**. Each query is metered in Langfuse (`cost_usd` on the trace) and `query_audit`; daily spend is tracked in `daily_cost_spend` with a webhook alert when `BUDGET_USD_DAILY` is exceeded. The per-user query cap and JSONB result caching bound marginal cost per visitor. +## Project layout -See [docs/cost.md](docs/cost.md) for the Langfuse dashboard screenshot and SQL to compute live averages. - ---- - -## Project structure +
+Full tree ``` InsightIQ/ -├── .github/ -│ └── workflows/ -│ ├── ci.yml ruff · pytest · tsc · trivy (every PR) -│ ├── cd.yml migrate · deploy · smoke test · rollback (main) -│ └── cost-monitor.yml daily Anthropic spend check +├── .github/workflows/ +│ ├── ci.yml ruff · pytest · tsc · trivy (every PR) +│ ├── cd.yml migrate · deploy · smoke test · rollback (main) +│ └── cost-monitor.yml daily Anthropic spend check ├── backend/ │ ├── app/ -│ │ ├── agent/ -│ │ │ └── sql_agent.py schema injection · generation · guard · healing · cache -│ │ ├── api/ -│ │ │ └── query.py POST /api/query · GET /api/history · GET /api/schema +│ │ ├── agent/sql_agent.py schema injection · generation · guard · healing · cache +│ │ ├── api/query.py POST /api/query · GET /api/history · GET /api/schema │ │ ├── auth/ -│ │ │ ├── router.py POST /api/auth/google · GET /api/auth/me -│ │ │ ├── service.py Google token verify · JWT · rate limit (atomic UPDATE) -│ │ │ └── dependencies.py get_current_user FastAPI dependency -│ │ ├── core/ -│ │ │ └── config.py Pydantic Settings · CORS · FRONTEND_URL +│ │ │ ├── router.py POST /api/auth/google · GET /api/auth/me +│ │ │ ├── service.py Google verify · JWT · rate limit (atomic UPDATE) +│ │ │ └── dependencies.py get_current_user dependency +│ │ ├── core/config.py Pydantic Settings · CORS · FRONTEND_URL │ │ └── db/ -│ │ ├── pool.py asyncpg pool · type codecs · timeout guard +│ │ ├── pool.py asyncpg pool · type codecs · timeout guard │ │ └── schema_loader.py schema + sample values · 5-min TTL cache -│ ├── Dockerfile multi-stage · non-root (UID 1001) · healthcheck +│ ├── Dockerfile multi-stage · non-root (UID 1001) · healthcheck │ └── requirements.txt ├── frontend/ │ ├── src/ -│ │ ├── api/client.ts typed fetch · Authorization header · abort signals -│ │ ├── context/ AuthContext — Google sign-in · JWT · refreshUser -│ │ ├── hooks/ useQuery · useHistory (with loadFromHistory) -│ │ ├── components/ TopNav · Sidebar · ChartView · DataTable · SQLViewer -│ │ └── pages/LoginPage.tsx split-screen · Google Identity Services SDK -│ ├── Dockerfile multi-stage Node build · nginx runtime · non-root -│ ├── nginx.conf SPA routing · security headers · no backend proxy -│ └── tsconfig.json strict · noImplicitAny · noUnusedLocals +│ │ ├── api/client.ts typed fetch · Authorization header · abort signals +│ │ ├── context/ AuthContext — Google sign-in · JWT · refreshUser +│ │ ├── hooks/ useQuery · useHistory (loadFromHistory) +│ │ ├── components/ TopNav · Sidebar · ChartView · DataTable · SQLViewer +│ │ └── pages/LoginPage.tsx split-screen · Google Identity Services +│ ├── Dockerfile multi-stage Node build · nginx runtime · non-root +│ ├── nginx.conf SPA routing · security headers · no backend proxy +│ └── tsconfig.json strict · noImplicitAny · noUnusedLocals ├── database/ -│ ├── migrations/ -│ │ └── 002_auth.sql users · query_history user_id + result_json -│ └── seed.py 1,000 customers · ~60K events · faker · reproducible (seed=42) +│ ├── migrations/002_auth.sql users · query_history user_id + result_json +│ └── seed.py 1,000 customers · ~60K events · Faker · seed=42 ├── infra/ -│ ├── provision.sh ACA environment · OIDC federated credentials -│ └── alerts.sh Azure Monitor · CPU · 5xx · latency · budget +│ ├── provision.sh ACA environment · OIDC federated credentials +│ └── alerts.sh Azure Monitor · CPU · 5xx · latency · budget ├── tests/ -│ ├── test_sql_agent.py read-only guard (12 cases) · serialisation · Pydantic -│ └── test_auth.py JWT unit tests · rate limit (atomic mock) · auth 401 -├── docker-compose.yml local: postgres + backend + frontend +│ ├── test_sql_agent.py read-only guard (12 cases) · serialisation · Pydantic +│ └── test_auth.py JWT · rate limit (atomic mock) · auth 401 +├── docker-compose.yml └── .env.example ``` + +
+ +--- + +*Built by Atti Rehman · [LinkedIn](https://www.linkedin.com/in/attirehman/) · [GitHub](https://github.com/AttiR)* diff --git a/docs/cost.md b/docs/cost.md index db17c13..d482e93 100644 --- a/docs/cost.md +++ b/docs/cost.md @@ -9,7 +9,22 @@ Per-query Claude spend is recorded in **Langfuse** (`cost_usd` on each `insighti 3. Open any trace → check metadata `cost_usd`, `input_tokens`, `output_tokens`. 4. For aggregate cost: **Analytics** → group by trace name or use the cost column if enabled on your plan. -Save a screenshot as `docs/cost.png` for your portfolio README. +## Example trace (production) + +![Langfuse trace — MRR trend query with one retry](cost.png) + +Real `insightiq_query` trace (claude-sonnet-4-5, June 2026): + +| Metric | Value | +|--------|-------| +| Question | “What is our MRR trend for the last 6 months?” | +| Total cost | **$0.118** | +| Tokens (prompt / completion) | 38,488 / 201 | +| Retries | 1 | +| Total latency | 17.45s | +| DB execution | 62ms · 6 rows | + +Each `generate_sql` span in the trace is ~19k prompt tokens and ~**$0.059** on its own. ## Typical cost per query @@ -17,10 +32,11 @@ At claude-sonnet-4-5 rates ($3/MTok in, $15/MTok out): | Scenario | Tokens (in / out) | Cost | |----------|-------------------|------| -| Measured (full schema prompt) | ~10,500 / ~130 | **~$0.03** | -| Optimized (caching / slim schema) | ~1,000 / ~130 | **~$0.004** | +| Measured — one `generate_sql` call | ~19,250 / ~100 | **~$0.06** | +| Measured — with 1 retry (trace above) | ~38,500 / ~200 | **~$0.12** | +| Optimized (caching / slim schema) | ~1,000 / ~100 | **~$0.004** | -Retries add linearly — a 3-attempt heal costs up to ~3× that. +Retries add linearly — each extra `generate_sql` costs another ~$0.06 at the current schema-injection size. ## Daily budget alert diff --git a/docs/cost.png b/docs/cost.png new file mode 100644 index 0000000..bd3582f Binary files /dev/null and b/docs/cost.png differ