knowledge-agents

A four-agent customer-support workflow built around one architectural decision worth defending: a two-stage Confidence Gate that decides whether the model is allowed to answer at all.

This repo started as my capstone for the Udacity Agentic AI nanodegree. I later did a docs and eval-harness pass on it, framed as a deployment retro: here's what I'd ship in two weeks, here's what I cut, here's what I measured, and here's how I'd hand it off.

If you're skimming for the substance, the four artifacts below are what's actually worth your time. Read in this order: CASE_STUDY.md, evals/results.md, ARCHITECTURE.md, DEPLOYMENT_PLAYBOOK.md. Eight minutes total.

---

Shipped / Cut / Measured

Shipped (v1)	Cut to v2	Measured
Four-agent LangGraph workflow with confidence-routed escalation	Tiered model routing (Haiku class / Sonnet class)	Classification accuracy on 15-example hand-graded dataset
Two-stage Confidence Gate with derived resolver-side confidence	Semantic retrieval (chromadb, sentence-transformers — deps installed but unused)	Escalation F1 (precision and recall, with recall weighted heavier)
Dual-SQLite architecture (read-only customer side, writable agent side)	Postgres + pgvector + PostgresSaver for multi-tenant concurrency	Tool-selection F1
agent_decisions audit log capturing every confidence score	Calibration loop that fits gate thresholds to historical outcomes	p50 / p95 latency
Eval harness with cost guardrail (defaults to $5 cap)	LLM-as-judge response-quality scoring	Per-ticket cost on gpt-4o-mini
Demo script and deployment handoff playbook	OTel spans and production observability	Failure-mode breakdown (under-escalation, over-escalation, retrieval-blind, safety breach)

Every cell links to the artifact that justifies it. The discipline behind this table is what CASE_STUDY.md is about.

---

The Confidence Gate

The interesting question in a multi-agent support graph isn't "how many agents?" — it's "when do you let the model commit to an answer?"

The Gate answers that with ==a two-stage router: escalate when the classifier isn't sure enough to act, and escalate again when the resolver isn't sure enough to ship.==

Stage 1 — Supervisor gate (over the classifier's self-reported confidence):

# my_submission/src/agentic/agents/supervisor_agent.py (paraphrased)
if classification_confidence < 0.20:    # critical: don't even try to resolve
    return ImmediateEscalation(priority="P1")
elif classification_confidence < 0.40:  # high: escalate, but enqueue
    return ImmediateEscalation(priority="P2")
else:
    return ProceedWithResolution(priority="standard")

Stage 2 — Resolver gate (over a derived confidence, not the resolver's self-report):

# my_submission/src/agentic/agents/resolver_agent.py (paraphrased)
resolver_confidence  = classification_confidence * 0.7
resolver_confidence -= 0.10 if customer_sentiment == "frustrated" else 0
resolver_confidence -= 0.15 if complexity_signals_present else 0

if resolver_confidence < 0.50:
    return Escalate()  # tried, not confident enough to send
else:
    return SendResponse()

The asymmetry is deliberate. The classifier reports its own confidence and the supervisor reads it. The resolver does not report its own confidence — it's computed from the classifier's number, penalized for known confidence-killers (frustrated sentiment, complexity flags). The resolver is structurally skeptical of itself, because LLMs are calibrated to sound confident.

Everything else — the four-agent split, the separate SupervisorAgent, the EscalationAgent's context-packaging job — is downstream of that two-stage gate. The supervisor exists as a separate agent because the gate needs an opinion-holder that isn't the resolver. If the resolver decided whether to escalate, the resolver would rationalize.

The thresholds (0.20, 0.40, 0.50) are reasonable defaults, not calibrated against outcomes. Calibrating them is the highest-leverage v2 item; see DEPLOYMENT_PLAYBOOK.md for the procedure.

---

Architecture

                    [START]
                       │
                       ▼
                 ┌───────────┐
                 │ classify  │   ClassifierAgent
                 │           │   category + entities + confidence
                 └─────┬─────┘
                       │
                       ▼
                 ┌───────────┐
                 │ supervise │   SupervisorAgent
                 │           │   ── Confidence Gate, stage 1 ──
                 └─────┬─────┘   < 0.20 → escalate P1 (critical)
                       │         < 0.40 → escalate P2 (high)
                       │         ≥ 0.40 → continue
                       ▼
            ┌──────────────────┐
            │ retrieve_knowledge│  KnowledgeRetrievalTool (keyword)
            └────────┬─────────┘
                     ▼
            ┌──────────────────┐
            │  execute_tools   │  AccountLookup, Subscription
            └────────┬─────────┘
                     ▼
                 ┌─────────┐
                 │ resolve │     ResolverAgent
                 │         │     ── Confidence Gate, stage 2 ──
                 └────┬────┘     derived = classification × 0.7 − penalties
                      │          < 0.50 → escalate
                      │          ≥ 0.50 → finalize
              ┌───────┴────────┐
              ▼                ▼
        ┌──────────┐    ┌──────────┐
        │ escalate │    │ finalize │
        └────┬─────┘    └────┬─────┘
             │               │
             └───────┬───────┘
                     ▼
                   [END]

Mermaid sequence diagrams for the happy path and both escalation paths, plus the full state contract for EnhancedAgentState, are in ARCHITECTURE.md.

The fictional company is CultPass, a cultural-experiences subscription service invented for the course (members book gallery nights, food tours, workshops). Every user, subscription, and reservation in cultpass.db is seed data. The classifier recognizes seven ticket categories: LOGIN_ISSUE, BILLING_PAYMENT, RESERVATION_BOOKING, TECHNICAL_ISSUE, ACCOUNT_MANAGEMENT, GENERAL_INQUIRY, ESCALATION_REQUIRED.

---

Eval results

A hand-graded benchmark lives at evals/. 15 examples covering happy-path, frustrated escalation, ambiguous low-confidence, paraphrased retrieval failures, off-policy requests, and adversarial inputs (prompt injection + social engineering).

Metric	Target	Status
Classification accuracy (non-adversarial)	≥ 80%	See evals/results.md
Escalation F1 (recall-weighted)	≥ 75%	See evals/results.md
Tool-selection F1	≥ 70%	See evals/results.md
p95 latency	< 8s	See evals/results.md
Per-ticket cost	< $0.05	See evals/results.md
Safety breaches on 2 adversarial examples	0	See evals/results.md

Cost guardrail: evals/runner.py defaults to a $5 budget cap and refuses to start above it without --override-budget. Dry-run cost estimate is printed before any API call.

Methodology in evals/rubric.md. To populate live numbers:

uv run --project my_submission python evals/runner.py --confirm   # ~$0.03 against gpt-4o-mini
uv run --project my_submission python evals/score.py

---

Where this breaks

The honest list. Each item is something I'd fix before letting this near real users.

Discovered live, while writing this README. Running demo.py against the system surfaced three P0/P1 bugs the original test suite missed: the classifier returns identical output for every ticket; the resolver returns prompt-template variables (literal {specific topic from ticket}) in its responses; and per-ticket latency is 4–10× over the documented target. Full punch list, evidence, and root-cause hypotheses in FINDINGS.md. The original comprehensive_tests.py returns 9/9 green; the system is broken. That gap is exactly why the eval harness exists.

1. Keyword retrieval is the load-bearing weakness. KnowledgeRetrievalTool scores articles by lexical overlap (title × 3, tags × 2, content × 1). Two eval examples (eval-010, eval-011) phrase real problems in non-canonical language — the retrieval misses, the resolver doesn't get the relevant article, and the Confidence Gate may or may not catch the downstream miss. Fix: wire up the already-installed chromadb + sentence-transformers deps; keep keyword search as fallback.

2. The thresholds aren't calibrated. 0.20, 0.40, 0.50 are vibes, not measurements. The data to calibrate them sits in agent_decisions already. The calibration loop is the v2 keystone. Procedure documented in DEPLOYMENT_PLAYBOOK.md.

3. One model for all four agents. gpt-4o-mini everywhere means the high-volume classifier pays the same per-token rate as the customer-facing resolver. Tiered routing (Haiku-class for classify/supervise, Sonnet-class for resolve/escalate) is a measurable-ROI v2 win.

4. Module-level orchestrator instantiation. enhanced_workflow.py:741 does enhanced_uda_hub = EnhancedUDAHubOrchestrator() at import time. Any consumer pays four LLM client inits just to import the module. Fix: remove the singleton; require explicit construction.

5. MemorySaver is in-process. LangGraph's default checkpointer lives in memory. Two concurrent invocations on the same thread_id will interleave state non-deterministically. Fine for one user in a notebook; broken the moment more than one user is in flight. Swap to SqliteSaver or PostgresSaver.

6. Confidence is reported, not calibrated. The classifier returns a number between 0 and 1; nothing in the loop verifies that 0.85 actually corresponds to 85% historical correctness. The infrastructure to fix this (logging in agent_decisions) exists; the calibration step doesn't.

---

Run it yourself

Three commands, then a demo.

# 1. Configure
cp my_submission/.env.example my_submission/.env
# Edit my_submission/.env to paste OPENAI_API_KEY

# 2. Install
cd my_submission && uv sync && cd ..

# 3. Demo (4 hand-picked tickets, ~30 seconds, ~$0.02)
uv run --project my_submission python demo.py

The demo exercises four distinct paths through the graph: auto-resolve, Stage-1 escalation, Stage-2 escalation, and tool-driven resolution. Each ticket's full agent trace is printed; a markdown transcript is saved to demo_output.md.

For the eval harness, see evals/README.md.

---

What's in this repo

Artifact	What it demonstrates	Audience
CASE_STUDY.md	Customer translation, scope-cut discipline, deployment-retro narrative	Anyone evaluating engineering judgment
ARCHITECTURE.md	Technical reasoning, sequence diagrams, state contract, extension points	Tech leads, eng managers
evals/	Measurement rigor, adversarial-input handling, failure-mode taxonomy	Tech leads, ML engineers
FINDINGS.md	Three P0/P1 bugs surfaced by running my own demo; root-cause analysis	Anyone evaluating diagnostic skill
DEPLOYMENT_PLAYBOOK.md	Handoff quality, Postgres migration recipe, on-call runbook	Eng managers, customer ops
demo.py + demo_script.md	Prototype velocity, runnable proof	Anyone with 3 minutes
demo_output.md	Live trace from running demo.py — including the bug evidence	Anyone evaluating "does it actually run"
my_submission/	The actual code, plus the original course-submission readme	Anyone tracing claims to source

The my_submission/ directory is the original course capstone — kept intact as audit trail. The course-graded README is at my_submission/readme.md. Treat it as a longer, more pedagogical version of this document.

---

Related reading

I write about agentic-AI infrastructure at sharadja.in. Posts that touch on the design ideas in this repo:

• Orchestrating AI Agents — the supervisor pattern beyond customer support
• The 14K Token Debt — system prompt as architecture, prompt gravity
• Your MCP Servers Are Costing You 10 Seconds Per Session — schema gravity, tool overhead as real overhead

---

Course context

This was originally my capstone submission for the Udacity Agentic AI Nanodegree, project: Knowledge Agents. The grader-facing README and the full rubric audit trail live at my_submission/readme.md. The reframe in this document is post-hoc.

---

Sharad Jain — sharadja.in — @Imsharad. The blog is the technical reading; this repo is the working artifact. Reachable via the email on the blog.