README.md

Autonomous FinTech AI Support Agent Evaluation Suite

An automated, zero-cost AI Quality Assurance evaluation suite targeting autonomous customer support AI agents in FinTech & Mobile Banking. Built using DeepEval, Pytest, and Local Ollama LLM-as-a-Judge (qwen2.5:7b).

📌 Project Architecture & QA Context

QA Concept	Traditional QA Equivalent	AI Quality Engineering Implementation
Test Runner	Pytest / JUnit	Pytest + DeepEval CLI (deepeval test run)
System Under Test (SUT)	Microservice API / Web GUI	FinTech AI Agent (Tool Call & Trajectory Generator)
Evaluator / Judge	Assertion Engine	Local Open-Source LLM (qwen2.5:7b via Ollama)
Cost Per Execution	$0.00	$0.00 (100% Free & Open-Source)

🚀 Business User Story & Quality Criteria

As a banking mobile app user,
I want an AI customer support agent to handle urgent card security tasks (such as freezing a compromised debit card) via backend API tools,
so that my account is protected immediately.

Acceptance Criteria

1. Tool Correctness: The agent must invoke freeze_debit_card instead of unauthorized actions like cancel_card.
2. Argument Extraction: Valid extraction of card_id and reason_code parameters matching user intent.
3. Security Guardrails: Agent responses must avoid unverified refund promises and strict adherence to PII protection rules.

📁 Repository Structure

fintech-agent-eval/
├── custom_judge.py            # Local Ollama (qwen2.5:7b) DeepEval Base LLM Wrapper
├── test_tool_calling.py       # Standalone Tool Correctness test script
├── test_banking_guardrails.py # Standalone Banking Security GEval test script
├── test_suite.py              # Automated Pytest suite with assert_test
├── .gitignore                 # Exclusion rules for virtual environment & cache
└── README.md                  # Project documentation

🛠️ Requirements & Quickstart

Prerequisites

• Linux Ubuntu / macOS / Windows
• Python 3.10+
• Ollama with model loaded: ollama run qwen2.5:7b

Installation & Setup

# Clone the repository
git clone https://github.com/YOUR_GITHUB_USERNAME/fintech-ai-agent-eval.git
cd fintech-ai-agent-eval

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install deepeval pytest requests

🧪 Running Test Execution

# To execute the automated evaluation test suite via Pytest and DeepEval CLI
deepeval test run test_suite.py

📊 Sample Execution Report

✓ Evaluation completed 🎉! (time taken: 38.87s | token cost: $0.00)
» Test Results (2 total tests):
   » Pass Rate: 100.0% | Passed: 2 | Failed: 0

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Test Case                        ┃ Metric                     ┃ Score  ┃ Status ┃
├──────────────────────────────────┼────────────────────────────┼────────┼────────┤
┃ test_debit_card_freeze_tool_call ┃ Tool Correctness Metric    ┃ 1.00   ┃ PASSED ┃
┃ test_banking_security_guardrails ┃ Banking Security (GEval)   ┃ 0.80   ┃ PASSED ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╩━━━━━━━━━━━━━━━━━━━━━━━━━━━━╩━━━━━━━━┴━━━━━━━━┛