Skip to content

TESTER_GUIDE.md

Latest commit

 

History

History
134 lines (108 loc) · 7.95 KB

TESTER_GUIDE.md

File metadata and controls

134 lines (108 loc) · 7.95 KB
Raw

🛡️ Chargeback Evidence Responder — Evaluator & Tester Guide

Welcome to the Chargeback Evidence Responder project evaluation suite. This enterprise-grade SaaS application uses machine learning classification and grounded LLM generation to automate payment dispute adjudication and merchant defense package drafting.

🚀 Quickstart for Evaluators

1-Click Interactive Launcher

Double-click run.bat or run from your terminal:

run.bat

This opens the Interactive Evaluator Control Menu:

============================================================
  CHARGEBACK EVIDENCE RESPONDER - EVALUATOR DASHBOARD
============================================================

  [1] Setup & Start Application (Full Quickstart)
  [2] Run Automated Integration API Tests (14 Endpoint Suite)
  [3] Test ML Model Robustness & Edge Case Handling
  [4] Check Database & Backend Connectivity
  [5] Check Real API Connectivity & System Diagnostics
  [6] Validate Production Build
  [7] Exit

Select [1] to install dependencies, train the ML model artifact, and launch the dev server. Access the dashboard at: 👉 http://localhost:3000

🏛️ System Architecture

┌──────────────────────────────────────────────────────────────────────────────────────────┐
│                                   React 19 Frontend                                      │
│  - Real-Time Dispute Operations Table & Multi-Field Filters                              │
│  - Case Evidentiary Dossier & Decision Cockpit                                           │
│  - Held-Out Test Evaluation & Calibration Check Visualizer                               │
│  - Interactive Network Dispute-Ratio Risk Config Tuner                                   │
└────────────────────────────────────────┬─────────────────────────────────────────────────┘
                                         │ REST API (JSON)
                                         ▼
┌──────────────────────────────────────────────────────────────────────────────────────────┐
│                           Java 21 / Spring Boot 3.3 Backend (:8080)                      │
│  - API Gateway, MySQL Ledger Repository & Audit Trail Logger                             │
│  - Dispute-Ratio-Aware Decision Engine: T_eff = min(T_cap, T_base + alpha*(R_loss/R_ceil)^2)│
│  - Dynamic Risk Endpoints: GET /api/risk/ratio-status & POST /api/risk/config            │
│  - Mandatory Policy Guard: Restricts LLM defense drafting strictly to STRONG cases       │
└───────────────────┬──────────────────────────────────────────────────┬───────────────────┘
                    │ REST (HTTP)                                      │ REST (HTTP / SDK)
                    ▼                                                  ▼
┌───────────────────────────────────────┐          ┌───────────────────────────────────────┐
│     FastAPI ML Prediction Service     │          │      Anthropic Claude / Gemini AI      │
│  - Algorithm: RandomForestClassifier  │          │  - Evidence Grounding Engine          │
│  - Trained on 10,000 Dispute Cases    │          │  - 8-Part Formal Rebuttal Structure   │
│  - Outputs Calibrated Win Probability │          │  - Zero Fictitious Claim Rule        │
└───────────────────────────────────────┘          └───────────────────────────────────────┘

🧪 Evaluation Test Scenarios

Scenario A: High-Confidence Winnable Dispute (STRONG Zone)

  • Sample Case: CB-1024 or CB-1027
  • Case Profile: Transaction INR 50,000, 3DS authentication confirmed, item delivered with tracking & customer receipt acknowledgment.
  • Expected Outcome:
    1. ML Classifier calculates win probability $\ge 0.70$ (e.g. 94%).
    2. Decision assigned: STRONG.
    3. Action: Defend Dispute enabled. LLM drafts formal 8-part defense rebuttal letter.

Scenario B: Contested / Ambiguous Dispute (BORDERLINE Zone)

  • Sample Case: CB-1026
  • Case Profile: Transaction INR 28,500, marked delivered without signature lock, prior customer support inquiry logged.
  • Expected Outcome:
    1. ML Classifier calculates win probability between $0.40$ and $0.70$ (e.g. 56%).
    2. Decision assigned: BORDERLINE.
    3. Action: Manual Review Required. Automated LLM defense generation is restricted to prevent low-precision submissions.

Scenario C: Unwinnable Dispute (WEAK Zone)

  • Sample Case: CB-1025 or CB-1028
  • Case Profile: Missing tracking number, unconfirmed delivery, or prior full refund already issued.
  • Expected Outcome:
    1. ML Classifier calculates win probability $< 0.40$ (e.g. 18%).
    2. Decision assigned: WEAK.
    3. Action: Recommend Refund. Protects merchant ratio and avoids arbitration penalty fees.

⚡ Automated CLI Verification Commands

Evaluators can execute all system checks directly via CLI commands:

Command Purpose Expected Result
run.bat Run 1-Click Interactive Dashboard Open browser at http://localhost:3000
python ml/evaluate_model.py Evaluate ML Model Accuracy & Thresholds Test Accuracy: 85.80% | ROC-AUC: 0.8477
node database/test_connection.cjs Verify Database Connectivity & Schema Database schema verified & active
GET /api/health/full Full Diagnostic Health & Real API Check status: "UP", ml_classifier: "UP"
npm run build Production Bundle Build Check Clean build in dist/ directory

🔧 Environment Configuration (.env)

The application runs out of the box with zero external dependencies using built-in evidence-grounded templates. To test real-time live LLM generation with Google Gemini 3.8 Flash, set your API key in .env:

GEMINI_API_KEY=your_actual_gemini_api_key_here

📊 Summary of Held-Out Test Evaluation (2,000 Cases)

Metric Score Operational Value
Test Accuracy 85.80% Overall correct dispute discrimination
Precision ($p \ge 0.50$) 88.55% Base binary precision
Recall ($p \ge 0.50$) 95.09% Captures 95% of winnable legitimate disputes
ROC-AUC 0.8477 Strong probabilistic ranking ability
Defense Precision ($p \ge 0.70$) 92.49% 92.5% win rate on automated defense submissions
Net Recovered Value ₹28,036,118.45 INR 30.3M won minus INR 2.3M false positive cost

🏆 Project Selection Checklist for Evaluators

  • Full-stack architecture (React 19 + TypeScript + Java 21 Spring Boot + Python FastAPI + MySQL + Gemini AI).
  • Real-time interactive UI with filters, decision cockpit, and model threshold policy tuner.
  • Machine Learning classifier trained on 10,000 cases with held-out test split evaluation.
  • Grounded LLM generation with strict 8-part formal rebuttal formatting.
  • 14-point automated test suite and 1-click evaluation script (run.bat).