ToolChecker MCP Server

MCP (Model Context Protocol) server for AI tool comparison and verification. Provides three tools for extracting comparison requirements, gathering sources, and generating comprehensive comparison reports.

Features

Three MCP Tools:
- tc_extract: Extract comparison specifications from natural language queries
- tc_sources: Gather relevant sources for comparison
- tc_report: Generate comprehensive comparison reports in Notion-friendly markdown
Mock Mode: Development mode with realistic dummy data (no Core API required)
Robust Error Handling: Automatic retries with exponential backoff for transient failures
Request Tracking: Unique request IDs for end-to-end tracing
Data Privacy: Automatic masking of sensitive data in logs
Type Safety: Full Pydantic validation for all inputs and outputs

Architecture

┌─────────────────────┐
│  ChatGPT Dev Mode   │
│  (or any MCP host)  │
└──────────┬──────────┘
           │ HTTPS
           │
           ▼ GET /mcp (SSE stream)
    ┌──────────────────────┐
    │  MCP Server          │
    │  ├─ GET  /mcp        │ ← SSE endpoint (text/event-stream)
    │  ├─ POST /mcp/messages│ ← JSON-RPC 2.0 (initialize, tools/list, tools/call)
    │  └─ POST /mcp        │ ← Legacy custom protocol (dev/testing)
    └──────────┬───────────┘
               │
        ┌──────┴───────┐
        │ Mock Mode?   │
        └──┬────────┬──┘
           │ Yes    │ No
           ▼        ▼
      ┌─────────┐  ┌───────────┐
      │  Mock   │  │ Core API  │
      │ Service │  │  Client   │
      └─────────┘  └───────────┘

MCP Protocol Support

This server implements the MCP (Model Context Protocol) specification for seamless integration with ChatGPT Developer Mode and other MCP-compatible clients.

Endpoints

GET /mcp - SSE Stream Endpoint
- Returns: Content-Type: text/event-stream
- Purpose: Establishes a Server-Sent Events (SSE) connection
- Sends initial endpoint event with /mcp/messages URI
- Maintains connection with periodic pings
POST /mcp/messages - JSON-RPC Endpoint
- Accepts: Content-Type: application/json
- Protocol: JSON-RPC 2.0
- Methods:
  - initialize - Initialize MCP session
  - tools/list - Get available tools
  - tools/call - Execute a tool
POST /mcp - Legacy Custom Protocol (Optional)
- Simple {"tool": "...", "arguments": {...}} format
- For development and testing convenience

MCP Protocol Flow

1. Client connects to GET /mcp
   └─> Server responds with text/event-stream
       └─> Sends endpoint event: {"uri": "/mcp/messages"}

2. Client sends JSON-RPC to POST /mcp/messages
   Request:  {"jsonrpc":"2.0","id":1,"method":"initialize","params":{...}}
   Response: {"jsonrpc":"2.0","id":1,"result":{...}}

3. Client lists tools
   Request:  {"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}
   Response: {"jsonrpc":"2.0","id":2,"result":{"tools":[...]}}

4. Client calls a tool
   Request:  {"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"tc_extract","arguments":{...}}}
   Response: {"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"..."}]}}

Installation

Prerequisites

Python 3.9+
pip

Setup

Clone or download the project:

cd toolchecker-mcp

Create a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install dependencies:

pip install -r requirements.txt

Configure environment variables:

cp .env.example .env

Edit .env with your settings:

# For development (mock mode)
MOCK_MODE=true

# For production (with real Core API)
MOCK_MODE=false
CORE_API_BASE_URL=https://api.toolchecker.example.com
CORE_API_KEY=your_actual_api_key_here

# Server configuration
HOST=0.0.0.0
PORT=8000
LOG_LEVEL=INFO

Running the Server

Development Mode (Mock)

# Activate virtual environment
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Run server
python -m app.main

Or using uvicorn directly (recommended for production and ngrok):

uvicorn app.main:app --reload --port 8000 --proxy-headers --forwarded-allow-ips="*"

Options explained:

--proxy-headers: Enable proxy header support (required for ngrok and reverse proxies)
--forwarded-allow-ips="*": Trust all forwarded IPs (use specific IPs in production)

The server will start at http://localhost:8000

Production Mode

Set MOCK_MODE=false in .env
Configure CORE_API_BASE_URL and CORE_API_KEY
Run with production settings:

uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 4

Testing the Server

Health Check

curl http://localhost:8000/health

Expected response:

{
  "status": "healthy",
  "mock_mode": true,
  "core_api_url": "N/A (mock mode)"
}

Test MCP Protocol (Recommended)

1. Test SSE Endpoint

curl -i -N -H "Accept: text/event-stream" http://localhost:8000/mcp

Expected response:

HTTP/1.1 200 OK
content-type: text/event-stream; charset=utf-8
cache-control: no-cache

event: endpoint
data: {"jsonrpc": "2.0", "method": "endpoint", "params": {"uri": "/mcp/messages"}}

event: ping
data: {}

2. Test Initialize

curl -X POST http://localhost:8000/mcp/messages \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "initialize",
    "params": {
      "protocolVersion": "2024-11-05",
      "clientInfo": {"name": "test-client", "version": "1.0.0"}
    }
  }'

3. Test Tools List

curl -X POST http://localhost:8000/mcp/messages \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'

4. Test Tool Call (tc_extract)

curl -X POST http://localhost:8000/mcp/messages \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 3,
    "method": "tools/call",
    "params": {
      "name": "tc_extract",
      "arguments": {"query": "Compare ChatGPT and Claude"}
    }
  }'

Test tc_extract Tool

curl -X POST http://localhost:8000/mcp \
  -H "Content-Type: application/json" \
  -d '{
    "tool": "tc_extract",
    "arguments": {
      "query": "Compare ChatGPT and Claude for content writing",
      "context": {
        "use_case": "tool_comparison",
        "output_format": "notion_1page"
      }
    }
  }'

Test tc_sources Tool

curl -X POST http://localhost:8000/mcp \
  -H "Content-Type: application/json" \
  -d '{
    "tool": "tc_sources",
    "arguments": {
      "comparison_spec": {
        "tools": ["ChatGPT", "Claude"],
        "decision_criteria": ["Pricing", "Features"],
        "followup_questions": [],
        "assumptions": []
      },
      "source_policy": "official_first"
    }
  }'

Test tc_report Tool

curl -X POST http://localhost:8000/mcp \
  -H "Content-Type: application/json" \
  -d '{
    "tool": "tc_report",
    "arguments": {
      "comparison_spec": {
        "tools": ["ChatGPT", "Claude"],
        "decision_criteria": ["Pricing", "Features"],
        "followup_questions": [],
        "assumptions": []
      },
      "sources": [
        {
          "title": "ChatGPT Pricing",
          "url": "https://openai.com/pricing",
          "publisher": "OpenAI",
          "type": "pricing",
          "accessed_at": "2026-01-17T10:00:00Z",
          "excerpt": "ChatGPT pricing information",
          "relevance": 0.9
        }
      ],
      "report_style": "notion_1page"
    }
  }'

Running Tests

# Activate virtual environment
source venv/bin/activate

# Run all tests
pytest

# Run with coverage
pytest --cov=app --cov-report=html

# Run specific test file
pytest tests/test_mock_service.py -v

Test coverage includes:

✓ Mock service extract returns valid output
✓ Mock service sources returns valid schema-compliant sources
✓ Mock service report generates markdown
✓ API endpoint health checks
✓ MCP endpoint tool invocation
✓ Error handling and validation

Connecting to ChatGPT Developer Mode

Step 1: Start Your Local Server

python -m app.main

The server will start at http://localhost:8000

Step 2: Expose Server via ngrok (for testing)

Install and configure ngrok:

# Install ngrok (Mac)
brew install ngrok/ngrok/ngrok

# Configure authtoken (get from https://dashboard.ngrok.com/get-started/your-authtoken)
ngrok config add-authtoken YOUR_TOKEN_HERE

# Start tunnel
ngrok http 8000

You'll see output like:

Forwarding  https://7917866fb318.ngrok-free.app -> http://localhost:8000

Step 3: Verify MCP Endpoint

Test the SSE endpoint:

# Local
curl -i -N -H "Accept: text/event-stream" http://localhost:8000/mcp

# Via ngrok
curl -i -N -H "Accept: text/event-stream" https://YOUR-NGROK-URL.ngrok-free.app/mcp

You should see:

HTTP/1.1 200 OK
content-type: text/event-stream; charset=utf-8

event: endpoint
data: {"jsonrpc": "2.0", "method": "endpoint", "params": {"uri": "/mcp/messages"}}

Step 4: Configure ChatGPT Developer Mode

Open ChatGPT and enable Developer Mode
Add MCP Server with the following URL:

For ngrok tunnel:
```
https://YOUR-NGROK-URL.ngrok-free.app/mcp
```
For production deployment:
```
https://your-domain.com/mcp
```
ChatGPT will:
- Connect to GET /mcp (SSE stream)
- Call POST /mcp/messages with initialize method
- Call tools/list to discover available tools
- Make the 3 tools available: tc_extract, tc_sources, tc_report

Step 5: Use Tools in ChatGPT

You can now ask ChatGPT to use the tools:

"Use tc_extract to compare ChatGPT and Claude"
"Gather sources for comparing Notion and Confluence"
"Generate a comparison report for Slack vs Microsoft Teams"

Important Notes

ngrok free tier: URL changes on each session (consider paid plan for static URL)
CORS: Already configured to allow all origins (restrict in production)
Mock Mode: Default mode uses dummy data - set MOCK_MODE=false for production
Monitoring: View requests at http://localhost:4040 (ngrok web interface)

Troubleshooting ChatGPT Developer Mode Connection

Common Issues and Solutions

Issue 1: "Expected Content-Type text/event-stream, got application/json"

Cause: ChatGPT is connecting to the wrong endpoint or server is not responding with SSE.

Solution:

# Test SSE endpoint
curl -i -N -H "Accept: text/event-stream" http://localhost:8000/mcp

# Should return:
# HTTP/1.1 200 OK
# content-type: text/event-stream; charset=utf-8

Issue 2: POST /mcp returns 500 Internal Server Error

Cause: Server may have syntax errors or missing dependencies.

Solution:

# Check server logs (set LOG_LEVEL=DEBUG in .env)
tail -f /tmp/mcp_server.log

# Verify syntax
python3 -m py_compile app/main.py

# Restart server
./run.sh

Issue 3: Requests to URL-encoded paths like /%7B%22jsonrpc%22...

Cause: Client is sending JSON-RPC data in URL instead of request body.

Server Response: The server now detects this and returns a helpful error:

{
  "error": "It appears you sent JSON-RPC data in the URL path...",
  "hint": "Use POST /mcp/messages with JSON in request body"
}

Debug Mode

Enable debug logging to see full request/response details:

# In .env file
LOG_LEVEL=DEBUG

# Restart server
./run.sh

Debug logs include:

Full request body (first 500 chars)
Stack traces for all errors
Detailed JSON-RPC routing information
Malformed path detection

Testing Endpoint Compatibility

Test 1: SSE Stream

curl -i -N -H "Accept: text/event-stream" https://YOUR-NGROK-URL.ngrok-free.app/mcp

Expected: Content-Type: text/event-stream with event: endpoint message.

Test 2: JSON-RPC Initialize

curl -X POST https://YOUR-NGROK-URL.ngrok-free.app/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}'

Expected: JSON response with result.protocolVersion and result.serverInfo.

Test 3: Tools List

curl -X POST https://YOUR-NGROK-URL.ngrok-free.app/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'

Expected: JSON response with result.tools array containing 3 tools.

Supported Endpoints Summary

Endpoint	Method	Content-Type	Purpose
`/mcp`	GET	text/event-stream	SSE connection for MCP
`/mcp`	POST	application/json	Hybrid: JSON-RPC or custom format
`/mcp/messages`	POST	application/json	JSON-RPC only
`/health`	GET	application/json	Health check

Protocol Detection

The POST /mcp endpoint automatically detects the request format:

If body contains "jsonrpc" field: Routes to JSON-RPC handler
Otherwise: Routes to legacy custom format handler

This allows both ChatGPT (JSON-RPC) and custom clients to use the same endpoint.

Project Structure

toolchecker-mcp/
├── app/
│   ├── __init__.py
│   ├── main.py              # FastAPI app and MCP endpoint
│   ├── config.py            # Settings and configuration
│   ├── models/
│   │   ├── __init__.py
│   │   └── schemas.py       # Pydantic models for all tools
│   ├── services/
│   │   ├── __init__.py
│   │   ├── core_api_client.py   # Core API client with retry logic
│   │   └── mock_service.py      # Mock service for development
│   └── utils/
│       ├── __init__.py
│       ├── logger.py        # Logging configuration
│       └── request_id.py    # Request ID generation
├── tests/
│   ├── __init__.py
│   ├── conftest.py
│   ├── test_mock_service.py
│   └── test_api.py
├── .env.example
├── .gitignore
├── pytest.ini
├── requirements.txt
└── README.md

API Reference

MCP Request Format

{
  "tool": "tc_extract | tc_sources | tc_report",
  "arguments": { /* tool-specific */ },
  "request_id": "optional_custom_id"
}

MCP Response Format

Success:

{
  "success": true,
  "data": { /* tool-specific output */ },
  "request_id": "tc_abc123...",
  "error": null,
  "debug_code": null
}

Error:

{
  "success": false,
  "data": null,
  "error": "User-friendly error message",
  "request_id": "tc_abc123...",
  "debug_code": "ERROR_CODE"
}

Tool Schemas

tc_extract

Input:

{
  "query": "string (1-5000 chars)",
  "context": {
    "use_case": "tool_comparison",
    "output_format": "notion_1page",
    "constraints": {
      "budget": "string|null",
      "team_size": "number|null",
      "region": "string|null",
      "must_have": ["string"]
    }
  }
}

Output:

{
  "comparison_spec": {
    "tools": ["string"],
    "decision_criteria": ["string"],
    "followup_questions": ["string"],
    "assumptions": ["string"]
  }
}

tc_sources

Input:

{
  "comparison_spec": { /* from tc_extract */ },
  "source_policy": "official_first | balanced"
}

Output:

{
  "sources": [
    {
      "title": "string",
      "url": "string",
      "publisher": "string|null",
      "type": "official|docs|pricing|policy|benchmark|news|review",
      "accessed_at": "ISO datetime",
      "excerpt": "string",
      "relevance": 0.0-1.0,
      "notes": "string|null"
    }
  ],
  "coverage": {
    "criteria_covered": ["string"],
    "missing": ["string"]
  }
}

tc_report

Input:

{
  "comparison_spec": { /* from tc_extract */ },
  "sources": [ /* from tc_sources */ ],
  "report_style": "notion_1page"
}

Output:

{
  "report": {
    "title": "string",
    "version": "TC v0.1",
    "created_at": "ISO datetime",
    "summary_5lines": ["string (1-5 items)"],
    "comparison_table": "markdown table",
    "claims": [
      {
        "claim": "string",
        "evidence": { "url": "string", "excerpt": "string" },
        "confidence": "high|medium|low"
      }
    ],
    "risks": ["string"],
    "decision_guide": ["string"],
    "sources_list": [
      { "url": "string", "accessed_at": "ISO datetime" }
    ]
  },
  "rendered_markdown": "string (Notion-friendly)"
}

Configuration

Environment Variables

Variable	Default	Description
`MOCK_MODE`	`true`	Use mock service instead of Core API
`CORE_API_BASE_URL`	-	Base URL for Core API
`CORE_API_KEY`	-	API key for Core API authentication
`HOST`	`0.0.0.0`	Server bind host
`PORT`	`8000`	Server port
`LOG_LEVEL`	`INFO`	Logging level (DEBUG/INFO/WARNING/ERROR)
`REQUEST_TIMEOUT`	`30`	HTTP request timeout in seconds
`MAX_RETRIES`	`2`	Maximum retry attempts for failed requests
`RETRY_BACKOFF_FACTOR`	`2.0`	Exponential backoff multiplier
`MAX_QUERY_LOG_LENGTH`	`200`	Maximum query length in logs

Deployment

Using Docker (Optional)

Create Dockerfile:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/

EXPOSE 8000

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Build and run:

docker build -t toolchecker-mcp .
docker run -p 8000:8000 --env-file .env toolchecker-mcp

Production Checklist

Set MOCK_MODE=false
Configure valid CORE_API_BASE_URL and CORE_API_KEY
Use HTTPS (reverse proxy with nginx/caddy)
Set appropriate CORS origins in production
Configure log aggregation
Set up monitoring and alerting
Use multiple workers: --workers 4
Configure rate limiting
Set up health check monitoring

Troubleshooting

Server won't start

Check Python version: python --version (requires 3.9+)
Verify all dependencies installed: pip install -r requirements.txt
Check port availability: lsof -i :8000

Tests failing

Ensure virtual environment is activated
Run: pip install -r requirements.txt
Check test environment: pytest --version

Core API connection errors

Verify CORE_API_BASE_URL is correct
Check CORE_API_KEY is valid
Test network connectivity: curl $CORE_API_BASE_URL/health
Check logs for detailed error messages

Mock mode not working

Verify MOCK_MODE=true in .env
Restart server after changing .env
Check logs for mock service activation

License

MIT

Support

For issues and questions:

GitHub Issues: [Create an issue]
Documentation: See this README
Logs: Check server output for detailed error messages with request IDs

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
app		app
tests		tests
.env.example		.env.example
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CHATGPT_INTEGRATION.md		CHATGPT_INTEGRATION.md
DEPLOYMENT_GUIDE.md		DEPLOYMENT_GUIDE.md
MCP_PROTOCOL.md		MCP_PROTOCOL.md
PROJECT_STRUCTURE.md		PROJECT_STRUCTURE.md
QUICKSTART.md		QUICKSTART.md
README.md		README.md
STATUS.md		STATUS.md
SUMMARY.md		SUMMARY.md
TREE.txt		TREE.txt
pytest.ini		pytest.ini
requirements.txt		requirements.txt
run.bat		run.bat
run.sh		run.sh
test.sh		test.sh

Folders and files

Latest commit

History

Repository files navigation

ToolChecker MCP Server

Features

Architecture

MCP Protocol Support

Endpoints

MCP Protocol Flow

Installation

Prerequisites

Setup

Running the Server

Development Mode (Mock)

Production Mode

Testing the Server

Health Check

Test MCP Protocol (Recommended)

1. Test SSE Endpoint

2. Test Initialize

3. Test Tools List

4. Test Tool Call (tc_extract)

Test tc_extract Tool

Test tc_sources Tool

Test tc_report Tool

Running Tests

Connecting to ChatGPT Developer Mode

Step 1: Start Your Local Server

Step 2: Expose Server via ngrok (for testing)

Step 3: Verify MCP Endpoint

Step 4: Configure ChatGPT Developer Mode

Step 5: Use Tools in ChatGPT

Important Notes

Troubleshooting ChatGPT Developer Mode Connection

Common Issues and Solutions

Issue 1: "Expected Content-Type text/event-stream, got application/json"

Issue 2: POST /mcp returns 500 Internal Server Error

Issue 3: Requests to URL-encoded paths like /%7B%22jsonrpc%22...

Debug Mode

Testing Endpoint Compatibility

Supported Endpoints Summary

Protocol Detection

Project Structure

API Reference

MCP Request Format

MCP Response Format

Tool Schemas

tc_extract

tc_sources

tc_report

Configuration

Environment Variables

Deployment

Using Docker (Optional)

Production Checklist

Troubleshooting

Server won't start

Tests failing

Core API connection errors

Mock mode not working

License

Support

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages