Skip to content

AGENTS.md

Latest commit

 

History

History
114 lines (86 loc) · 7.95 KB

AGENTS.md

File metadata and controls

114 lines (86 loc) · 7.95 KB
Raw

Cache Hunter — AI Agent Guide

Project Overview

Transparent HTTP proxy for vLLM that logs all LLM API traffic to SQLite for cache debugging. Detects vLLM prefix caching behavior by analyzing latency, token patterns, and context coherence across requests.

Stack: TypeScript 5, Node.js (ESM), Express 4, sql.js (SQLite WASM), React 19, Vite 6, Vitest 4

Architecture: OpenFox → Cache Hunter Proxy (port 8787) → vLLM (port 8000). Requests are forwarded transparently; request/response pairs are logged to per-session SQLite DBs. A web UI on port 4000 visualizes sessions and hash grids.

Directory Structure

├── src/                     # Backend source (TypeScript, ESM)
│   ├── index.ts             # Entry point — creates HTTP server, wires ProxyEngine + Express app
│   ├── app.ts               # Express routes: /api/config, /api/proxy/*, /api/capture/*, /api/sessions/*
│   ├── proxy-engine.ts      # ProxyEngine class — manages HTTP server lifecycle, forwards requests
│   ├── proxy.ts             # Legacy createProxyHandler function (older pattern, still tested)
│   ├── logger.ts            # AsyncLogger — batched SQLite writer via sql.js
│   ├── session-manager.ts   # Session CRUD — manifest.json tracks sessions, per-session .db files
│   ├── ws-server.ts         # WebSocket server attached to HTTP server for real-time UI updates
│   ├── hash-grid.ts         # Builds a hash-based grid comparing messages across API calls
│   ├── context-tree.ts      # MD5-based content hashing + cumulative context chain validation
│   ├── context-validator.ts # Prints validation reports from hash grid data
│   ├── parse-api.ts         # Parses /v1/chat/completions and /v1/responses request bodies
│   ├── schema.sql           # SQLite schema (requests + responses tables with indexes)
│   ├── analyze-patterns.ts  # CLI script for cache pattern analysis from a DB
│   ├── visualize-context.ts # CLI script for context coherence visualization
│   ├── hash-tree.ts         # CLI script — builds hash grid + runs validation report
│   ├── diff-prompts.ts      # CLI script — diffs system prompts between calls
│   ├── diff-system-prompts.ts
│   └── compare-system-prompt.ts
├── frontend/                # React SPA
│   ├── src/
│   │   ├── App.tsx          # Main app — proxy controls, session list, hash grid viewer
│   │   ├── main.tsx         # React entry point
│   │   ├── components/
│   │   │   └── HashGrid.tsx # Hash grid table with modals, diff view, tool cards
│   │   ├── hooks/
│   │   │   ├── useApi.ts    # Typed fetch wrapper + API client
│   │   │   └── useWebSocket.ts # Auto-reconnecting WS hook
│   │   └── styles/
│   └── dist/                # Built frontend served by Express
├── data/                    # SQLite databases + manifest.json
├── html-tree/               # Static HTML tree visualization (generated by hash-tree.ts)
└── .openfox/dev.json        # Dev server config

Build, Lint, Test Commands

npm run dev              # Concurrent backend (tsx watch) + frontend (vite)
npm start                # Production-like: backend + frontend
npm run build            # Build frontend only
npm test                 # Vitest in watch mode
npm run test:run         # Vitest single run
npx vitest run src/foo.test.ts  # Single test file
npx vitest run -t "test name"   # Single test by name pattern
npx tsc --noEmit         # Type-check without emitting

There is no linter configured. TypeScript strict mode is the only static check.

Code Conventions

  • ESM everywhere — "type": "module", all imports use .js extension (e.g. import './proxy-engine.js')
  • Strict TypeScript — strict: true in tsconfig
  • Functional helpers — pure functions preferred (e.g., hashContent, parseRequestBody, extractCacheSalt)
  • Classes for stateful services — ProxyEngine (extends EventEmitter), AsyncLogger
  • No semicolons — project convention (all existing code omits them)
  • No JSDoc comments — code is self-documenting with descriptive names
  • Tests co-located — *.test.ts next to source files
  • CLI scripts use #!/usr/bin/env tsx shebang for direct execution

Key Abstractions

ProxyEngine (src/proxy-engine.ts)

Extends EventEmitter. Manages an HTTP server that proxies to vLLM. Emits 'log' events with ProxyResult containing request + response data. Has startCapture()/stopCapture() toggle — only emits logs when capturing. Uses crypto.randomUUID() for correlation IDs.

AsyncLogger (src/logger.ts)

Batches log entries and flushes to SQLite every 100ms or 50 entries. Uses sql.js (SQLite compiled to WASM) entirely in-memory, saving to disk via db.export(). Implements transactional inserts with rollback on failure.

Session Manager (src/session-manager.ts)

File-based session tracking via data/manifest.json. Each capture creates a new .db file. Sessions are active or completed. finalizeSession() reads request count from the DB. getSessionHashGrid() joins requests+responses and builds a hash grid for the UI.

Hash Grid (src/hash-grid.ts)

Core visualization abstraction. Takes an array of completions (messages + tools + path) and produces a 2D grid where rows = messages (+ tools row), columns = API calls. Each cell is a 4-char MD5 hash of the content. Also produces hash_map (hash → content lookup) and lines (formatted for display).

Context Tree (src/context-tree.ts)

MD5-based content hashing (4-char hex). buildContextTree() creates cumulative context chain — each turn's hash depends on all previous turns. Used for coherence validation.

Parse API (src/parse-api.ts)

Normalizes both /v1/chat/completions (OpenAI-style) and /v1/responses (OpenAI Responses API) request bodies into a common { messages, tools } format. Handles SSE token extraction for the Responses API.

WebSocket Server (src/ws-server.ts)

Simple broadcaster — attaches to the HTTP server, sends typed JSON messages to all connected clients. Used by the React frontend for live updates.

Common Pitfalls

  1. sql.js is WASM-based — it's an in-memory database. You MUST call db.export() + writeFileSync() to persist. The AsyncLogger saves after every flush; session-manager opens DBs read-only for queries.

  2. Two proxy implementations coexist — proxy-engine.ts (class-based, current) and proxy.ts (function-based, legacy). Both are tested. New code should use ProxyEngine. The legacy proxy.ts is still imported by integration tests.

  3. SSE streaming — The proxy handles streaming responses by buffering chunks in responseBody while piping to the client. Token extraction for /v1/responses happens from the SSE event stream, not JSON. See extractSSETokenUsage() in parse-api.ts.

  4. Dual API support — Both /v1/chat/completions and /v1/responses endpoints are supported. They have different body formats. Always use parseRequestBody() to normalize. The path field determines parsing strategy.

  5. Hash collisions — 4-char MD5 hex hashes (16-bit) are used for grid visualization. Collisions are theoretically possible but practically unlikely for small-to-medium datasets. The hash_map stores full content for lookup.

  6. Port conflicts — The proxy binds to port 0 in tests (random port), but defaults to 8787 in production. The web server defaults to 4000. Frontend dev server (Vite) runs on 5173 with proxy to 4000.

  7. Model override — ProxyEngine fetches the active model from vLLM's /v1/models endpoint on start and rewrites model fields in proxied requests to match. This ensures consistent model attribution.

  8. No linter — Only TypeScript strict mode catches issues. Run npx tsc --noEmit to type-check. There's no ESLint or Prettier configuration.