diff --git a/README.md b/README.md index f60fefa..d6a12a8 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ A 24-tool MCP server for Claude Code that catches ambiguous instructions before [![npm](https://img.shields.io/npm/v/preflight-dev)](https://www.npmjs.com/package/preflight-dev) [![Node 18+](https://img.shields.io/badge/node-18%2B-brightgreen?logo=node.js&logoColor=white)](https://nodejs.org/) -[Quick Start](#quick-start) · [How It Works](#how-it-works) · [Tool Reference](#tool-reference) · [Configuration](#configuration) · [Scoring](#the-12-category-scorecard) +[Quick Start](#quick-start) · [How It Works](#how-it-works) · [Tool Reference](#tool-reference) · [Usage Examples](examples/USAGE_EXAMPLES.md) · [Configuration](#configuration) · [Scoring](#the-12-category-scorecard) @@ -76,21 +76,36 @@ The pattern is always the same: vague prompt → Claude guesses → wrong output ## Quick Start -### Option A: Claude Code CLI (fastest) +### Option A: npx (zero install — fastest) + +One command, nothing to clone or install: ```bash -claude mcp add preflight -- npx tsx /path/to/preflight/src/index.ts +claude mcp add preflight -- npx -y preflight-dev ``` -With environment variables: +With your project directory (recommended — enables contracts, file search, and context-aware triage): ```bash claude mcp add preflight \ -e CLAUDE_PROJECT_DIR=/path/to/your/project \ - -- npx tsx /path/to/preflight/src/index.ts + -- npx -y preflight-dev +``` + +Restart Claude Code. All 24 tools activate automatically. + +### Option B: npm global install + +Install once, available everywhere: + +```bash +npm install -g preflight-dev +claude mcp add preflight -- preflight-dev ``` -### Option B: Clone & configure manually +### Option C: Clone & configure manually + +Best for contributors or if you want to modify preflight itself: ```bash git clone https://github.com/TerminalGravity/preflight.git @@ -104,9 +119,9 @@ Add to your project's `.mcp.json`: "mcpServers": { "preflight": { "command": "npx", - "args": ["tsx", "/path/to/preflight/src/index.ts"], + "args": ["tsx", "/absolute/path/to/preflight/src/index.ts"], "env": { - "CLAUDE_PROJECT_DIR": "/path/to/your/project" + "CLAUDE_PROJECT_DIR": "/absolute/path/to/your/project" } } } @@ -115,12 +130,7 @@ Add to your project's `.mcp.json`: Restart Claude Code. The tools activate automatically. -### Option C: npm (global) - -```bash -npm install -g preflight-dev -claude mcp add preflight -- preflight-dev -``` +> **Note:** Paths in `.mcp.json` must be absolute — relative paths won't resolve correctly. --- @@ -406,6 +416,12 @@ This prevents the common failure mode: changing a shared type in one service and ## Configuration Reference +> **Want a ready-to-use starting point?** Copy the example configs: +> ```bash +> cp -r examples/.preflight /path/to/your/project/ +> ``` +> See [`examples/.preflight/README.md`](examples/.preflight/README.md) for details. + ### `.preflight/config.yml` Drop this in your project root. Every field is optional — defaults are sensible. @@ -600,6 +616,78 @@ src/ └── ... # One file per tool ``` +## Troubleshooting + +### Tools don't show up in Claude Code + +**Symptom:** You added the MCP config but Claude doesn't see any preflight tools. + +1. Make sure you restarted Claude Code after editing `.mcp.json` +2. Check the path in your config is absolute, not relative — `npx tsx /Users/you/preflight/src/index.ts` +3. Run the server directly to check for startup errors: + ```bash + npx tsx /path/to/preflight/src/index.ts + ``` + If it crashes on startup, the error will tell you what's missing. + +### LanceDB / timeline search not working + +**Symptom:** `search_timeline` returns empty results or errors about the database. + +- LanceDB stores data in `~/.preflight/projects//timeline.lance/` +- You need to **ingest sessions first** — run `preflight_onboard_project` with your project dir, or use the CLI: `preflight-dev init` +- If you get native module errors, make sure your Node version matches your OS architecture (especially on Apple Silicon — don't use x64 Node via Rosetta) +- To reset a corrupt database, delete the `.lance` directory and re-ingest: + ```bash + rm -rf ~/.preflight/projects/YOUR_PROJECT/timeline.lance + ``` + +### `CLAUDE_PROJECT_DIR` not set + +**Symptom:** Tools that need project context (contracts, file search) return nothing useful. + +Set it in your `.mcp.json` env block: +```json +"env": { + "CLAUDE_PROJECT_DIR": "/absolute/path/to/your/project" +} +``` +Or export it before running Claude Code: +```bash +export CLAUDE_PROJECT_DIR=/path/to/your/project +claude +``` + +### `preflight_check` says everything is "TRIVIAL" + +This is by design for short, unambiguous commands like `git status` or `ls`. The triage engine only flags prompts that are genuinely ambiguous. If you want stricter checking, add keywords to `always_check` in `.preflight/triage.yml`: + +```yaml +always_check: + - refactor + - update + - change +``` + +### npm global install: `preflight-dev: command not found` + +After `npm install -g preflight-dev`, your shell may not see the new binary. Try: +```bash +# Check where npm puts global bins +npm bin -g +# Make sure that directory is in your PATH +export PATH="$(npm bin -g):$PATH" +``` + +### High memory usage during session ingestion + +Large JSONL session files (100MB+) can spike memory. Set `NODE_OPTIONS` to increase the heap: +```bash +NODE_OPTIONS="--max-old-space-size=4096" npx tsx src/index.ts +``` + +--- + ## License MIT — do whatever you want with it. diff --git a/examples/.preflight/README.md b/examples/.preflight/README.md new file mode 100644 index 0000000..b8aaf8c --- /dev/null +++ b/examples/.preflight/README.md @@ -0,0 +1,25 @@ +# `.preflight/` Example Config + +Copy this directory into your project root to configure preflight: + +```bash +cp -r examples/.preflight /path/to/your/project/ +``` + +## Files + +| File | Purpose | +|------|---------| +| `config.yml` | Main config — profile, related projects, thresholds, embeddings | +| `triage.yml` | Triage rules — which keywords trigger which classification level | +| `contracts/*.yml` | Manual contract definitions — supplement auto-extraction | + +## Quick Setup + +1. Copy the directory: `cp -r examples/.preflight ./` +2. Edit `config.yml` — set your `related_projects` paths +3. Edit `triage.yml` — add your domain-specific keywords to `always_check` +4. Optionally add contracts in `contracts/` for planned or external APIs +5. Commit `.preflight/` to your repo so your team shares the same config + +All fields are optional. Defaults work well out of the box — only customize what you need. diff --git a/examples/.preflight/config.yml b/examples/.preflight/config.yml new file mode 100644 index 0000000..0ad12e8 --- /dev/null +++ b/examples/.preflight/config.yml @@ -0,0 +1,29 @@ +# .preflight/config.yml — drop this in your project root +# All fields are optional. Defaults are sensible. +# See: https://github.com/TerminalGravity/preflight#configuration-reference + +# Profile controls overall verbosity +# "minimal" — only flag ambiguous+, skip clarification detail +# "standard" — default behavior +# "full" — maximum detail on every non-trivial prompt +profile: standard + +# Related projects for cross-service awareness +# Preflight will search these projects' indexes when your prompt +# touches shared contracts (types, routes, schemas). +related_projects: + # - path: /absolute/path/to/auth-service + # alias: auth-service + # - path: /absolute/path/to/shared-types + # alias: shared-types + +# Behavioral thresholds +thresholds: + session_stale_minutes: 30 # warn if no activity for this long + max_tool_calls_before_checkpoint: 100 # suggest checkpoint after N tool calls + correction_pattern_threshold: 3 # min corrections before forming a pattern + +# Embedding configuration +embeddings: + provider: local # "local" (Xenova, zero config) or "openai" + # openai_api_key: sk-... # only needed if provider is "openai" diff --git a/examples/.preflight/contracts/api.yml b/examples/.preflight/contracts/api.yml new file mode 100644 index 0000000..754c5da --- /dev/null +++ b/examples/.preflight/contracts/api.yml @@ -0,0 +1,47 @@ +# .preflight/contracts/api.yml — manual contract definitions +# These supplement auto-extracted contracts from your codebase. +# Manual definitions win on name conflicts with auto-extracted ones. +# +# Use this when: +# - You have contracts that aren't in code yet (planned APIs) +# - Auto-extraction misses something important +# - You want to document cross-service agreements explicitly + +- name: User + kind: interface + description: Core user object shared across services + fields: + - name: id + type: string + required: true + - name: email + type: string + required: true + - name: role + type: "'admin' | 'member' | 'viewer'" + required: true + - name: createdAt + type: Date + required: true + +- name: "POST /api/users" + kind: route + description: Create a new user account + fields: + - name: body + type: "{ email: string, role: string }" + required: true + - name: response + type: "{ user: User, token: string }" + required: true + +- name: "GET /api/users/:id" + kind: route + description: Fetch user by ID + fields: + - name: params + type: "{ id: string }" + required: true + - name: response + type: User + required: true diff --git a/examples/.preflight/triage.yml b/examples/.preflight/triage.yml new file mode 100644 index 0000000..22b05d3 --- /dev/null +++ b/examples/.preflight/triage.yml @@ -0,0 +1,38 @@ +# .preflight/triage.yml — controls the triage classification engine +# Customize which prompts get flagged, skipped, or escalated. + +rules: + # Prompts containing these words → always at least AMBIGUOUS + # Add domain terms that are too vague without context + always_check: + - rewards + - permissions + - migration + - schema + # - billing # uncomment for your domain + # - onboarding + + # Prompts containing these words → TRIVIAL (pass through immediately) + # Common low-risk commands that don't need analysis + skip: + - commit + - format + - lint + - "git status" + - "git log" + + # Prompts containing these words → CROSS-SERVICE + # Triggers search across related_projects defined in config.yml + cross_service_keywords: + - auth + - notification + - event + - webhook + # - payment + # - analytics + +# How aggressively to classify +# "relaxed" — more prompts pass as clear (faster, less interruption) +# "standard" — balanced (recommended) +# "strict" — more prompts flagged as ambiguous (thorough, more interruptions) +strictness: standard diff --git a/examples/USAGE_EXAMPLES.md b/examples/USAGE_EXAMPLES.md new file mode 100644 index 0000000..55a2100 --- /dev/null +++ b/examples/USAGE_EXAMPLES.md @@ -0,0 +1,205 @@ +# Usage Examples + +Real-world examples of preflight tools in action inside Claude Code. + +--- + +## 1. Catching a Vague Prompt with `preflight_check` + +You type this into Claude Code: + +``` +fix the login bug +``` + +Preflight intercepts and returns: + +``` +⚠️ Ambiguity detected — routing to clarify_intent + +I found 3 potential login issues: + 1. JWT token not refreshing on expiry (src/auth/jwt.ts:42) + 2. OAuth callback returning 404 after redirect (src/auth/oauth.ts:118) + 3. "Remember me" checkbox not persisting session (src/components/LoginForm.tsx:23) + +Which one? Or describe the symptoms and I'll narrow it down. +``` + +Instead of Claude guessing and editing the wrong file, you get a targeted question. One round-trip instead of three. + +--- + +## 2. Scoping Work Before Coding with `scope_work` + +``` +> scope_work "Add rate limiting to the /api/upload endpoint" + +📋 Execution Plan +───────────────── +Goal: Add rate limiting to POST /api/upload + +Files to touch: + - src/api/routes/upload.ts (add middleware) + - src/middleware/rateLimit.ts (create) + - src/config/limits.ts (add upload limits) + - tests/api/upload.test.ts (add rate limit tests) + +Dependencies: + - Check if express-rate-limit is already installed + - Review existing middleware chain in src/api/index.ts + +Risks: + - Existing upload tests may break if rate limit hits in CI + - Need to decide: per-IP or per-user limiting? + +Estimated scope: ~120 lines across 4 files +``` + +--- + +## 3. Enriching a Sub-Agent Task with `enrich_agent_task` + +You want to spawn a sub-agent for a refactor: + +``` +> enrich_agent_task "refactor the database queries to use the new ORM" + +📦 Enriched Task Brief +────────────────────── +Original: "refactor the database queries to use the new ORM" + +Context added: + - ORM: Drizzle (see drizzle.config.ts, src/db/schema.ts) + - 14 files with raw SQL queries (grep results attached) + - Existing patterns: src/db/queries/users.ts uses query builder already + - Related contracts: UserRecord, PostRecord types in src/types/db.ts + - Test coverage: 8/14 query files have corresponding tests + +Suggested task for sub-agent: + "Refactor raw SQL queries in src/db/queries/ to use Drizzle ORM. + Follow the pattern in src/db/queries/users.ts. Schema is in + src/db/schema.ts. Run `npm test -- --grep db` after each file. + 14 files total — see list below." +``` + +The sub-agent gets everything it needs instead of fumbling through the codebase. + +--- + +## 4. Resolving Vague Follow-ups with `sharpen_followup` + +After Claude fixes a bug, you say: + +``` +> do the same for the others +``` + +Without preflight, Claude guesses what "the others" means. With it: + +``` +> sharpen_followup "do the same for the others" + +🔍 Resolved "the others" from conversation context: + +You just fixed a missing null check in src/api/users.ts:34. +Similar pattern found in 4 other files: + 1. src/api/posts.ts:28 — same nullable field access + 2. src/api/comments.ts:41 — same pattern + 3. src/api/teams.ts:55 — same pattern + 4. src/api/billing.ts:19 — slightly different (optional chain exists but incomplete) + +Shall I apply the fix to all 4? +``` + +--- + +## 5. Checking Session Health with `check_session_health` + +``` +> check_session_health + +🏥 Session Health +───────────────── +⏱ Session duration: 47 min +📝 Uncommitted files: 8 (⚠️ commit soon) +🔄 Last commit: 38 min ago (⚠️ overdue) +💬 Turn count: 23 +📊 Context usage: ~62% (healthy) +🎯 Focus score: B (2 directory switches) + +Recommendation: Commit your current changes before continuing. +Run `checkpoint` to save state. +``` + +--- + +## 6. Searching Past Sessions with `search_history` + +``` +> search_history "how did we handle the Stripe webhook retry logic" + +🔍 Found 3 relevant results across 2 sessions: + +Session 2024-12-15 (confidence: 0.92) + "Implemented exponential backoff for Stripe webhook retries + in src/webhooks/stripe.ts. Uses 3 retries with 1s/4s/16s delays. + Idempotency key stored in Redis." + +Session 2024-11-28 (confidence: 0.78) + "Discussed webhook retry strategies. Decided against queue-based + approach due to infrastructure complexity. Went with in-process + retry with circuit breaker." +``` + +Never re-research decisions you already made. + +--- + +## 7. Getting a Scorecard with `generate_scorecard` + +``` +> generate_scorecard type=trend period=week + +📊 Weekly Trend — Jan 20–26, 2025 +────────────────────────────────── +Overall: B+ (78/100) ↑ from B (74) last week + +Improving: + ✅ Clarification: 82 → 89 (+7) — more file refs in prompts + ✅ Verification: 65 → 78 (+13) — running tests before done + +Declining: + ⚠️ Compaction Mgmt: 71 → 58 (-13) — 3 compactions without prior commit + ⚠️ Session Lifecycle: 80 → 72 (-8) — longer gaps between commits + +Top tip: Run `checkpoint` before long exploratory conversations +to avoid losing work on compaction. +``` + +--- + +## 8. Quick Prompt Grading with `prompt_score` + +``` +> prompt_score "update the thing" + +📝 Prompt Score: D (32/100) +─────────────────────────── +Specificity: F — no file, function, or component named +Scope: F — "the thing" is completely ambiguous +Actionability: D — "update" is vague (add? change? remove?) +Done-condition: F — no way to verify completion + +Try: "Update the user avatar upload handler in src/api/users.ts +to accept WebP format and add a test case" → A (94/100) +``` + +--- + +## Tips + +- **Start every session** with `preflight_check` on your first prompt — it'll route to the right tool automatically +- **Use `checkpoint`** every 20–30 minutes or before exploring a new direction +- **Run `search_history`** before implementing something — you may have solved it before +- **Check `session_stats`** at the end of a session to spot waste patterns +- **Set up `.preflight/config.yml`** to tune thresholds for your team (see [examples/.preflight/](/.preflight/)) diff --git a/src/lib/git.ts b/src/lib/git.ts index a32ee3c..1066b2a 100644 --- a/src/lib/git.ts +++ b/src/lib/git.ts @@ -1,4 +1,4 @@ -import { execFileSync } from "child_process"; +import { execFileSync, execSync } from "child_process"; import { PROJECT_DIR } from "./files.js"; import type { RunError } from "../types.js"; @@ -35,6 +35,31 @@ function gitCmd(cmdStr: string, opts?: { timeout?: number }): string { return run(cmdStr.split(/\s+/), opts); } +/** + * Run a shell command string (supports pipes, redirects, ||, &&). + * Use only with hardcoded command strings — never with user input. + * Returns stdout on success, descriptive error string on failure. + */ +export function shellRun(cmd: string, opts: { timeout?: number } = {}): string { + try { + return execSync(cmd, { + cwd: PROJECT_DIR, + encoding: "utf-8", + timeout: opts.timeout || 10000, + maxBuffer: 1024 * 1024, + stdio: ["pipe", "pipe", "pipe"], + }).trim(); + } catch (e: any) { + const timedOut = e.killed === true || e.signal === "SIGTERM"; + if (timedOut) { + return `[timed out after ${opts.timeout || 10000}ms]`; + } + const output = e.stdout?.trim() || e.stderr?.trim(); + if (output) return output; + return `[command failed: ${cmd} (exit ${e.status ?? "?"})]`; + } +} + /** Get the current branch name. */ export function getBranch(): string { return run(["branch", "--show-current"]); diff --git a/src/tools/audit-workspace.ts b/src/tools/audit-workspace.ts index d4306bd..471c2c0 100644 --- a/src/tools/audit-workspace.ts +++ b/src/tools/audit-workspace.ts @@ -1,5 +1,5 @@ import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; -import { run } from "../lib/git.js"; +import { run, shellRun } from "../lib/git.js"; import { readIfExists, findWorkspaceDocs } from "../lib/files.js"; /** Extract top-level work areas from file paths generically */ @@ -36,7 +36,7 @@ export function registerAuditWorkspace(server: McpServer): void { {}, async () => { const docs = findWorkspaceDocs(); - const recentFiles = run("git diff --name-only HEAD~10 2>/dev/null || echo ''").split("\n").filter(Boolean); + const recentFiles = shellRun("git diff --name-only HEAD~10 2>/dev/null || echo ''").split("\n").filter(Boolean); const sections: string[] = []; // Doc freshness @@ -75,7 +75,7 @@ export function registerAuditWorkspace(server: McpServer): void { // Check for gap trackers or similar tracking docs const trackingDocs = Object.entries(docs).filter(([n]) => /gap|track|progress/i.test(n)); if (trackingDocs.length > 0) { - const testFilesCount = parseInt(run("find tests -name '*.spec.ts' -o -name '*.test.ts' 2>/dev/null | wc -l").trim()) || 0; + const testFilesCount = parseInt(shellRun("find tests -name '*.spec.ts' -o -name '*.test.ts' 2>/dev/null | wc -l").trim()) || 0; sections.push(`## Tracking Docs\n${trackingDocs.map(([n]) => { const age = docStatus.find(d => d.name === n)?.ageHours ?? "?"; return `- .claude/${n} — last updated ${age}h ago`; diff --git a/src/tools/checkpoint.ts b/src/tools/checkpoint.ts index e086f01..3b6a795 100644 --- a/src/tools/checkpoint.ts +++ b/src/tools/checkpoint.ts @@ -2,7 +2,7 @@ import { z } from "zod"; import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; import { writeFileSync, existsSync, mkdirSync } from "fs"; import { join, dirname } from "path"; -import { run, getBranch, getStatus, getLastCommit, getStagedFiles } from "../lib/git.js"; +import { run, shellRun, getBranch, getStatus, getLastCommit, getStagedFiles } from "../lib/git.js"; import { PROJECT_DIR } from "../lib/files.js"; import { appendLog, now } from "../lib/state.js"; @@ -88,7 +88,7 @@ ${dirty || "clean"} const result = run(`${addCmd} && git commit -m "${commitMsg.replace(/"/g, '\\"')}" 2>&1`); if (result.includes("commit failed") || result.includes("nothing to commit")) { // Rollback: unstage if commit failed - run("git reset HEAD 2>/dev/null"); + shellRun("git reset HEAD 2>/dev/null"); commitResult = `commit failed: ${result}`; } else { commitResult = result; diff --git a/src/tools/clarify-intent.ts b/src/tools/clarify-intent.ts index 32efa3a..3ef2bb5 100644 --- a/src/tools/clarify-intent.ts +++ b/src/tools/clarify-intent.ts @@ -1,6 +1,6 @@ import { z } from "zod"; import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; -import { run, getBranch, getStatus, getRecentCommits, getDiffFiles, getStagedFiles } from "../lib/git.js"; +import { run, shellRun, getBranch, getStatus, getRecentCommits, getDiffFiles, getStagedFiles } from "../lib/git.js"; import { findWorkspaceDocs, PROJECT_DIR } from "../lib/files.js"; import { searchSemantic } from "../lib/timeline-db.js"; import { getRelatedProjects } from "../lib/config.js"; @@ -152,10 +152,10 @@ export function registerClarifyIntent(server: McpServer): void { let hasTestFailures = false; if (!area || area.includes("test") || area.includes("fix") || area.includes("ui") || area.includes("api")) { - const typeErrors = run("pnpm tsc --noEmit 2>&1 | grep -c 'error TS' || echo '0'"); + const typeErrors = shellRun("pnpm tsc --noEmit 2>&1 | grep -c 'error TS' || echo '0'"); hasTypeErrors = parseInt(typeErrors, 10) > 0; - const testFiles = run("find tests -name '*.spec.ts' -maxdepth 4 2>/dev/null | head -20"); + const testFiles = shellRun("find tests -name '*.spec.ts' -maxdepth 4 2>/dev/null | head -20"); const failingTests = getTestFailures(); hasTestFailures = failingTests !== "all passing" && failingTests !== "no test report found"; diff --git a/src/tools/enrich-agent-task.ts b/src/tools/enrich-agent-task.ts index 236edfa..8b5e594 100644 --- a/src/tools/enrich-agent-task.ts +++ b/src/tools/enrich-agent-task.ts @@ -1,6 +1,6 @@ import { z } from "zod"; import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; -import { run, getDiffFiles } from "../lib/git.js"; +import { run, shellRun, getDiffFiles } from "../lib/git.js"; import { PROJECT_DIR } from "../lib/files.js"; import { getConfig, type RelatedProject } from "../lib/config.js"; import { existsSync, readFileSync } from "fs"; @@ -42,11 +42,11 @@ function findAreaFiles(area: string): string { /** Find related test files for an area */ function findRelatedTests(area: string): string { - if (!area) return run("git ls-files 2>/dev/null | grep -E '\\.(spec|test)\\.(ts|tsx|js|jsx)$' | head -10"); + if (!area) return shellRun("git ls-files 2>/dev/null | grep -E '\\.(spec|test)\\.(ts|tsx|js|jsx)$' | head -10"); const safeArea = shellEscape(area.split(/\s+/)[0]); const tests = run(`git ls-files 2>/dev/null | grep -E '\\.(spec|test)\\.(ts|tsx|js|jsx)$' | grep -i '${safeArea}' | head -10`); - return tests || run("git ls-files 2>/dev/null | grep -E '\\.(spec|test)\\.(ts|tsx|js|jsx)$' | head -10"); + return tests || shellRun("git ls-files 2>/dev/null | grep -E '\\.(spec|test)\\.(ts|tsx|js|jsx)$' | head -10"); } /** Get an example pattern from the first matching file */ diff --git a/src/tools/prompt-score.ts b/src/tools/prompt-score.ts index 1cecf01..9e91532 100644 --- a/src/tools/prompt-score.ts +++ b/src/tools/prompt-score.ts @@ -40,7 +40,7 @@ interface ScoreResult { feedback: string[]; } -function scorePrompt(text: string): ScoreResult { +export function scorePrompt(text: string): ScoreResult { const feedback: string[] = []; let specificity: number; let scope: number; diff --git a/src/tools/sequence-tasks.ts b/src/tools/sequence-tasks.ts index 22dea23..12b16ae 100644 --- a/src/tools/sequence-tasks.ts +++ b/src/tools/sequence-tasks.ts @@ -1,7 +1,7 @@ // CATEGORY 6: sequence_tasks — Sequencing import { z } from "zod"; import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; -import { run } from "../lib/git.js"; +import { run, shellRun } from "../lib/git.js"; import { now } from "../lib/state.js"; import { PROJECT_DIR } from "../lib/files.js"; import { existsSync } from "fs"; @@ -90,7 +90,7 @@ export function registerSequenceTasks(server: McpServer): void { // For locality: infer directories from path-like tokens in task text if (strategy === "locality") { // Use git ls-files with a depth limit instead of find for performance - const gitFiles = run("git ls-files 2>/dev/null | head -1000"); + const gitFiles = shellRun("git ls-files 2>/dev/null | head -1000"); const knownDirs = new Set(); for (const f of gitFiles.split("\n").filter(Boolean)) { const parts = f.split("/"); diff --git a/src/tools/session-handoff.ts b/src/tools/session-handoff.ts index d199462..ac62a5d 100644 --- a/src/tools/session-handoff.ts +++ b/src/tools/session-handoff.ts @@ -2,7 +2,7 @@ import { z } from "zod"; import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; import { existsSync, readFileSync } from "fs"; import { join } from "path"; -import { run, getBranch, getRecentCommits, getStatus } from "../lib/git.js"; +import { run, shellRun, getBranch, getRecentCommits, getStatus } from "../lib/git.js"; import { readIfExists, findWorkspaceDocs } from "../lib/files.js"; import { STATE_DIR, now } from "../lib/state.js"; @@ -44,7 +44,7 @@ export function registerSessionHandoff(server: McpServer): void { // Only try gh if it exists if (hasCommand("gh")) { - const openPRs = run("gh pr list --state open --json number,title,headRefName 2>/dev/null || echo '[]'"); + const openPRs = shellRun("gh pr list --state open --json number,title,headRefName 2>/dev/null || echo '[]'"); if (openPRs && openPRs !== "[]") { sections.push(`## Open PRs\n\`\`\`json\n${openPRs}\n\`\`\``); } diff --git a/src/tools/session-health.ts b/src/tools/session-health.ts index bd6a819..1129e88 100644 --- a/src/tools/session-health.ts +++ b/src/tools/session-health.ts @@ -27,7 +27,9 @@ export function registerSessionHealth(server: McpServer): void { const dirtyCount = dirty ? dirty.split("\n").filter(Boolean).length : 0; const lastCommit = getLastCommit(); const lastCommitTimeStr = getLastCommitTime(); - const uncommittedDiff = run("git diff --stat | tail -1"); + const diffStatOutput = run(["diff", "--stat"]); + const diffStatLines = diffStatOutput.split("\n").filter(Boolean); + const uncommittedDiff = diffStatLines.length > 0 ? diffStatLines[diffStatLines.length - 1] : ""; // Parse commit time safely const commitDate = parseGitDate(lastCommitTimeStr); diff --git a/src/tools/sharpen-followup.ts b/src/tools/sharpen-followup.ts index db5acaa..df2aadd 100644 --- a/src/tools/sharpen-followup.ts +++ b/src/tools/sharpen-followup.ts @@ -1,7 +1,7 @@ // CATEGORY 4: sharpen_followup — Follow-up Specificity import { z } from "zod"; import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; -import { run } from "../lib/git.js"; +import { run, shellRun } from "../lib/git.js"; import { now } from "../lib/state.js"; /** Parse git porcelain output into deduplicated file paths, handling renames (R/C) */ @@ -87,7 +87,7 @@ export function registerSharpenFollowup(server: McpServer): void { // Gather context to resolve ambiguity const contextFiles: string[] = [...(previous_files ?? [])]; const recentChanged = getRecentChangedFiles(); - const porcelainOutput = run("git status --porcelain 2>/dev/null"); + const porcelainOutput = shellRun("git status --porcelain 2>/dev/null"); const untrackedOrModified = parsePortelainFiles(porcelainOutput); const allKnownFiles = [...new Set([...contextFiles, ...recentChanged, ...untrackedOrModified])].filter(Boolean); diff --git a/src/tools/token-audit.ts b/src/tools/token-audit.ts index b7aad2c..8d04bb2 100644 --- a/src/tools/token-audit.ts +++ b/src/tools/token-audit.ts @@ -1,7 +1,7 @@ // CATEGORY 5: token_audit — Token Efficiency import { z } from "zod"; import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; -import { run } from "../lib/git.js"; +import { run, shellRun } from "../lib/git.js"; import { readIfExists, findWorkspaceDocs, PROJECT_DIR } from "../lib/files.js"; import { loadState, saveState, now, STATE_DIR } from "../lib/state.js"; import { readFileSync, existsSync, statSync } from "fs"; @@ -39,8 +39,8 @@ export function registerTokenAudit(server: McpServer): void { let wasteScore = 0; // 1. Git diff size & dirty file count - const diffStat = run("git diff --stat --no-color 2>/dev/null"); - const dirtyFiles = run("git diff --name-only 2>/dev/null"); + const diffStat = shellRun("git diff --stat --no-color 2>/dev/null"); + const dirtyFiles = shellRun("git diff --name-only 2>/dev/null"); const dirtyList = dirtyFiles.split("\n").filter(Boolean); const dirtyCount = dirtyList.length; diff --git a/src/tools/verify-completion.ts b/src/tools/verify-completion.ts index 732532f..8c515f5 100644 --- a/src/tools/verify-completion.ts +++ b/src/tools/verify-completion.ts @@ -1,6 +1,6 @@ import { z } from "zod"; import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; -import { run, getStatus } from "../lib/git.js"; +import { run, shellRun, getStatus } from "../lib/git.js"; import { PROJECT_DIR } from "../lib/files.js"; import { existsSync } from "fs"; import { join } from "path"; @@ -34,7 +34,7 @@ function detectTestRunner(): string | null { /** Check if a build script exists in package.json */ function hasBuildScript(): boolean { try { - const pkg = JSON.parse(run("cat package.json 2>/dev/null")); + const pkg = JSON.parse(shellRun("cat package.json 2>/dev/null")); return !!pkg?.scripts?.build; } catch { return false; } } @@ -80,7 +80,7 @@ export function registerVerifyCompletion(server: McpServer): void { // 3. Tests if (!skip_tests) { const runner = detectTestRunner(); - const changedFiles = run("git diff --name-only HEAD~1 2>/dev/null").split("\n").filter(Boolean); + const changedFiles = shellRun("git diff --name-only HEAD~1 2>/dev/null").split("\n").filter(Boolean); let testCmd = ""; if (runner === "playwright") { diff --git a/tests/patterns.test.ts b/tests/patterns.test.ts new file mode 100644 index 0000000..55051fe --- /dev/null +++ b/tests/patterns.test.ts @@ -0,0 +1,70 @@ +import { describe, it, expect } from "vitest"; +import { matchPatterns, formatPatternMatches, type CorrectionPattern } from "../src/lib/patterns.js"; + +function makePattern(overrides: Partial = {}): CorrectionPattern { + return { + id: "p1", + pattern: "Recurring correction: auth, token, refresh, expired", + keywords: ["auth", "token", "refresh", "expired"], + frequency: 3, + lastSeen: new Date().toISOString(), + context: "Token refresh was failing silently", + examples: ["fix the auth token"], + ...overrides, + }; +} + +describe("matchPatterns", () => { + it("matches when 2+ keywords appear in prompt", () => { + const patterns = [makePattern()]; + const matches = matchPatterns("the auth token is broken", patterns); + expect(matches).toHaveLength(1); + }); + + it("does not match with only 1 keyword", () => { + const patterns = [makePattern()]; + const matches = matchPatterns("the auth system is down", patterns); + expect(matches).toHaveLength(0); + }); + + it("returns empty for empty patterns list", () => { + expect(matchPatterns("anything here", [])).toEqual([]); + }); + + it("matches case-insensitively", () => { + const patterns = [makePattern()]; + const matches = matchPatterns("AUTH TOKEN issue", patterns); + expect(matches).toHaveLength(1); + }); + + it("can match multiple patterns", () => { + const patterns = [ + makePattern({ id: "p1", keywords: ["auth", "token", "refresh"] }), + makePattern({ id: "p2", keywords: ["deploy", "docker", "build"] }), + ]; + const matches = matchPatterns("auth token deploy docker", patterns); + expect(matches).toHaveLength(2); + }); +}); + +describe("formatPatternMatches", () => { + it("returns empty string for no matches", () => { + expect(formatPatternMatches([])).toBe(""); + }); + + it("formats matches with header and details", () => { + const result = formatPatternMatches([makePattern()]); + expect(result).toContain("Known patterns matched"); + expect(result).toContain("corrected 3x"); + expect(result).toContain("Token refresh"); + }); + + it("numbers multiple matches", () => { + const result = formatPatternMatches([ + makePattern({ id: "p1" }), + makePattern({ id: "p2", pattern: "Another pattern" }), + ]); + expect(result).toContain("1."); + expect(result).toContain("2."); + }); +}); diff --git a/tests/prompt-score.test.ts b/tests/prompt-score.test.ts new file mode 100644 index 0000000..9178566 --- /dev/null +++ b/tests/prompt-score.test.ts @@ -0,0 +1,82 @@ +import { describe, it, expect } from "vitest"; +import { scorePrompt } from "../src/tools/prompt-score.js"; + +describe("scorePrompt", () => { + it("gives high score to specific, actionable prompt with done condition", () => { + const result = scorePrompt( + "Rename the `processOrder` function in `src/orders/handler.ts` to `handleOrder`. Tests should still pass." + ); + expect(result.total).toBeGreaterThanOrEqual(85); + expect(result.grade).toMatch(/^A/); + expect(result.specificity).toBe(25); // has file path + backtick identifier + expect(result.actionability).toBe(25); // "rename" + expect(result.doneCondition).toBe(25); // "should" + "pass" + }); + + it("gives low score to vague prompt", () => { + const result = scorePrompt("make it better"); + expect(result.total).toBeLessThanOrEqual(40); + expect(result.grade).toMatch(/^[DF]/); + expect(result.feedback.length).toBeGreaterThan(0); + }); + + it("detects action verbs correctly", () => { + const result = scorePrompt("fix the bug"); + expect(result.actionability).toBe(25); + }); + + it("penalizes vague verbs", () => { + const result = scorePrompt("make things work"); + expect(result.actionability).toBe(15); + expect(result.feedback.some(f => f.includes("Vague verb"))).toBe(true); + }); + + it("rewards file paths for specificity", () => { + const result = scorePrompt("update src/lib/auth.ts"); + expect(result.specificity).toBe(25); + }); + + it("gives partial specificity for generic file references", () => { + const result = scorePrompt("update the file"); + expect(result.specificity).toBe(15); + }); + + it("flags unbounded scope", () => { + const result = scorePrompt("fix all the errors"); + expect(result.scope).toBe(10); + }); + + it("rewards bounded scope", () => { + const result = scorePrompt("only update the header component"); + expect(result.scope).toBe(25); + }); + + it("detects done conditions with should/must/expect", () => { + const withShould = scorePrompt("change X so it should return 42"); + expect(withShould.doneCondition).toBe(25); + + const withQuestion = scorePrompt("why is this failing?"); + expect(withQuestion.doneCondition).toBe(20); + }); + + it("returns valid grade for every score range", () => { + // Perfect prompt + const perfect = scorePrompt( + "Refactor only the `validate` function in `src/utils/validator.ts` — it should return a boolean" + ); + expect(["A+", "A", "A-", "B+", "B", "B-", "C+", "C", "D", "F"]).toContain(perfect.grade); + + // Empty prompt + const empty = scorePrompt(""); + expect(["A+", "A", "A-", "B+", "B", "B-", "C+", "C", "D", "F"]).toContain(empty.grade); + }); + + it("includes congratulatory message for perfect scores", () => { + const result = scorePrompt( + "Rename `processOrder` in `src/orders/handler.ts` to `handleOrder`. Only this function. Tests should pass." + ); + if (result.total >= 90) { + expect(result.feedback.some(f => f.includes("Excellent"))).toBe(true); + } + }); +});