diff --git a/README.md b/README.md index 6d03f5d..7e8b3f5 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,129 @@ The pattern is always the same: vague prompt → Claude guesses → wrong output --- +## Usage Walkthrough + +Here's what preflight actually looks like in practice — real tool calls with realistic output. + +### 1. Vague prompt → automatic clarification + +You type into Claude Code: + +``` +> update the dashboard +``` + +Claude calls `preflight_check` with `{ "prompt": "update the dashboard" }`. Triage classifies this as **ambiguous** (< 50 chars, no file refs). You get back: + +``` +⚠️ AMBIGUOUS — needs clarification before proceeding + +📂 Git State: + Branch: feat/analytics + 3 uncommitted files: src/components/Dashboard.tsx, src/hooks/useDashboard.ts, src/api/metrics.ts + +🔍 Ambiguity Signals: + - "update" is vague — add data? change layout? fix a bug? restyle? + - "dashboard" matches 4 files across the project + - No specific file path or line number referenced + +💡 Try something like: + - "add the new revenue chart to src/components/Dashboard.tsx using the metrics API from src/api/metrics.ts" + - "fix the loading spinner that won't dismiss in the analytics dashboard" + - "update dashboard layout to 3-column grid per the Figma mock" +``` + +### 2. Multi-step task → scoped execution plan + +``` +> refactor the payments module to support Stripe and add webhook handling +``` + +Triage: **multi-step** (multiple verbs + cross-concern). `scope_work` and `sequence_tasks` fire automatically: + +``` +🔶 MULTI-STEP — breaking this down before starting + +📋 Execution Plan: + Step 1: Extract payment interface (src/payments/provider.ts) + → Define PaymentProvider interface with charge(), refund(), webhook() + → Risk: LOW — new file, no existing consumers yet + + Step 2: Implement Stripe adapter (src/payments/stripe.ts) + → Implement PaymentProvider using stripe SDK + → Dependencies: stripe npm package + → Risk: MEDIUM — needs API keys, error handling for network failures + + Step 3: Add webhook endpoint (src/api/webhooks/stripe.ts) + → POST /api/webhooks/stripe with signature verification + → Risk: HIGH — security-sensitive, needs crypto verification + → ⚠️ Checkpoint recommended before this step + + Step 4: Update consumers (src/checkout/*, src/subscriptions/*) + → Swap direct calls to new PaymentProvider interface + → 6 files affected + +🧪 Suggested checkpoints: after Step 1, after Step 3 +⏱️ Estimated tokens: ~45,000 across all steps +``` + +### 3. Correction pattern matching + +You previously logged a correction with `log_correction`: + +``` +> log_correction: "When I said 'update types', I meant only the shared types in packages/shared, not the local type files in each service" +``` + +Next time you say: + +``` +> update the types +``` + +Preflight catches it: + +``` +⚠️ PATTERN MATCH — you've corrected this before + +🔄 Previous correction (2 days ago): + "When I said 'update types', I meant only the shared types in packages/shared, + not the local type files in each service" + +Did you mean shared types in packages/shared? If so, try: + "update shared types in packages/shared/types.ts" +``` + +### 4. Cross-service awareness + +``` +> add a loyalty points field to the user profile +``` + +Triage: **cross-service** (detected `user` + field change pattern, config has related projects). Contracts are searched: + +``` +🔗 CROSS-SERVICE — this change affects multiple projects + +📂 Current project: web-app + - src/types/user.ts → UserProfile interface (line 12) + - prisma/schema.prisma → User model (line 34) + +🔗 Related projects: + - mobile-app: src/api/types.ts → UserProfile (mirrors web-app, line 8) + - rewards-service: src/models/user.ts → UserRecord (includes loyalty_points already?) + - analytics-pipeline: src/schemas/user-events.avro → UserEvent schema + +⚠️ Changing UserProfile requires updates in 3 services. + Suggested order: + 1. prisma/schema.prisma (source of truth) + migrate + 2. web-app src/types/user.ts + 3. mobile-app src/api/types.ts (notify mobile team) + 4. analytics-pipeline schema (may need backfill) +``` + +--- + ## Quick Start ### Option A: npx (fastest — no install) diff --git a/examples/.preflight/config.yml b/examples/.preflight/config.yml index f59170f..297d8d7 100644 --- a/examples/.preflight/config.yml +++ b/examples/.preflight/config.yml @@ -1,35 +1,46 @@ -# .preflight/config.yml — Drop this in your project root -# -# This is an example config for a typical Next.js + microservices setup. -# Every field is optional — preflight works with sensible defaults out of the box. -# Commit this to your repo so the whole team gets the same preflight behavior. - -# Profile controls how much detail preflight returns. -# "minimal" — only flags ambiguous+ prompts, skips clarification detail +# .preflight/config.yml — Drop this in your project root. +# Every field is optional. Preflight works out of the box with sensible defaults. +# Commit this to your repo so your whole team gets the same behavior. + +# Profile controls how much detail preflight surfaces. +# "minimal" — only flag ambiguous+, skip clarification detail # "standard" — balanced (default) # "full" — maximum detail on every non-trivial prompt profile: standard -# Related projects for cross-service awareness. -# Preflight will search these for shared types, routes, and contracts -# so it can warn you when a change might break a consumer. +# Related projects for cross-service contract awareness. +# Preflight scans these for shared types, interfaces, and API routes +# so it can warn when a change in one project might break another. related_projects: - - path: /Users/you/code/auth-service - alias: auth - - path: /Users/you/code/billing-api - alias: billing - - path: /Users/you/code/shared-types - alias: types - -# Behavioral thresholds — tune these to your workflow + # Example: a backend API that your frontend talks to + # - path: /Users/you/projects/my-api + # alias: api + + # Example: a shared types package + # - path: /Users/you/projects/shared-types + # alias: shared + +# Behavioral thresholds — tune these to your workflow. thresholds: - session_stale_minutes: 30 # Warn if no activity for this long - max_tool_calls_before_checkpoint: 100 # Suggest a checkpoint after N tool calls - correction_pattern_threshold: 3 # Min corrections before flagging a pattern + # Warn if no session activity for this many minutes + session_stale_minutes: 30 -# Embedding provider for semantic search over session history. -# "local" uses Xenova transformers (no API key needed, runs on CPU). -# "openai" uses text-embedding-3-small (faster, needs OPENAI_API_KEY). + # Suggest a checkpoint after this many tool calls in one session + max_tool_calls_before_checkpoint: 100 + + # Minimum correction occurrences before flagging as a pattern + # (e.g., if you keep correcting "use the auth module" 3+ times, + # preflight learns to warn you proactively) + correction_pattern_threshold: 3 + +# Embedding configuration for vector search (timeline tools). +# Local embeddings work offline with no API key — they just need +# a one-time ~90MB model download on first use. embeddings: + # "local" — Xenova/all-MiniLM-L6-v2, runs on-device, no API key needed + # "openai" — OpenAI text-embedding-3-small, needs OPENAI_API_KEY + # "ollama" — Local Ollama server, needs `ollama serve` running provider: local - # openai_api_key: sk-... # Uncomment if using openai provider + + # Only needed if provider is "openai": + # openai_api_key: sk-... diff --git a/examples/.preflight/contracts/api.yml b/examples/.preflight/contracts/api.yml index 512543f..7a2ebd3 100644 --- a/examples/.preflight/contracts/api.yml +++ b/examples/.preflight/contracts/api.yml @@ -1,17 +1,15 @@ -# .preflight/contracts/api.yml — Manual contract definitions +# .preflight/contracts/api.yml — Manual contract definitions. # -# Define shared types and interfaces that preflight should know about. -# These supplement auto-extracted contracts from your codebase. -# Manual definitions win on name conflicts with auto-extracted ones. +# These supplement auto-extraction. Use them when: +# - A related service isn't in TypeScript (preflight can't auto-extract) +# - You want to pin a specific contract shape for stability +# - You're documenting an external API your project depends on # -# Why manual contracts? -# - Document cross-service interfaces that live in docs, not code -# - Define contracts for external APIs your services consume -# - Pin down types that are implicit (e.g., event payloads) +# Preflight uses these to warn when a prompt might affect shared types. - name: User kind: interface - description: Core user model shared across all services + description: Core user object shared between frontend and API fields: - name: id type: string @@ -19,40 +17,29 @@ - name: email type: string required: true - - name: tier - type: "'free' | 'pro' | 'enterprise'" + - name: role + type: "'admin' | 'member' | 'viewer'" required: true - name: createdAt type: Date required: true -- name: AuthToken +- name: ApiResponse kind: interface - description: JWT payload structure from auth-service + description: Standard API response envelope fields: - - name: userId - type: string - required: true - - name: permissions - type: string[] - required: true - - name: expiresAt - type: number - required: true - -- name: WebhookPayload - kind: interface - description: Standard webhook envelope for inter-service events - fields: - - name: event - type: string - required: true - - name: timestamp - type: string + - name: success + type: boolean required: true - name: data - type: Record - required: true - - name: source + type: T + required: false + - name: error type: string - required: true + required: false + +# Add your own shared types below: +# - name: OrderStatus +# kind: enum +# description: Order lifecycle states +# values: [pending, confirmed, shipped, delivered, cancelled] diff --git a/examples/.preflight/triage.yml b/examples/.preflight/triage.yml index b3d394e..4aa02d1 100644 --- a/examples/.preflight/triage.yml +++ b/examples/.preflight/triage.yml @@ -1,45 +1,43 @@ -# .preflight/triage.yml — Controls how preflight classifies your prompts -# -# The triage engine routes prompts into categories: -# TRIVIAL → pass through (commit, format, lint) -# CLEAR → well-specified, no intervention needed -# AMBIGUOUS → needs clarification before proceeding -# MULTI-STEP → complex task, preflight suggests a plan -# CROSS-SERVICE → touches multiple projects, pulls in contracts -# -# Customize the keywords below to match your domain. +# .preflight/triage.yml — Controls how preflight classifies your prompts. +# Customize these rules to match your project's domain vocabulary. rules: - # Prompts containing these words are always flagged as AMBIGUOUS. - # Add domain-specific terms that tend to produce vague prompts. + # Prompts containing these keywords are always flagged as AMBIGUOUS or higher. + # Add domain-specific terms that tend to produce vague requests. always_check: - rewards - permissions - migration - schema - - pricing # example: your billing domain - - onboarding # example: multi-step user flows + # Add your own: + # - billing + # - deployment + # - auth - # Prompts containing these words skip checks entirely (TRIVIAL). - # These are safe, mechanical tasks that don't need guardrails. + # Prompts containing these keywords pass through as TRIVIAL. + # These are routine commands where clarification adds no value. skip: - commit - format - lint - - prettier - - "git push" + # Add your own: + # - typecheck + # - "git status" - # Prompts containing these words trigger CROSS-SERVICE classification. - # Preflight will search related_projects for relevant types and routes. + # Prompts containing these keywords trigger CROSS-SERVICE checks. + # These scan related_projects (from config.yml) for shared contracts. cross_service_keywords: - auth - notification - event - webhook - - billing # matches the related_project alias + # Add your own: + # - graphql + # - grpc + # - queue # How aggressively to classify prompts. -# "relaxed" — more prompts pass as clear (experienced users) +# "relaxed" — more prompts pass as clear (less interruption) # "standard" — balanced (default) -# "strict" — more prompts flagged as ambiguous (new teams, complex codebases) +# "strict" — more prompts flagged as ambiguous (maximum savings) strictness: standard diff --git a/src/tools/token-audit.ts b/src/tools/token-audit.ts index b7aad2c..dd8a9ee 100644 --- a/src/tools/token-audit.ts +++ b/src/tools/token-audit.ts @@ -7,9 +7,42 @@ import { loadState, saveState, now, STATE_DIR } from "../lib/state.js"; import { readFileSync, existsSync, statSync } from "fs"; import { join } from "path"; -/** Shell-escape a filename for safe interpolation */ -function shellEscape(s: string): string { - return s.replace(/'/g, "'\\''"); +/** Count lines in a file using Node APIs (no shell). Returns 0 on error. */ +function countFileLines(filePath: string): number { + try { + const abs = join(PROJECT_DIR, filePath); + const content = readFileSync(abs, "utf-8"); + return content.split("\n").length; + } catch { + return 0; + } +} + +/** Get file size in bytes using Node APIs. Returns 0 on error. */ +function getFileSize(filePath: string): number { + try { + const abs = join(PROJECT_DIR, filePath); + return statSync(abs).size; + } catch { + return 0; + } +} + +/** Read the tail of a file (last N bytes). */ +function readFileTail(absPath: string, maxBytes: number): string { + try { + const stat = statSync(absPath); + if (stat.size <= maxBytes) { + return readFileSync(absPath, "utf-8"); + } + const buf = Buffer.alloc(maxBytes); + const fd = require("fs").openSync(absPath, "r"); + require("fs").readSync(fd, buf, 0, maxBytes, stat.size - maxBytes); + require("fs").closeSync(fd); + return buf.toString("utf-8"); + } catch { + return ""; + } } /** @@ -39,8 +72,8 @@ export function registerTokenAudit(server: McpServer): void { let wasteScore = 0; // 1. Git diff size & dirty file count - const diffStat = run("git diff --stat --no-color 2>/dev/null"); - const dirtyFiles = run("git diff --name-only 2>/dev/null"); + const diffStat = run(["diff", "--stat", "--no-color"]); + const dirtyFiles = run(["diff", "--name-only"]); const dirtyList = dirtyFiles.split("\n").filter(Boolean); const dirtyCount = dirtyList.length; @@ -63,8 +96,7 @@ export function registerTokenAudit(server: McpServer): void { for (const f of dirtyList.slice(0, 30)) { // Use shell-safe quoting instead of interpolation - const wc = run(`wc -l < '${shellEscape(f)}' 2>/dev/null`); - const lines = parseInt(wc) || 0; + const lines = countFileLines(f); estimatedContextTokens += lines * AVG_LINE_BYTES * AVG_TOKENS_PER_BYTE; if (lines > 500) { largeFiles.push(`${f} (${lines} lines)`); @@ -80,8 +112,7 @@ export function registerTokenAudit(server: McpServer): void { // 3. CLAUDE.md bloat check const claudeMd = readIfExists("CLAUDE.md", 1); if (claudeMd !== null) { - const stat = run(`wc -c < '${shellEscape("CLAUDE.md")}' 2>/dev/null`); - const bytes = parseInt(stat) || 0; + const bytes = getFileSize("CLAUDE.md"); if (bytes > 5120) { patterns.push(`CLAUDE.md is ${(bytes / 1024).toFixed(1)}KB — injected every session, burns tokens on paste`); recommendations.push("Trim CLAUDE.md to essentials (<5KB). Move reference docs to files read on-demand"); @@ -137,9 +168,7 @@ export function registerTokenAudit(server: McpServer): void { } // Read with size cap: take the tail if too large - const raw = stat.size <= MAX_TOOL_LOG_BYTES - ? readFileSync(toolLogPath, "utf-8") - : run(`tail -c ${MAX_TOOL_LOG_BYTES} '${shellEscape(toolLogPath)}'`); + const raw = readFileTail(toolLogPath, MAX_TOOL_LOG_BYTES); const lines = raw.trim().split("\n").filter(Boolean); totalToolCalls = lines.length; diff --git a/tests/triage.test.ts b/tests/triage.test.ts new file mode 100644 index 0000000..adef2e1 --- /dev/null +++ b/tests/triage.test.ts @@ -0,0 +1,257 @@ +import { describe, it, expect } from "vitest"; +import { triagePrompt, type TriageResult } from "../src/lib/triage.js"; + +// Helper +const triage = (prompt: string, config?: Parameters[1]) => + triagePrompt(prompt, config); + +describe("triagePrompt", () => { + // ── Trivial ──────────────────────────────────────────────────────────── + + describe("trivial classification", () => { + it.each([ + "commit", + "lint", + "run tests", + "push", + "build", + "format", + "test", + ])("classifies '%s' as trivial", (prompt) => { + expect(triage(prompt).level).toBe("trivial"); + }); + + it("classifies short commands with args as trivial", () => { + expect(triage("commit -m fix typo").level).toBe("trivial"); + }); + + it("returns high confidence for trivial", () => { + expect(triage("commit").confidence).toBeGreaterThanOrEqual(0.85); + }); + + it("returns empty recommended_tools for trivial", () => { + expect(triage("commit").recommended_tools).toEqual([]); + }); + }); + + // ── Skip keywords ───────────────────────────────────────────────────── + + describe("skip keywords", () => { + it("returns trivial when prompt matches a skip keyword", () => { + const result = triage("please deploy now", { skip: ["deploy"] }); + expect(result.level).toBe("trivial"); + expect(result.confidence).toBe(0.95); + }); + + it("is case-insensitive for skip keywords", () => { + expect(triage("DEPLOY", { skip: ["deploy"] }).level).toBe("trivial"); + }); + }); + + // ── Clear ────────────────────────────────────────────────────────────── + + describe("clear classification", () => { + it("classifies prompt with file path as clear", () => { + const result = triage("fix the null check in src/auth/jwt.ts line 42"); + expect(result.level).toBe("clear"); + }); + + it("classifies detailed prompt with file refs as clear", () => { + const result = triage( + "add error handling to the parseToken function in src/auth/jwt.ts" + ); + expect(result.level).toBe("clear"); + }); + + it("includes verify-files-exist when file refs present", () => { + const result = triage("refactor src/auth/jwt.ts to use async/await"); + expect(result.recommended_tools).toContain("verify-files-exist"); + }); + + it("adjusts confidence down in strict mode", () => { + const relaxed = triage("update src/auth/jwt.ts exports", { + strictness: "standard", + }); + const strict = triage("update src/auth/jwt.ts exports", { + strictness: "strict", + }); + expect(strict.confidence).toBeLessThanOrEqual(relaxed.confidence); + }); + }); + + // ── Ambiguous ────────────────────────────────────────────────────────── + + describe("ambiguous classification", () => { + it("classifies short vague prompt as ambiguous", () => { + expect(triage("fix the auth bug").level).toBe("ambiguous"); + }); + + it("classifies prompt with vague pronouns as ambiguous", () => { + const result = triage("fix it"); + expect(result.level).toBe("ambiguous"); + expect(result.reasons.some((r) => r.includes("vague"))).toBe(true); + }); + + it("classifies vague verbs without targets as ambiguous", () => { + expect(triage("update the code").level).toBe("ambiguous"); + }); + + it("does NOT flag vague verbs when file refs are present", () => { + const result = triage("fix src/auth/jwt.ts"); + expect(result.level).not.toBe("ambiguous"); + }); + + it("recommends clarify-intent for ambiguous", () => { + const result = triage("fix the auth bug"); + expect(result.recommended_tools).toContain("clarify-intent"); + }); + }); + + // ── always_check keywords ───────────────────────────────────────────── + + describe("always_check keywords", () => { + it("forces at least ambiguous for always_check keywords", () => { + const result = triage("update the migration scripts", { + alwaysCheck: ["migration"], + }); + expect(result.level).toBe("ambiguous"); + }); + + it("is case-insensitive", () => { + const result = triage("run MIGRATION", { alwaysCheck: ["migration"] }); + expect(result.level).toBe("ambiguous"); + }); + }); + + // ── Cross-service ────────────────────────────────────────────────────── + + describe("cross-service classification", () => { + it("detects cross-service keywords", () => { + const result = triage("update the shared schema for auth", { + crossServiceKeywords: ["shared schema"], + }); + expect(result.level).toBe("cross-service"); + expect(result.cross_service_hits).toBeDefined(); + }); + + it("detects related project aliases", () => { + const result = triage("add tiered rewards from rewards-api", { + relatedAliases: ["rewards-api"], + }); + expect(result.level).toBe("cross-service"); + }); + + it("detects built-in cross-service terms", () => { + const result = triage( + "update the user interface contract definition for the billing module" + ); + expect(result.level).toBe("cross-service"); + }); + + it("recommends search-related-projects", () => { + const result = triage("check the event schema", { + crossServiceKeywords: ["event schema"], + }); + expect(result.recommended_tools).toContain("search-related-projects"); + }); + }); + + // ── Multi-step ───────────────────────────────────────────────────────── + + describe("multi-step classification", () => { + it("detects sequential language (then)", () => { + const result = triage( + "refactor auth to OAuth2 then update all API consumers" + ); + expect(result.level).toBe("multi-step"); + }); + + it("detects 'first...then' patterns", () => { + expect( + triage("first update the schema, then regenerate the types").level + ).toBe("multi-step"); + }); + + it("detects numbered lists", () => { + const result = triage("todo:\n1) update schema\n2) run migration\n3) test"); + expect(result.level).toBe("multi-step"); + }); + + it("detects bullet lists", () => { + const result = triage("please do:\n- update auth\n- fix tests\n- deploy"); + expect(result.level).toBe("multi-step"); + }); + + it("detects files in different directories", () => { + const result = triage( + "update src/auth/jwt.ts and tests/auth/jwt.test.ts with the new token format" + ); + expect(result.level).toBe("multi-step"); + }); + + it("recommends sequence-tasks for multi-step", () => { + const result = triage("first do X then do Y"); + expect(result.recommended_tools).toContain("sequence-tasks"); + }); + }); + + // ── Priority ordering ───────────────────────────────────────────────── + + describe("classification priority", () => { + it("skip keyword beats everything", () => { + // This would otherwise be multi-step + const result = triage("deploy first then restart", { skip: ["deploy"] }); + expect(result.level).toBe("trivial"); + }); + + it("multi-step beats cross-service", () => { + // Has both cross-service terms and multi-step indicators + const result = triage( + "update the schema definition then update all API consumers", + { crossServiceKeywords: ["schema"] } + ); + expect(result.level).toBe("multi-step"); + }); + + it("cross-service beats ambiguous", () => { + const result = triage("fix it in the schema", { + crossServiceKeywords: ["schema"], + }); + // "fix it" is vague but "schema" is cross-service keyword + expect(result.level).toBe("cross-service"); + }); + }); + + // ── Edge cases ───────────────────────────────────────────────────────── + + describe("edge cases", () => { + it("handles empty prompt", () => { + const result = triage(""); + expect(result.level).toBeDefined(); + expect(result.confidence).toBeGreaterThan(0); + }); + + it("handles very long prompt", () => { + const long = "update the authentication module ".repeat(100); + const result = triage(long); + expect(result.level).toBeDefined(); + }); + + it("handles no config", () => { + const result = triagePrompt("fix the auth bug"); + expect(result.level).toBeDefined(); + }); + + it("returns valid TriageResult shape", () => { + const result = triage("do something"); + expect(result).toHaveProperty("level"); + expect(result).toHaveProperty("confidence"); + expect(result).toHaveProperty("reasons"); + expect(result).toHaveProperty("recommended_tools"); + expect(Array.isArray(result.reasons)).toBe(true); + expect(Array.isArray(result.recommended_tools)).toBe(true); + expect(result.confidence).toBeGreaterThanOrEqual(0); + expect(result.confidence).toBeLessThanOrEqual(1); + }); + }); +});