From 36ddedad70258eb8cdd185d522fbc94f8adb5d07 Mon Sep 17 00:00:00 2001 From: Joseph Mearman Date: Sun, 15 Feb 2026 10:26:17 +0000 Subject: [PATCH] fix(prompts): require API verification before flagging hallucinated APIs Add WebFetch to review mode tools and update prompts to instruct the reviewer to check node_modules, lockfiles, or official docs before claiming an API doesn't exist. Training data may be stale for newer dependency versions. --- .github/prompts/review/01-command.md | 5 +++++ .github/prompts/review/02-red-flags.md | 2 +- action.yml | 2 +- 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/.github/prompts/review/01-command.md b/.github/prompts/review/01-command.md index 14b8f01..23905b0 100644 --- a/.github/prompts/review/01-command.md +++ b/.github/prompts/review/01-command.md @@ -25,3 +25,8 @@ toward the end of the diff. Only flag issues you are confident about. A false positive wastes more of the author's time than a missed minor issue. If you are unsure, leave it out. + +When reviewing unfamiliar APIs or newer library versions, verify your claims +before commenting. Use `Read` to check `node_modules` exports or lock file +versions, or `WebFetch` to check official docs. Do not rely solely on training +data for API surface knowledge — it may be outdated. diff --git a/.github/prompts/review/02-red-flags.md b/.github/prompts/review/02-red-flags.md index 632fba2..14305bb 100644 --- a/.github/prompts/review/02-red-flags.md +++ b/.github/prompts/review/02-red-flags.md @@ -1,6 +1,6 @@ ## Red flags to watch for in diffs -- **Hallucinated APIs** — method calls or library functions that don't exist in the dependency version used +- **Hallucinated APIs** — method calls or library functions that don't exist in the dependency version used. Before flagging, verify against the actual package (check `node_modules`, `package-lock.json`, or fetch the package's docs/changelog). Your training data may be stale — if a dependency version is newer than you expect, check rather than assume. - **Phantom dependencies** — imports for packages not in `package.json` / `requirements.txt` / equivalent manifest - **Silent failure** — try-catch returning fallbacks instead of propagating, optional chaining on data that should always exist, defaults on internal state hiding upstream bugs - **Over-abstraction** — unnecessary design patterns or enterprise-grade abstractions for simple problems diff --git a/action.yml b/action.yml index 87f9019..c0fd89f 100644 --- a/action.yml +++ b/action.yml @@ -184,7 +184,7 @@ runs: COMMON_TOOLS="mcp__github__get_pull_request,mcp__github__get_pull_request_files,mcp__github__get_pull_request_diff,mcp__github__get_pull_request_reviews,mcp__github__list_pull_request_comments,mcp__github__create_pending_pull_request_review,mcp__github__add_comment_to_pending_review,mcp__github__submit_pending_pull_request_review,mcp__github__create_and_submit_pull_request_review,mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*),Bash(gh pr edit:*),Bash(gh label *),Bash(gh api:*)" if [ "$MODE" = "review" ]; then - TOOLS="$COMMON_TOOLS,Read,Grep,Glob,Bash(git diff:*),Bash(git log:*),Bash(git blame:*),Bash(gh issue create:*),Bash(gh pr create:*)" + TOOLS="$COMMON_TOOLS,Read,Grep,Glob,WebFetch,Bash(git diff:*),Bash(git log:*),Bash(git blame:*),Bash(gh issue create:*),Bash(gh pr create:*)" else TOOLS="$COMMON_TOOLS,Bash(gh issue comment:*),Bash(gh issue view:*)" fi