The canary for your MCP server.
Probe it. Drill it with YAML tests. Lint its config.
Catch failures in CI — before your users do.
You built an MCP server. You tested it by clicking around the Inspector — once. Then you changed a tool description, shipped it, and found out from users that the server hangs on startup.
The Inspector is a great GUI and a terrible regression suite. It can't run
in CI, it can't assert anything, and it won't tell you that your npx entry
is missing -y or that two tools have identical descriptions so the model
picks between them at random.
mcpcanary is the missing command line:
npx mcpcanary probe -- npx -y @modelcontextprotocol/server-everything # does it even start?
npx mcpcanary init -- node ./build/server.js # scaffold a drill file
npx mcpcanary run drills/ # do the tools still work?
npx mcpcanary lint --probe # will clients choke on my config?- Probe replaces "let me open the Inspector and see". One command answers: does it start, what protocol version, what capabilities, how many tools, are any descriptions empty? Exit code tells CI the truth.
- Drills are YAML, not code. A tool call plus assertions in six lines. No test framework to wire up, no client library to learn.
- Lint catches the failure modes nobody warns you about:
npxwithout-yhanging every headless client,mcpserversinstead ofmcpServerssilently ignored, hardcoded API keys, remote servers over plainhttp://, prompt-injection phrasing in third-party tool descriptions, duplicate tool descriptions… - Built for CI first: stable exit codes,
--jsonoutput, GitHub Actions annotations (::error) out of the box.
npm install -D mcpcanary # as a dev dependency
# or just run it
npx mcpcanary --helpNode ≥ 20.
# 0. Let init write your first drill file from a live probe:
mcpcanary init -- node ./build/server.jscreated mcp.drill.yaml with 6 drill(s) scaffolded from your server
next: review the TODO expectations, then run:
mcpcanary run mcp.drill.yaml
init lists your real tool names, scaffolds arguments from each tool's input
schema, and leaves clearly-marked TODO expectations for you to tighten.
(If you skip this step, the YAML below is everything there is.)
# 1. Does my server start and what does it expose?
mcpcanary probe -- node ./build/server.jstiny-fixture @ 1.2.3 — connected in 104ms
capabilities: tools, resources
tools (2)
· add — Adds two numbers and returns the sum.
· echo — Echoes back whatever it receives.
# 2. Write a drill file…# mcp.drill.yaml
server:
command: node
args: [./build/server.js]
timeoutMs: 10000
drills:
- name: adds numbers
tool: add
arguments: { a: 2, b: 3 }
expect:
textEquals: "5"
latencyMsAtMost: 2000
- name: errors surface properly
tool: divide
arguments: { a: 1, b: 0 }
expect:
isError: true
textContains: zero
- name: structured JSON is intact
tool: stats
expect:
jsonPath: $.total
jsonEquals: 42
- name: the toolbox is complete
listTools:
contains: [add, divide, stats]# 3. Run it (add it to `npm test`)
mcpcanary run mcp.drill.yamlmcp.drill.yaml — my-server@1.0.0
✓ adds numbers 3ms
✓ errors surface properly 1ms
✓ structured JSON is intact 2ms
✓ the toolbox is complete
ok: 1 file, 4 drills, 4 passed, 118ms
When something breaks you get the assertion, the actual value, and the server's last stderr lines — which is usually the whole debugging session:
✗ bad math
· text equals "3" — got "2"
ok: …
✗ unreachable
server failed to start or respond: connect ECONNREFUSED
stderr tail:
| boom: missing OPENAI_API_KEY
mcpcanary lint scans .mcp.json, .cursor/mcp.json, .vscode/mcp.json,
Claude Desktop / Claude Code configs and server.json manifests it finds.
| Rule | Severity | Catches |
|---|---|---|
| C001 | error | mcpservers / MCP_SERVERS key typos that clients silently ignore |
| C002 | info | config files with no server entries |
| C003–C004 | error | malformed entries; neither command nor url |
| C005 | error | npx/bunx/uvx without -y — startup hangs at the install prompt in every headless client |
| C006 | warning | relative paths that only work on your machine |
| C007–C008 | warn/error | Windows cmd /c pitfalls; malformed env |
| C009 | error | hardcoded API keys (sk-…, AWS, GitHub, Slack) in config files |
| C010–C011 | error/warning | invalid URLs; remote servers over plain http:// |
| M001–M007 | error/warning | registry manifest problems: missing fields, non-reverse-DNS names, invalid npm identifiers |
| D001–D008 | error/warning | live tool audits (--probe): missing descriptions, name-only descriptions, duplicate descriptions across tools, invalid tool names, oversized context-burning descriptions, prompt-injection phrasing |
Add it to review CI and stop bad configs at the PR:
mcpcanary lint --probeA drill file has an optional server: block (stdio via command/args/env/cwd,
or HTTP via url/headers), an optional default timeoutMs, and a drills: list.
| Drill type | Fields | Asserts on |
|---|---|---|
| tool call | tool, arguments, expect |
result content |
| listing | listTools / listResources / listPrompts |
advertised names |
Expectations for tool calls:
| Key | Meaning |
|---|---|
textContains |
string or array of strings, all must appear |
textEquals |
exact (trimmed) text match |
textMatches |
regular expression source |
isError |
whether the tool result is an error |
jsonPath (+ jsonEquals / jsonContains / jsonExists) |
JSONPath into structuredContent, falling back to JSON embedded in text output |
latencyMsAtMost |
wall-clock budget for the call |
Listing expectations: contains, excludes, countExactly, countAtLeast, countAtMost.
Exit codes: 0 everything passed · 1 failures/errors · 2 usage problem (nothing to run, unreadable file).
Annotations are emitted automatically when GITHUB_ACTIONS=true; failures show
up inline on the PR:
name: mcp-drills
on: [push, pull_request]
jobs:
drill:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- run: npm ci && npm run build
- run: npx mcpcanary run drills/
- run: npx mcpcanary lint --probe| Inspector (official) | GUI test platforms | mcpcanary | |
|---|---|---|---|
| Runs in CI | ✗ (interactive GUI) | partial | ✓ |
| Declarative assertions | ✗ | some | ✓ YAML |
Config linting (npx -y, secrets, typos) |
✗ | ✗ | ✓ |
| Tool-description audit | manual | manual | ✓ automated |
| Setup | open app | account/API keys | one npm package, zero config |
| Output for bots | ✗ | ✗ | --json + GH annotations |
mcpcanary doesn't replace the Inspector for interactive exploration — it replaces re-opening the Inspector every time you change something.
See ROADMAP.md for the full plan. Highlights:
- v0.2 —
initscaffolding ✅, standalone GitHub Action,--watch, session reuse - v0.3 — snapshot testing, SSE transport, schema linting depth, token-cost report
- v0.4 — registry health sweep, pass/fail badges, preset drill packs
PRs welcome — see CONTRIBUTING.md.