Skip to content

feat(mcp): allow the test tool to target specific paths - #1676

Open
HadiHassan22 wants to merge 1 commit into
VeryGoodOpenSource:mainfrom
HadiHassan22:feat/mcp-test-paths
Open

feat(mcp): allow the test tool to target specific paths#1676
HadiHassan22 wants to merge 1 commit into
VeryGoodOpenSource:mainfrom
HadiHassan22:feat/mcp-test-paths

Conversation

@HadiHassan22

Copy link
Copy Markdown

Description

Closes #1675.

The test MCP tool had no way to run a subset of a suite, so every call ran the whole thing. There was no workaround through the existing arguments: directory is deliberately applied as the working directory rather than as a target (per the existing NOTE in _parseTest), and tags / exclude_tags only filter on annotations already present in the test source.

The CLI has no such gap — very_good test test/foo_test.dart works today, because the test commands keep argResults.rest and forward it to the runner. This just exposes that through MCP.

Changes

  • Adds an optional paths argument to the test tool: a list of strings, each a test file or directory.
  • _parseTest appends them after every option, so they are parsed as rest rather than consumed as the value of a preceding option.
  • Works for both very_good test and very_good dart test; omitting the argument leaves behaviour exactly as before.

The argument description also notes that targeting specific paths disables the test optimization step. That is pre-existing CLI behaviour via TestCLIRunner.isTargettingTestFiles, not something this PR changes — it just makes it discoverable to a caller who can no longer see the command line.

Why this matters for MCP callers

An agent driving the CLI through MCP was strictly less capable than one shelling out. Beyond the wasted wall-clock of running everything to check one directory, the tool returns the runner's full output, so a suite whose failure produces a large widget-tree or stack dump can return well over 100k characters — nearly all of it irrelevant to the tests the caller cared about. Narrowing the run is the cheapest mitigation available.

Testing

Three tests added to test/src/mcp/mcp_server_test.dart, covering both branches of the new code:

  • paths forwarded as positional targets
  • paths appended after options (asserts ordering with dart + concurrency)
  • an empty list adds no positional targets

Verified locally:

  • dart format lib test — no changes
  • dart analyze --fatal-infos --fatal-warnings lib test — no issues
  • very_good dart test -x pull-request-only — 516 passing

Open questions

Happy to change the argument name (paths vs targets vs test_paths) or its shape — I used a string array, though packages_get's ignore uses a comma-separated string, so let me know if you'd rather stay consistent with that.

The `test` tool exposed no way to run a subset of a suite, so every call
ran everything. The CLI already supports positional test targets — the
test command forwards `argResults.rest` to the runner — but the MCP tool
had no argument that reached them, and `directory` is deliberately
applied as the working directory rather than as a target.

Adds an optional `paths` array that is appended after every option, so
the args land in `rest`. Behaviour matches the CLI, including the
existing rule that targeting specific files disables test optimization.
@HadiHassan22
HadiHassan22 requested a review from a team as a code owner August 5, 2026 02:16
@jmadren

jmadren commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Some points to raise on this PR:

  1. paths is a footgun that reproduces the #1600 symptom. From a monorepo root with no pubspec.yaml, very_good dart test packages/a/test/a_test.dart exits 66 — the exact failure from #1599/#1600. An agent that puts a package path into paths instead of directory lands right back there. The description should say the paths are test targets inside the package selected by directory, and that a package path belongs in directory.

  2. recursive: true + paths is incoherent. TestCLIRunner spawns per package with workingDirectory: cwd and passes options.rest verbatim, so the same relative path is applied in every package. Measured: very_good dart test -r test/a_test.dart passes in packages/a, fails to load in packages/b, exit 69. Pre-existing CLI behavior, but the MCP surface now exposes both knobs side by side with no shell to make it obvious — and it contradicts the new arg's "relative to the project root" wording, which is only true when recursive is off.

  3. Minor: emitting ['--', ...paths] would be strictly more robust (a path beginning with - currently becomes a UsageException), and matches what isTargettingTestFiles documents. The -- is stripped from rest by the parser, so nothing downstream changes.

  4. Minor: doc/mcp.md's test example JSON enumerates every argument and wasn't updated.

  5. One inaccuracy in the existing codebase that the PR inherits rather than introduces: TestCLIRunner.isTargettingTestFiles's doc comment says "the user can only target test files by using the -- option terminator." That's false — allowTrailingOptions is on, so bare positionals work, which is exactly why the PR works at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: allow the test MCP tool to target specific paths

2 participants