test(evals): Migrate evals to vitest harnesses

[dcramer]

Move the eval suite onto the current vitest-evals harness API and remove the legacy scorer/task-runner utilities. The eval specs now use shared helpers for prediction, full MCP tool-call runs, and embedded search-agent runs, which keeps traces, usage data, and report metadata centralized.

GitHub Eval Reporting

The eval workflow now writes JSON/JUnit output and publishes the Evaluation Results check with getsentry/vitest-evals@v0. Root shortcuts also serve the last JSON result with the local report UI.

Harness Documentation

Refresh the eval README, testing guide, adding-tools guide, and PR guidance so new evals stay harness-first and avoid reintroducing legacy boilerplate.

[sentry-warden]

Eval workflow never triggers on changes inside packages/mcp-core/src/tools/ subdirectories

The path filter packages/mcp-core/src/tools* uses a bare * which GitHub Actions does not allow to match /, so edits to any file under packages/mcp-core/src/tools/ (e.g. tools/support/search-events/agent.ts) will silently skip the eval run; change the filter to packages/mcp-core/src/tools/**.

Evidence

• GitHub Actions path filters use minimatch semantics: * matches any char except /, so packages/mcp-core/src/tools* matches only paths whose name begins with tools directly in src/ (e.g. a hypothetical src/toolsHelpers.ts), not src/tools/....
• packages/mcp-core/src/tools/ contains support/, search-events/, search-issues/, etc.; all agent files live one or more levels below tools/.
• This PR itself changes packages/mcp-core/src/tools/support/search-events/agent.ts, which would not match the filter and would not trigger the eval workflow.
• No other workflow in .github/workflows/ has a path filter covering packages/mcp-core/src/tools/**, so these changes go completely unchecked by evals on push/PR.

_{Identified by Warden code-review}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 417ecf4. Configure here.}

[sentry-warden]

get_sentry_resource breadcrumbs description claims event ID support the implementation does not provide

The tool surface advertises - breadcrumbs: issue shortId or event ID (and the resourceId param description repeats "issue shortId or event ID for breadcrumbs"), but the breadcrumbs path only accepts an issue identifier. resolveResourceParams sets issueId: resourceId.toUpperCase() and the handler calls fetchAndFormatBreadcrumbs(...) -> getLatestEventForIssue({ issueId }), which hits /organizations/<org>/issues/<issueId>/events/latest/. An event ID placed in the <issueId> path position will not resolve, and even when a specific event URL is supplied the handler always fetches the issue's latest event. The description therefore overstates supported inputs, which can mislead the model into passing event IDs that fail. This is a tool-definition/metadata accuracy issue, not a functional security defect.

Evidence

• resolveResourceParams breadcrumbs case (get-sentry-resource.ts:164-169) sets issueId: resourceId.toUpperCase(); there is no event-ID branch or issue lookup from an event ID.
• Handler routes to fetchAndFormatBreadcrumbs(apiService, organizationSlug, resolved.issueId!) (get-sentry-resource.ts ~692).
• fetchAndFormatBreadcrumbs calls getLatestEventForIssue({ organizationSlug, issueId }) (breadcrumbs.ts:11) which delegates to getEventForIssue with eventId: "latest", path /organizations/${org}/issues/${issueId}/events/latest/ (client.ts:2124,2205-2210) — issueId must be an issue shortId/numeric ID.
• Description text lives at get-sentry-resource.ts:540 and :587 and is mirrored into the generated wire surface (toolDefinitions.json:931,957; skillDefinitions.json) which all repeat "issue shortId or event ID" for breadcrumbs.

_{Identified by Warden mcp-audit · 5H6-MUT}