feat: use MCP conformance baseline instead of scenario allowlist

[trepel]

Summary

Move the MCP conformance task from a hardcoded per-scenario loop to a single --suite all run against the 2025-11-25 spec

Changes

Bump to conformance tests v0.1.16

Add parameter to specify a baseline file - a file with scenario names that are expected to fail (at least one check in that scenario ends up with failure or warning)

Move the MCP conformance task from a hardcoded per-scenario loop to a single --suite all run against the 2025-11-25 spec and ignore expected failures based on baseline file

Update reporting to RP so that each reported test entry contains both scenario name and unique check-id. Previously the entries only contained scenario name so if scenario contained more that one check there were mutiple entries reported with the same name in RP.

Added logic to reporting so that scenarios from baseline file are treated as skipped in RP

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 75261760-e6c7-4191-abc4-763baf860597

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}