Skip to content

Add a tool-failure shape-mutation operator (nested / alternate-key / absent) #45

Description

@AshwinUgale

Context

Surfaced by a commenter on the muteval↔tracelint dev.to post. deny_tool_output emits one fixed shape ({"status": "declined"}) keyed to /status, so pairing it with a failure_when on /status proves the rule matches the shape the mutant emits, not the shapes real providers emit — a fair critique. The next operator should mutate the failure's representation, not its value, to test the robustness of the failure contract itself.

What to add

A shape-mutation operator (or small family) over tool outputs:

  • nested code: {"outcome": {"code": "card_declined"}}
  • alternate key / value scheme
  • absent — a 200 with the failure field dropped entirely. This deserves its own operator: it's the mutant that reveals a contract which passes vacuously on a missing field.

Why absent matters

A failure_when/eval contract that keys on /status can't separate not-declined from not-present, so a dropped field passes clean — a green suite for exactly the regression the contract was written to catch. This is confirmed on the tracelint side (a declared failure_when whose pointer is absent currently passes clean instead of failing closed — filed separately in AshwinUgale/tracelint). The absent mutant is the muteval-side probe that surfaces that class of hole in a user's suite.

Acceptance criteria

  • New operator(s) mutating a tool output's failure representation (nested, alternate-key, absent); one mutant per tool output per shape; target="tools".
  • absent removes the declared failure field (or emits an unrelated-shape body).
  • Registered in OPERATORS (near deny_tool_output, src/muteval/mutators.py:623) and src/muteval/severity.py (HIGH — same class as deny/corrupt).
  • Tests per operator.

Cross-reference

Two halves of one story: muteval injects the absent shape; tracelint must not pass it vacuously. Link the tracelint issue once filed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions