Context
Surfaced by a commenter on the muteval↔tracelint dev.to post. deny_tool_output emits one fixed shape ({"status": "declined"}) keyed to /status, so pairing it with a failure_when on /status proves the rule matches the shape the mutant emits, not the shapes real providers emit — a fair critique. The next operator should mutate the failure's representation, not its value, to test the robustness of the failure contract itself.
What to add
A shape-mutation operator (or small family) over tool outputs:
- nested code:
{"outcome": {"code": "card_declined"}}
- alternate key / value scheme
- absent — a 200 with the failure field dropped entirely. This deserves its own operator: it's the mutant that reveals a contract which passes vacuously on a missing field.
Why absent matters
A failure_when/eval contract that keys on /status can't separate not-declined from not-present, so a dropped field passes clean — a green suite for exactly the regression the contract was written to catch. This is confirmed on the tracelint side (a declared failure_when whose pointer is absent currently passes clean instead of failing closed — filed separately in AshwinUgale/tracelint). The absent mutant is the muteval-side probe that surfaces that class of hole in a user's suite.
Acceptance criteria
- New operator(s) mutating a tool output's failure representation (nested, alternate-key, absent); one mutant per tool output per shape;
target="tools".
absent removes the declared failure field (or emits an unrelated-shape body).
- Registered in
OPERATORS (near deny_tool_output, src/muteval/mutators.py:623) and src/muteval/severity.py (HIGH — same class as deny/corrupt).
- Tests per operator.
Cross-reference
Two halves of one story: muteval injects the absent shape; tracelint must not pass it vacuously. Link the tracelint issue once filed.
Context
Surfaced by a commenter on the muteval↔tracelint dev.to post.
deny_tool_outputemits one fixed shape ({"status": "declined"}) keyed to/status, so pairing it with afailure_whenon/statusproves the rule matches the shape the mutant emits, not the shapes real providers emit — a fair critique. The next operator should mutate the failure's representation, not its value, to test the robustness of the failure contract itself.What to add
A shape-mutation operator (or small family) over tool outputs:
{"outcome": {"code": "card_declined"}}Why
absentmattersA
failure_when/eval contract that keys on/statuscan't separate not-declined from not-present, so a dropped field passes clean — a green suite for exactly the regression the contract was written to catch. This is confirmed on the tracelint side (a declaredfailure_whenwhose pointer is absent currently passes clean instead of failing closed — filed separately in AshwinUgale/tracelint). Theabsentmutant is the muteval-side probe that surfaces that class of hole in a user's suite.Acceptance criteria
target="tools".absentremoves the declared failure field (or emits an unrelated-shape body).OPERATORS(neardeny_tool_output,src/muteval/mutators.py:623) andsrc/muteval/severity.py(HIGH — same class as deny/corrupt).Cross-reference
Two halves of one story: muteval injects the absent shape; tracelint must not pass it vacuously. Link the tracelint issue once filed.