Add support for soft performance failures

[vkarak]

It is often the case that a performance test fails a reference threshold marginally and transiently. The test is marked as a failure, although in reality it is not a real failure. The suggestion is to allow tests introducing a halo region outside the threshold where a test would pass with a warning.

[victorusu]

I understand the motivation behind this request. The current performance threshold model in ReFrame is effectively a hard decision model — a step function. A value is either inside the reference bounds (pass) or outside (fail), with no intermediate state.

The proposal introduces something analogous to a switching function (borrowing MD terminology): instead of a hard boundary, we define a halo region where the test passes with a warning. Just like in Molecular Dynamics, where hard step functions introduce spurious forces that are artifacts of the model, hard performance thresholds can introduce spurious failures — marginal, transient deviations that create noise and operational toil.

Importantly, the pass/fail decision is not purely an engineering property of the system; it is contextual and business-driven. Providing a more expressive decision model enables new use cases and better aligns with operational reality. I am fully in favor of this feature.

That said, it has implications for dependency semantics. Today, the behavior is clear: if a test fails, its dependents fail. With a “pass with warning” state, the test technically passes — but semantically it is not equivalent to a clean pass. If we do not pay attention, dependent tests will have no way of knowing whether a dependency passed cleanly or within the halo region.

If we introduce this new state, I think we should also consider exposing it through the API, so that dependency resolution (or user logic) can optionally take it into account.

Overall, +1 from my side — with some thought needed around dependency semantics and API visibility of this new outcome state.