which metrics are appropriate for transformations?

[nickreich]

Here’s a question that we’ve run into as we’re working on adding metrics computed on a transformed scale to the standard hubverse dashboards:

• it makes sense to compute some metrics on, say, the log scale. Like, WIS or CRPS or MSE or MAE.
• But it doesn’t make sense (although it’s not exactly “wrong”) to do transformations when computing, say, interval coverage. It would just be the same number as when not transforming.
• my understanding is that scoringutils allows for all metrics to be computed on transformed data.
• Is there some standard way to think about which metrics it “makes sense” (or doesn’t) to evaluate in the context of specific transformations?
• In a practical sense, we’re “turning on” transformed scores for all metrics except for interval coverage (and maybe bias) right now on the hub dashboards, but were trying to figure out if there were a cleaner way to determine which metrics available in scoringutils would make sense (or not) on a transformed scale.

[nikosbosse]

Hmm. We could document it in the docs for the built-in scores. The next level up would be tagging the built-in scores somehow. But ultimately I'm not sure there really is anything sensible we could do other than amending the docs.
cc @sbfnk @seabbs