Hi,
I’ve been experimenting with implementing a new metric based on the linear predictor (enabled by #551, thanks again for this work).
While testing it, I noticed a discrepancy between:
- the value returned by
collect_metrics() after last_fit(), and
- the value obtained by manually calling my metric function on
stats::predict(type = "lp").
I’m fairly confident (99.9%) that this is due to the fact that last_fit() ultimately calls parsnip::predict() with the default increasing = TRUE argument, which flips the direction of the linear predictor.
This behavior is not visible for royston_survival() since it is direction-invariant. However, for metrics that depend on the sign of the linear predictor, it’s a bit annoying that the result depends on whether you compute it via last_fit() or by hand.
Would it be possible to expose an option in new_linear_pred_survival_metric() to control the increasing argument used internally by last_fit()? Alternatively, is there a recommended way to ensure consistency between the two approaches?
Thanks again for the ongoing work on this infrastructure.
Hi,
I’ve been experimenting with implementing a new metric based on the linear predictor (enabled by #551, thanks again for this work).
While testing it, I noticed a discrepancy between:
collect_metrics()afterlast_fit(), andstats::predict(type = "lp").I’m fairly confident (99.9%) that this is due to the fact that
last_fit()ultimately callsparsnip::predict()with the defaultincreasing = TRUEargument, which flips the direction of the linear predictor.This behavior is not visible for
royston_survival()since it is direction-invariant. However, for metrics that depend on the sign of the linear predictor, it’s a bit annoying that the result depends on whether you compute it vialast_fit()or by hand.Would it be possible to expose an option in
new_linear_pred_survival_metric()to control theincreasingargument used internally bylast_fit()? Alternatively, is there a recommended way to ensure consistency between the two approaches?Thanks again for the ongoing work on this infrastructure.