torch.from_numpy throws error from pytorch_ops.backward

[Aharonez]

Hey,
We've been trying to use this code in catboost and got the following error:
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
from this line:
torch.from_numpy(ctx.numpy_obj.vjp(grad_output.numpy()))

If we'll change to tensor.detach the error will resolved but may broke something else within pytorch.

Any idea on how to solve it?

Thanks!