Variational GP with derivatives and monotonic gp

[ankushaggarwal]

I have implemented an Indexed version of VariationalStrategy that allows us to use derivative information. I have used it to create a monotonic GP with a composite likelihood. The new variational strategy is called VariationalStrategyIndexed, but it could be easily merged with the normal VariationalStrategy (I can do that if that is preffered).

I have also provided an example on how to use it. One minor issue though, as raised in a bug report #1554, the slicing raises a spurious error. To suppress it, we have to set gpytorch.linear_operator.settings.debug._default = False. This pull request is related to #1639 (comment)

[gpleiss]

Sorry for the slow reply @ankushaggarwal . I'm a little hesitant about adding this PR for a few reasons. 1) I'm not sure how re-usable VariationalStrategyIndex is, and I think it's also too specific to add to VariationStrategy proper. 2) I'm not sure this is the best/correct way to use derivative information with variational models. Derivative observations qualify as an inter-domain GP, and for variational GPs its best to factorize over the derivative and non-derivative observations (see https://arxiv.org/pdf/2003.01115.pdf for more details).

[ankushaggarwal]

Sorry for the slow reply @ankushaggarwal . I'm a little hesitant about adding this PR for a few reasons. 1) I'm not sure how re-usable VariationalStrategyIndex is, and I think it's also too specific to add to VariationStrategy proper. 2) I'm not sure this is the best/correct way to use derivative information with variational models. Derivative observations qualify as an inter-domain GP, and for variational GPs its best to factorize over the derivative and non-derivative observations (see https://arxiv.org/pdf/2003.01115.pdf for more details).

Thank you @gpleiss for your response. No problem. Since we needed a monotonic GP which needs variational GP with Bernouille likelihood for derivatives (as pointed out by you in #1639 (comment)). The current implementation by me works for us in practice, but we would be very much interested in the variational GPs that handle derivatives through factorization over the derivative and non-derivative observations. So please let me know when this gets implemented in gpytorch, and we would love to use this more suitable approach.