Question: explanation of fit_gpytorch_mll

[matthewcarbone]

In v0.7.2, the new recommended way to train gpytorch models is with fit_gpytorch_mll. However, it is really unclear what this function actually does. It does not expose many parameters, and its documentation (and even source code) are really confusing to understand.

Is it possible someone can point me in the right direction, or explain what fit_gpytorch_mll actually does? It's weird, this function seems to work better than any custom training loop I write, but I have no idea what it's doing or how to control it!

Thanks!

[Balandat]

What kind of model are you using? Have you used fit_gpytorch_model before?

@j-wilson can provide more detail on this, but at a high level fit_gpytorch_mll is essentially just a high level interface that dispatches to different model-specific fitting routines under the hood. So if you have used fit_gpytorch_model it's essentially doing the same as that for all intents and purposes, just with a different name.

[matthewcarbone]

What kind of model are you using?

SingleTaskGP

Have you used fit_gpytorch_model before?

Yes. To be totally honest it wasn't clear what that was doing either. But with fit_gpytorch_mll, there's some pretty unintuitive calls to a "Dispatcher" which I cannot make sense of. Figured this was as good a time to ask as any.

It is true that this is not necessarily information that I need to know specifically, but I would really like to know what types of fitting procedure fit_gpytorch_mll is performing. For example, I can write my own training loop with some number of epochs and whatnot, but the amount of control when using this function is seemingly quite minimal. Again, I'm not complaining, as this function seems to work better than anything I can do myself, but I would like to know what it's doing for when I, for example, write a paper.

[Balandat]

there's some pretty unintuitive calls to a "Dispatcher" which I cannot make sense of

This is basically implementing a multiple dispatch pattern, where you can register different implementations of the fitting procedure depending on the type of MLL, likelihood, and model at hand. @j-wilson can shed some more light on the specific implementation.

Again, I'm not complaining, as this function seems to work better than anything I can do myself, but I would like to know what it's doing

At a high level, what happens under the hood (here or in fit_gpytorch_model before) is that we use the L-BFGS-B algorithm implemented in scipy.minimize in order to find the MAP estimate for the model's hyperparameter. This is a quasi-second order optimization method can achieve much faster convergence than the stochastic first-order methods typically used e.g. for NN training. This is happening in fit_gpytorch_scipy which is called from the top level fit_gpytorch_mll (or previously fit_gpytorch_model).

[j-wilson]

Hi @x94carbone, happy to explain what I can. Below, I've provided some details on how the dispatcher works and outlined how parameters (here: keyword arguments) are exposed in the updated API.

1. Dispatcher

fit_gpytorch_mll uses multiple dispatching to determine "what to do" with a given mll. When called,fit_gpytorch_mll identifies mll instances according to the tuple type(mll), type(mll.likelihood), type(mll.model). This "signature" gets compared with the signatures used to register subroutines via the dispatcher.register mechanic. The dispatcher is responsible* for comparing these signatures and determining the most appropriate choice of subroutine for a given mll.

If the mll's signature matches a registered subroutine foo's signature exactly, then dispatcher will internally call foo while passing the positional arguments it received (whose types constitute the signature) as well as any keywords. More generally, the choice of subroutine is determined by traversing through set of registered subroutines from least to most generic (as dictated by class hierarchies) until a compatible method is found. Here, compatibility is assessed by checking whether the positional arguments passed to dispatcher subclass a subroutine's signature types .

MDNotImplementedError is a special exception that MD subroutines can throw to tell the dispatcher that they "give up". The dispatcher will intercept this error and proceed to call the next-most-specific, compatible subroutine.

*The dispatcher is usually responsible for calling type on its positional arguments to produce the signature tuples described above. We've modified things slightly to allow types to be passed in directly and done that here to make it clear that we always use mll.model and mll.likelihood rather than the arguments.

2. Exposing parameters

fit_gpytorch_mll actually exposes all** the same keyword arguments as the original fit_gyptorch_model. These get passed down to the various subroutines by the dispatcher. The main difference is that, now, any keyword arguments the user would like to pass down to optimizer should be supplied via the new optimizer_kwargs: Dict[str, Any] keyword.

Additionally, note that each subroutine is responsible for documenting its own keyword arguments and similarly for optimizer and optimizer_kwargs.

**max_retries was renamed max_attempts to avoid confusion.

[matthewcarbone]

Hi all, that really helps, thank you. IMO you should consider adding some of this text into the documentation. I can't be the only one who is curious!