Best practices: Use double precision in BoTorch!

[saitcakmak]

The use of single precision arithmetic, i.e., the use of default torch.float32 tensors, commonly lead to numerical issues when working with Gaussian Processes. For example, when the training inputs (train_X) are clustered closely in the search space, the resulting covariance matrix ends up being (numerically) singular, leading to errors while computing the Cholesky factorization. To help address these numerical issues, we strongly recommend using double precision in BoTorch.

All you need to do to use double precision is to add dtype=torch.double in the tensor constructors.

torch.tensor([1.0, 2.0], dtype=torch.double)
torch.rand(2, 4, dtype=torch.double)
torch.ones(5, 3, dtype=torch.double, device=torch.device("cuda"))
...

[K-Kielak]

Can confirm, a lot problems and hard to debug issues go away when using BoTorch in double precision!

To add to that, if you already have a lot of code that does not explicitly set dtype, the easiest way to migrate to double precision is to do torch.set_default_dtype(torch.float64) before any of the tensors are defined.

[aegonwolf]

Hi @saitcakmak , as a question, when I was using float32 I was being directed here, now I've changed but have warnings of numerical instability (negative variances).
Did I have these before but the warnings were suppressed?
I also get more scipy warnings of non p.s.d matrices which I do not get with single precision.

[saitcakmak]

Hi @aegonwolf! Sorry for the late response here. The warning about negative variances does not necessarily mean that the variances are negative. GPyTorch checks if any of the variances is less than some value and raises a warning. The default for double is 1e-10: https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/settings.py#L296-L308

If you don't see any other warnings / errors, then it should be ok to ignore these. In either case, having these very small values is not great since it means you're close to limits of numerical precision, so scaling up the function values might be worth a try.

[ZoeLee0220]

@saitcakmak Hello! I'm a beginner in BoToRCH and in I'm using the learning document https://botorch.org/docs/getting_started的第一个例子就被引导到这个GitHub页面 so I was wondering if you could tell me how this double precision change should be modified, f thanks a lot!

[ZoeLee0220]

I'm sorry to have disturbed you! I have solved the problem