Maximum dimensions for optimisation

[taroko-mooncake]

Hi - I have read that bayesian inference struggles in high-dimensionality data, are there recommended limits on the number of dimensions / features that should be used in BoTorch optimisation?

[saitcakmak]

Hi @taroko-mooncake. The standard GP models (that are commonly used in BO) do not work well beyond 15-20 dimensions. However, we have other models that support high dimensional input spaces, such as SAASBO (paper: https://arxiv.org/abs/2103.00349) that @dme65 is currently working on enabling in BoTorch. You can readily use SAASBO via Ax. So, if you have a high dimensional search space, that might be a good place to start: https://ax.dev/tutorials/saasbo.html

[saitcakmak]

To add to that, TRBO is another approach we support that works in high dimensional input spaces: https://botorch.org/tutorials/turbo_1. It uses local models trained in trust regions to avoid the issues GPs face in large dimensions.

Papers on this are https://arxiv.org/pdf/1910.01739.pdf for single objective and https://arxiv.org/pdf/2109.10964.pdf for multi-objective optimization.

[Balandat]

Deep Kernel Learning (https://arxiv.org/abs/1511.02222) is another approach that can be easily implemented in BoTorch, but the feature extractor can be hard to fit, and optimizing the acquisition function for BO still needs to happen in the original dimensionality so this can be a limitation.