Validation of exploration capability of BO

[Obs01ete]

Hi the comunity! Is there any known way to validate the capability of bayesian optimization's recommendations on a historic dataset? BO is normally used in the loop with the black-box function evaluations. However if I have no access to the black-box function and only have a significant historic record of samples, is it possible to somehow assess the exploration capability of my specific implementation (with specific hyperparameters like lenghscale and UCB beta)? I could do folded cross-validation on the dataset, but this will only assess the quality of GP fit. The difficulty is with the acquisition function. How to assess that it is correct and the UCB beta is good. There is no way to query the acquisition function maximizer, but I can check GP values and the UCB value on the hold-out top-10% of the dataset samples by the (scalar) objective. Hower I am puzzeld how I can combine these values into a single robust metric. I'd appreciate your help!

[saitcakmak]

Hi @Obs01ete. To compare various methods on a given dataset, we typically fit a surrogate model on the dataset and run full BO loops to optimize the surrogate model. The surrogate can be any statistical model that interpolates the dataset to the full search space. Using a GP model is a convenient choice but it may not be great for comparing various length-scales (since the surrogate itself has a length-scale).

Is there a particular reason that you're using UCB (and trying to tune beta) rather than using some other acquisition function that doesn't require a tuning a hyper-parameter? We find that EI and its variants generally work much better than UCB. @SebastianAment recently introduced LogEI, which eliminates some numerical issues and performs quite a bit better.

[Obs01ete]

Hi @saitcakmak. Thanks for your answer. It is good to know that the approach with the surrogate model is the state of the art. I think I will try k-NN and gradient boosting regression as a surrogate model. k-NN needs scales for dimensions though, so it may be as inappropriate as GP.

I use UCB because of its simplicity, better interpretability and the beta parameter allowing to control exploration, which is very important for my problem. With EI it is trickier to get exploration parameter zeta right as mentioned in this paper. Maybe I shoould look into EI variants.

Hi @SebastianAment. Great to know you work on better acquisition functions! Please share a link to LogEI paper or BoTorch implementation if any. My quick search over the internet did not point to it.

[Obs01ete]

@saitcakmak thanks for recommending full simulated BO cycles, it works for me. I use sklearn boosting as a surrogate model.