Should be obsevation noizes normalized (standalized) or not?

[skiyohara]

Dear botorch developers and users,

I'm using FixedNoiseGP for Bayesian optimization with keeping the observation noise (train_Yvar) as zero (1e-6).
However, I'm wondering if I should the noise values 0 or not because 0 corresponds to a certain value ( a mean of the training targets due to the normalization of the training targets (train_Y) in my case.

Any help will be appreciated.

Shin

[sdaulton]

For numerical stability, you likely don't want to set the noise below 1e-6 in the transformed (e.g. normalized) outcome space. 1e-6 is the lower bound that we use in Ax for the noise level on the transformed outcomes: https://github.com/facebook/Ax/blob/5c3529ca0a8ccc8868ddb13e0669825fd427cdeb/ax/models/torch/botorch_defaults.py#L49

[skiyohara]

Thank you for your answers.
I know that 1e-6 is the lower bound, but the value is not 1e-6 when inversely normalized in my case. If I set the noise as 1e-6, this means that the noise is nearly equal to the mean of the normalized traininig targets (mean:0, standard deviation: 1), which leads to the assumption of a certain noise (not 1e-6). If I want to set the noise as 1e-6 (nearly zero), should I normalize the value (1e-6) beforehand in the same way as the training data and use the normalized value as the input noise for FixedNoiseGP?

[sdaulton]

As we do in Ax, I would recommend standardizing the training targets to be zero mean, standard deviation 1, and then setting the noise variance to be 1e-6 (in the standardized outcome space). In the unstandardized outcome space, the noise variance would be 1e-6 * sigma2, where sigma2 is the variance of the raw training targets.