Standardization vs Min-Max Normalization for Data Scaling

[m-julian]

Hello,

I am curious as to why standardization is recommended for both inputs and outputs for GP regression over other scaling methods such as min-max normalization. If training outputs are standardized, they have zero mean, so a zero mean GP could be used, after which test predictions can be inverse transformed (using the mean and standard deviation of the training data) to get the "true" predicted values. For inputs, it makes sense to scale the data in some way to remove the units associated with each feature, but both standardization and min-max normalization make the data unitless.

Could you use other methods such as min-max normalization to scale the data instead? For example, could you use min-max normalization for the inputs and standardize the outputs? Would using a scaling method other than standardization impact model performance significantly? I have not found many discussions on scaling data for GPs specifically, the best one I've found is this: https://stats.stackexchange.com/q/178245/317677, but it only covers standardization.

Any discussion on data scaling for GPs would be greatly appreciated, thanks!

[wjmaddox]

Yes, min-max normalization for inputs to [0, 1]^d and standardization of outputs is probably a good idea for numerical stability. For example, these types of transforms are implemented in BoTorch here for example, where Normalize(d = ...) is min-max normalization and Standardize(m=...) is standardization.

[m-julian]

Thanks for the link, I've implemented these transforms myself because I didn't know they were already implemented in BoTorch.

Is there a reason why only standardization is recommended though? I've only seen standardization done in the GPyTorch docs and in github issues here. Are default hyperparameter values better suited for standardization or is it something else that makes standardization better for GPs (for both inputs and outputs)?

[wjmaddox]

Yes 1) can be tricky but with 2) there's probably a way to adjust the period or to do an input transform to take that into account to properly upscale / downscale the period as necessary.

The larger "gotcha" so to speak is that your parameters (aka kernel lengthscales) will need to be transformed into the original scale if you want to interpret them in a "physical" manner.

[wjmaddox]

Also, one day, we will eventually move the transforms back into gpytorch and deal with 1) automatically, see #1652

[m-julian]

Great, thanks, appreciate the help!

[patel-zeel]

Hi @wjmaddox and @m-julian,

1) is straightforward for predictive mean transformation but what do we need to do to inverse transform the predictive covariance matrix or at least predictive variance? Is there a straightforward way to do this as well?

[wjmaddox]

Yes, you'll also need to back transform the predictive mean and variance. For standardization you need to just follow variance / stddev properties: $\sigma(c (X + b) ) = |c| \sigma(X)$. An example is here: https://github.com/pytorch/botorch/blob/3de2ccd55b41a1ca341772a6a2eb29ce787ecd16/botorch/models/transforms/outcome.py#L294 .