Input and Output transfrom

[SimonHerter]

Hi,
i'm currently testing to implement a custom GP model, i used the same setup as in the tutorial from the website (https://botorch.org/tutorials/custom_botorch_model_in_ax).

class SimpleCustomGP(ExactGP, GPyTorchModel):

_num_outputs = 1  # to inform GPyTorchModel API
def __init__(self, train_X, train_Y):
    # squeeze output dim before passing train_Y to ExactGP
    super().__init__(train_X, train_Y.squeeze(-1), GaussianLikelihood())
    self.mean_module = ConstantMean()
    self.covar_module = ScaleKernel(
        base_kernel=RBFKernel(ard_num_dims=train_X.shape[-1]),
    )
    self.to(train_X)  # make sure we're on the right device/dtype
    
def forward(self, x):
    mean_x = self.mean_module(x)
    covar_x = self.covar_module(x)
    return MultivariateNormal(mean_x, covar_x)

My question relates to the normalization of the input samples (train_X) and the standardization of the output samples (train_Y). This procedure is recommended in the docs (https://botorch.org/docs/models). I know that Botorch internally perform the normalization and standardization for the predefined models like the SingleTaskGP (as far as I understood).
Is the normalization and standardization also performed automatically for the SimpleCustomGP above? Or do i need to add that to the definition like (the scikit learn Normalizer and Standardscaler are used in this example):

class SimpleCustomGP(ExactGP, GPyTorchModel):

_num_outputs = 1  # to inform GPyTorchModel API   
def __init__(self, train_X, train_Y,GaussianLikelihood()):
    # squeeze output dim before passing train_Y to ExactGP   
    train_X=torch.tensor(Normalizer().fit_transform(train_X))
    train_Y=torch.tensor(StandardScaler().fit_transform(train_Y.reshape(-1, 1))).squeeze(-1)
    super().__init__(train_X, train_Y.squeeze(-1), likelihood)
    self.mean_module = ConstantMean()
    self.covar_module = ScaleKernel(base_kernel=RBFKernel(ard_num_dims=train_X.shape[-1]))
    self.to(train_X)  # make sure we're on the right device/dtype        
def forward(self, x):
    mean_x = self.mean_module(x)
    covar_x = self.covar_module(x)
    return MultivariateNormal(mean_x, covar_x)

[Balandat]

Hi, great question. When using Ax, a lot of the work for normalizing inputs and standardizing outputs is actually done by Ax itself in the modelbridge layer, so you don't really have to do this again in BoTorch. In particular, the default Ax transforms for the get_botorch factory do include UnitX and StandardizeY, which take care of this.

By the way, that tutorial is a bit stale (we're working on updating it), you should check out the new "modular botorch model" setup that makes hooking new botorch models and acquisition functions into Ax a lot easier and more modular (rather than having to specify constructor callables): https://ax.dev/tutorials/modular_botax.html

[SimonHerter]

Hi,
thanks for the quick reply, acutally i'm not using Ax. My pipeline is completly build in Botorch.
I pass my SimpleCustomGP from above to:

def _get_and_fit_simple_custom_gp(Xs, Ys, **kwargs):
    model = SimpleCustomGP(Xs, Ys,likelihood)
    mll = ExactMarginalLogLikelihood(model.likelihood, model)
    fit_gpytorch_model(mll)
    return model

model=_get_and_fit_simple_custom_gp(init_x,init_y)

After that i use the model in the ExpectedImprovement acquisition function and pass it to the optimize_acqf function. Then i'm looping over that setup. I tested this configuration with an easy example (just 2 Variables and 1 objective), i'm also aware of the best configuration. Somehow i only obtained reasonable results without the normalization and standardization. That's the reason why i thought that it's done internally. Nevertheless i'm confused as the normalization and standardization via the scikit library is decreasing the performance of the Optimization. I think i missing some important detail, it would be great if you can share your thoughts with me.

[saitcakmak]

Somehow i only obtained reasonable results without the normalization and standardization.

The way you're using scipy to normalize / standardize the inputs is quite different than how it is typically done in BoTorch. Your approach above only does the transformation at the model initialization. For transforms to work properly, you'd have to normalize your test inputs, and un-standardize the model outputs as well. Luckily, BoTorch has built-in support for doing these via the InputTransform and OutcomeTransform modules. Below shows how to modify your custom model to support these. I'd also recommend studying the SingleTaskGP (https://github.com/pytorch/botorch/blob/main/botorch/models/gp_regression.py) implementation for better understanding.

class SimpleCustomGP(ExactGP, GPyTorchModel):
    _num_outputs = 1  # to inform GPyTorchModel API
    def __init__(
        self, train_X, train_Y,
        outcome_transform: Optional[OutcomeTransform] = None,
        input_transform: Optional[InputTransform] = None,
    ):
        # Transform the outcomes before passing them to ExactGP. Inputs are transferred at `forward` and `posterior` calls.
        # The outcomes will get untransformed at `posterior` call as well.
        # So, you can use the model with X, Y from the original space while all the transforms get handled under the hood.
        if outcome_transform is not None:
            train_Y, _ = outcome_transform(train_Y)
        # squeeze output dim before passing train_Y to ExactGP
        # If you subclass BatchedMultiOutputGPyTorchModel, you can instead use _transform_tensor_args, which does additional things to support multiple outputs
        super().__init__(train_X, train_Y.squeeze(-1), GaussianLikelihood())
        self.mean_module = ConstantMean()
        self.covar_module = ScaleKernel(
            base_kernel=RBFKernel(ard_num_dims=train_X.shape[-1]),
        )
        if outcome_transform is not None:
            self.outcome_transform = outcome_transform
        if input_transform is not None:
            self.input_transform = input_transform
        self.to(train_X)  # make sure we're on the right device/dtype
        
    def forward(self, x):
        if self.training:
            x = self.transform_inputs(x)
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return MultivariateNormal(mean_x, covar_x)

Now that your model supports transforms, you can use normalization and standardization as follows:

from botorch.models.transforms import Normalize, Standardize
model = SimpleCustomGP(
    train_X, train_Y, likelihood, outcome_transform=Standardize(m=train_Y.shape[-1]), input_transform=Normalize(d=train_X.shape[-1])
)
mll = ExactMarginalLogLikelihood(model.likelihood, model)
fit_gpytorch_model(mll)