Multiple-output GP using GPytorch

[cghernan]

I am trying to train a Gaussian Process for my data below, following the basic tutorial https://docs.gpytorch.ai/en/stable/examples/01_Exact_GPs/Simple_GP_Regression.html

train_x=tensor of [4058, 12]
train_y=tensor of [4058, 140]

I get an error on the loss calculation:

loss = -mll(output, train_y)

saying that output (model(train_x)) and train_y don't have the same dimension. Given this, I tried MultitaskGPModel https://docs.gpytorch.ai/en/stable/examples/03_Multitask_Exact_GPs/Multitask_GP_Regression.html getting a very similar error

RuntimeError: The size of tensor a (568120) must match the size of tensor b (8116) at non-singleton dimension 0

Apparently MultitaskGPModel requires the same total number of entries to be equal. Is there a way to train a multiple-entry GP?

[supersjgk]

@cghernan I'm not getting any such errors following your input dimensions.
In MultitaskGPModel, make sure to set num_tasks = 140 in MultitaskMean, MultitaskKernel and MultitaskGaussianLikelihood.
If you still get errors, post your complete code.

[cghernan]

import pdb
import gpytorch

class MultitaskGPModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(MultitaskGPModel, self).__init__(train_x, train_y, likelihood)
        self.mean_module = gpytorch.means.MultitaskMean(
            gpytorch.means.ConstantMean(), num_tasks=140
        )
        self.covar_module = gpytorch.kernels.MultitaskKernel(
            gpytorch.kernels.RBFKernel(), num_tasks=140, rank=1
        )

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultitaskMultivariateNormal(mean_x, covar_x)


LR_INIT = 1e-3
# initialize likelihood and model
    likelihood = gpytorch.likelihoods.MultitaskGaussianLikelihood(num_tasks=140)
    model = MultitaskGPModel(train_x, train_y, likelihood)

    # Find optimal model hyperparameters
    model.train()
    likelihood.train()

    # Use the adam optimizer
    optimizer = torch.optim.Adam(model.parameters(), lr=LR_INIT) 

    # "Loss" for GPs - the marginal log likelihood
    mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)
    training_iter = 50
    for i in range(training_iter):
        # Zero gradients from previous iteration
        optimizer.zero_grad()
        # Output from model
        output = model(train_x)
        # Calc loss and backprop gradients
        loss = -mll(output, train_y)
        loss.backward()
        print('Iter %d/%d - Loss: %.3f   lengthscale: %.3f   noise: %.3f' % (
            i + 1, training_iter, loss.item(),
            model.covar_module.base_kernel.lengthscale.item(),
            model.likelihood.noise.item()
        ))
        optimizer.step()

The error that I get is still on loss = -mll(output, train_y):

  File "C:\Users\cg1116\Desktop\Python\example.py", line 167, in main
    loss = -mll(output, train_y)
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\gpytorch\module.py", line 31, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\gpytorch\mlls\exact_marginal_log_likelihood.py", line 64, in forward
    res = output.log_prob(target)
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\gpytorch\distributions\multitask_multivariate_normal.py", line 212, in log_prob
    return super().log_prob(value.reshape(*value.shape[:-2], -1))
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\gpytorch\distributions\multivariate_normal.py", line 193, in log_prob
    inv_quad, logdet = covar.inv_quad_logdet(inv_quad_rhs=diff.unsqueeze(-1), logdet=True)
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\linear_operator\operators\kronecker_product_added_diag_linear_operator.py", line 78, in inv_quad_logdet
    logdet_term = self._logdet() if logdet else None
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\linear_operator\operators\kronecker_product_added_diag_linear_operator.py", line 118, in _logdet
    _, evals_plus_i, _ = _symmetrize_kpadlt_constructor(lt, dlt)
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\linear_operator\operators\kronecker_product_added_diag_linear_operator.py", line 40, in _symmetrize_kpadlt_constructor
    evals, evecs = symm_prod.diagonalization()
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\linear_operator\operators\kronecker_product_linear_operator.py", line 148, in diagonalization
    return super().diagonalization(method=method)
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\linear_operator\utils\memoize.py", line 59, in g
    return _add_to_cache(self, cache_name, method(self, *args, **kwargs), *args, kwargs_pkl=kwargs_pkl)
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\linear_operator\operators\_linear_operator.py", line 1453, in diagonalization
    evals, evecs = self._symeig(eigenvectors=True)
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\linear_operator\operators\kronecker_product_linear_operator.py", line 338, in _symeig
    evals_, evecs_ = lt._symeig(eigenvectors=eigenvectors)
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\linear_operator\operators\_linear_operator.py", line 886, in _symeig
    evals, evecs = torch.linalg.eigh(self.to_dense().to(dtype=settings._linalg_dtype_symeig.value()))
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\linear_operator\utils\memoize.py", line 59, in g
    return _add_to_cache(self, cache_name, method(self, *args, **kwargs), *args, kwargs_pkl=kwargs_pkl)
  File "C:\Users\cg1116\AppData\Local\anaconda3\envs\pmor3\lib\site-packages\linear_operator\operators\matmul_linear_operator.py", line 132, in to_dense
    return torch.matmul(self.left_linear_op.to_dense(), self.right_linear_op.to_dense())
RuntimeError: expected m1 and m2 to have the same dtype, but got: double != float

I think it's because train_x is type torch.float64 and output is MultitaskMultivariateNormal?

[supersjgk]

How exactly are you defining your train_x, train_y and test_x?

[cghernan]

train_x and train_y are matrices of the above size. For example: train_x.dtype gives torch.float64 and train_x.shape gives torch.Size([4058, 12])