Difference between KroneckerMultiTaskGP and MultiTaskGP. In which situations to use which of these?

[Sam4896]

Hi,

1. I am very new to MTGP and need it for modelling geometry of glass product. I wish to know what is the difference between the KroneckerMTGP and just MTGP? When to use either of the two? The paper mentioned in the MTGP, they also use Kronecker product. So I am not able to understand the difference between BoTorch's KroneckerMTGP and MTGP.
2. In KroneckerMTGP, the task covar_module is defined using an index kernel. How are the parameters of the B and v learned? When comparing to the general ICM model, the index kernel looks very different. How are the latent functions for the multi-task kernel sampled? Also, where are the latent functions being learned?
3. In MTGP, what is the meaning of task feature? In the documentation, it is written as

The index of the task feature (-d <= task_feature <= d).

Could you please also explain, how to use the MTGP class?

[saitcakmak]

Hi @Sam4896. The MultiTaskGP is the basic model. It uses two kernels, a base kernel over the non-task input features (exactly the kernel you'd use in a GP that ignored the tasks) and a task kernel over the task features.

In MTGP, what is the meaning of task feature?

It seems that the argument naming / documentation is a bit confusing here. The task_feature argument of MultiTaskGP refers to the index of the task feature values in train_X. The train_X is n x (d + 1) where d is the # of non-task features (the inputs you'd have in a non-multi task model) and the +1 is for the task feature. If I index into it like train_X[..., task_feature], I should only get the values of the task features. If you format train_X like (x0, 0), (x1, 0), (x2, 1), ... where the task feature is always added at the end, then the task_feature argument would be -1.

The task feature is the index of a given task. If I have two tasks, the task feature would be 0 & 1, with the train_X taking values like (x0, 0), (x1, 0), (x2, 1), ... (where xi represents the non-task features).

The paper mentioned in the MTGP, they also use Kronecker product.

The Kronecker product used here is just between the base kernel and the task kernel. It's just a fancy way to say that Cov((x0, t0), (x1, t1)) = Cov_base(x0, x1) * Cov_task(t0, t1). This is true for both MultiTaskGP and KroneckerMultiTaskGP.

what is the difference between the KroneckerMTGP and just MTGP?

I can't claim to understand this in detail either, so take this with a grain of salt. On the surface, the two models are quite similar. However, there are key differences in input requirements and how the posterior is computed. For MTGP, we can have arbitrary observations from each task, like (x0, 0), (x1, 0), (x2, 1), ... but this is not allowed in KroneckerMTGP. There, we need block-design observations like (x0, 0), (x0, 1), (x1, 0), (x1, 1), ... where all xi are observed for all tasks. This enables highly efficient computational tricks that makes the model much more scalable. Whereas the MTGP would get quite slow after a couple hundred observations, KroneckerMTGP can scale up to tens of thousands of observations. You can refer to the paper for more details on this: https://arxiv.org/abs/2106.12997 cc @Balandat

In KroneckerMTGP, the task covar_module is defined using an index kernel. How are the parameters of the B and v learned? When comparing to the general ICM model, the index kernel looks very different. How are the latent functions for the multi-task kernel sampled? Also, where are the latent functions being learned?

Both MTGP & KroneckerMTGP use an IndexKernel for the task kernel. Assuming that B and v refer to the hyper-parameters of the IndexKernel, they'd be trained along with other model hyper parameters during model fitting. These are initialized using torch.randn and the IndexKernel does not use any latent functions besides these.

Could you please also explain, how to use the MTGP class?

You can refer to this tutorial for an example: https://botorch.org/tutorials/composite_mtbo

[Sam4896]

Hi @saitcakmak
Thanks for the reply. I have another question related to the answer that you provided.

1. In the IndexKernel, there are two hyperparameters 'B' and 'v'. In this paper, they explain that they take the Cholesky decomposition of coregionalisation matrix. I understand the B*B^T part of the index kernel. But what is the meaning of v in the index kernel. Why is it important?

Sorry if I am asking stupid questions but I am not able to related the ICM in the paper and index kernel.

[Sam4896]

Hi @saitcakmak

I am have some doubts and would really request your opinion about the same.

I tried implementing the KroneckerMTGP for my case where I have 55 training instances. The input is 4 dimensional and the output is 9 dimensional. But I didn't normalize the outputs/tasks. I used fit_gpytorch_mll and when I observed the model.covar_module.task_covar_module.covar_matrix, I got a heatmap for the covariance between the tasks. I also saw the correlation between the ouputs directly using the data. The two look like this:

1. I see that the learned covariance is somewhat similar to the data correlation heatmap. But the depedency between center tasks is not learned that much. I would really be thankful to you if you could share your interpretation of the result.
2. Should I normalize or standardize the tasks before using KroneckerMTGP? Also, should I normalize the inputs as well or can I use them as it is?
3. Is it possible to get the learned correlation between the tasks rather than covariance? I read it my stats course that the covariance represents the direction of dependency and the correlation represents the direction and strength of depedency between the variables. For example, can I say that with high learned covariance between the tasks represents high task interdependency?

[Sam4896]

Hi @saitcakmak
This question is related to multi-task and multi-output GP.

Suppose I have data from a process of a manufacturing process. 'A' experiments for a material 'a' and 'B; experiments for material 'b' are done. But the implemented process is the same. Each experiment for each material has 4 input parameters and yields 9 outputs that are represent some geometric charateristic. The 9 outputs might be correlated because they belong to the same geometry. Now my aim is to predict the 9 outputs if I am given the input parameters and if the material has property 'a' or 'b'.

1. If I understand correctly, in the documentation it is written that almost all models are multi-output and if there is a correlation between the outputs then it's better to use the MultiTaskGP (can either be the basic model of KroneckerMTGP). But in my case I am not able to understand if my case if multi-output or multi-task or MTMO? Because there's correlation beween the outputs of one experiment for each of the materials. Also, my aim is to learn from one material to another also. Which to me seems like multi-task inside the multi-task. And if I consider it as MTMO, then which should be MT and which part should be MO? Because according to documentation the correlation is not assumed in the MO case. So not interdepenceny will be learned for that case.
2. If I focus on only one material and use SingleTaskGP to model the outputs for the experiments 'A', then is any correlation between the outputs learned or it is just vector output without any learned dependencies? If there is any learned correlation then how is it different from KroneckerMultiTaskGP.