Input Uncertainty Implemented in Kernel Function

[CalumHarvey]

I am trying to implement the kernel function proposed in "Dallaire et al Learning Gaussian Process Models from Uncertain Data" which requires a kernel function that looks like this:

$$C_n \left( \left( u_i, \Sigma_i \right), \left( u_j, \Sigma_j \right) \right) = \frac{\sigma_{f}^{2} exp \left( \left( u_i - u_j \right)^T \left( W + \Sigma_i + \Sigma_j \right)^{-1} \left( u_i - u_j \right) \right)}{ | I + W^{-1} \left( \Sigma_i + \Sigma_j \right) |^\frac{1}{2}}$$

What would be the best way to implement this? The kernel function will require additional input parameters other than x1 and x2 as specified in the code which would cause issues when calling the forward method.

[brunzema]

Hi,

maybe I can help a little here. So first, I would suggest to use a ScaleKernel to take care of the $\sigma_f^2$ and then have covar = ScaleKernel(MyKernel(...)). Then, for implementing the main body of your kernel, you can define your $x$ as $x = [\mu_1, \mu_2,\dots \mu_d, \Sigma_{11}, \Sigma_{12},\dots, \Sigma_{dd}]$ and then reconstruct $\mu$ and $\Sigma$ in your forward method. Lastly, if $W$ is a fixed matrix, you can just pass this matrix during initialization. If $W$ is a diagonal matrix, you can probably just set has_lengthscale=True and then embed the vector in self.lengthscale as a diagonal matrix (see here in the docs).

If $W$ a full matrix where all entries should be learned, I am not fully sure how to do this, but maybe some else can help out (maybe .requires_grad = True is sufficient or registering it through nn.Parameter(...)?). My only suggestion here would be to parameterize $W$ as a Cholesky decomposition; specifically as $W = LL^T$, where $L=L_d +\text{diag}(\exp{l})$. Here, $l$ is a vector and through the exponential function you ensure that the diagonal of $L$ (hence $W$) remains positive during learning, and $L_d$ is a lower triangular matrix with zeros on the diagonal. With this you should then never (hopefully) run into issues with taking the inverse of $W$.

Hope this was helpful! :)

[CalumHarvey]

Hi,

Thank you for your help :)

In the time between writing the question and now I took the approach mentioned in your first paragraph so it is good to see that we had the same thinking. $$W$$ is indeed the lengthscale so has_lengthscale=True is sufficient to solve this.

I have put the code in a repository for future reference.