Per-dimension constraints on RBFKernel lengthscales

[Obs01ete]

I am fitting a GP into a dataset with 379 points in F=7 dimensions. I am using the combo of RBFKernel, FixedNoiseGP, ExactMarginalLogLikelihood and fit_gpytorch_mll. After fitting the kernel lengthscales are like this:

lengthscale: [0.17149281 0.44241985 0.55040405 0.09258469 0.00784354 0.50260822 0.40851717]

What is worrysome is that the 5th lengthscale is very low:

lengthscale[4] = 0.00784354

which leads to heavy overfit on the coreesponding feature.

As recommended here I put a LowerThan constraint on my RBFKernel lenghtscale like this:

rbf_kernel = RBFKernel(
    ard_num_dims=feature_pt.shape[-1],
    lengthscale_constraint=Interval(0.1, 0.3))

This regularizes the GP fine, but still feels like I may constrain other dimensions too much.

Q1: Is there a way to create a list of F separate constraints, so I can bound just one dimention's lengthscale?

Also I noticed that the learnt lengthscales are very unreliable, on each refit the values are significantly different.

Q2: How to achieve stable fitting of the GP in the sense that the lengthscales end up about the same on each refit?

[Obs01ete]

After the authors of GPytorch confirmed in this discussion that lengthscale bounds can be 1D tensors like the following snippet, I ran into an exception coming from BoTorch.

    rbf_kernel = RBFKernel(
        ard_num_dims=num_dims,
        lengthscale_constraint=Interval(
            torch.tensor(empirical_lengthscales, dtype=torch.float32),
            torch.tensor(empirical_lengthscales+0.01, dtype=torch.float32),
            ),
        )
    gp_kernel = ScaleKernel(rbf_kernel)
    gp = FixedNoiseGP(feature_pt, objective_pt, train_Yvar=objective_var_pt,
                      covar_module=gp_kernel)
    mll = ExactMarginalLogLikelihood(gp.likelihood, gp)
    ret = fit_gpytorch_mll(mll)

Exception:

Exception has occurred: ValueError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
only one element tensors can be converted to Python scalars
  File "C:\Users\dmitrii\anaconda3\envs\botorch\Lib\site-packages\botorch\optim\utils\numpy_utils.py", line 160, in get_bounds_as_ndarray
    lower = -inf if lower is None else float(lower)
  File "C:\Users\dmitrii\anaconda3\envs\botorch\Lib\site-packages\botorch\optim\core.py", line 93, in scipy_minimize
    bounds_np = get_bounds_as_ndarray(parameters, bounds)
  File "C:\Users\dmitrii\anaconda3\envs\botorch\Lib\site-packages\botorch\optim\fit.py", line 120, in fit_gpytorch_mll_scipy
    result = scipy_minimize(
  File "C:\Users\dmitrii\anaconda3\envs\botorch\Lib\site-packages\botorch\fit.py", line 252, in _fit_fallback
    optimizer(mll, closure=closure, **optimizer_kwargs)
  File "C:\Users\dmitrii\anaconda3\envs\botorch\Lib\site-packages\botorch\utils\dispatcher.py", line 93, in __call__
    return func(*args, **kwargs)
  File "C:\Users\dmitrii\anaconda3\envs\botorch\Lib\site-packages\botorch\fit.py", line 105, in fit_gpytorch_mll
    return FitGPyTorchMLL(
  File "C:\Users\dmitrii\_KAUST\git\modelling.py", line 179, in fit_model
    ret = fit_gpytorch_mll(mll)
  File "C:\Users\dmitrii\_KAUST\git\generate_proposals.py", line 21, in main
    data_and_model = fit_model(data_gdf, yield_objective_stddev, report)
  File "C:\Users\dmitrii\_KAUST\git\generate_proposals.py", line 42, in <module>
    main()
  File "C:\Users\dmitrii\anaconda3\envs\botorch\Lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\dmitrii\anaconda3\envs\botorch\Lib\runpy.py", line 194, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
ValueError: only one element tensors can be converted to Python scalars

Seem that BoTorch does not support tensor constraints for lengthscales. Any advice apart from fixing it and making a PR? :)

[saitcakmak]

Thanks for reporting. This is fixed in #1843

[Obs01ete]

@saitcakmak my highest appreciation for the blazing fast fix. I checked

pip install git+https://github.com/pytorch/botorch.git

and the bug is gone!