Leaving out out-of-design arms, when adapting searchspace constraints between iterations

[SebBT01]

Hey there,

I am experimenting with lowering limits in searchspace constraints between iterations using some heuristic, and I get the message e.g.:
ax.modelbridge.base: Leaving out out-of-design observations for arms: 0_0, 6_3, 6_2, 2_0, 3_0, 6_0, 4_0, 6_4, 5_0, 6_5, 1_0

I have located in the modelbridge code base, the option to fit on data out of design or not.

My question is, what would be the pros and cons of doing it or not?

I can imagine if we are talking about an optimization with e.g. trust regions and many arms, then maybe it will reduce the models ability to model data locally if its fit on out of design data?

I am working with a quite small setup of e.g. <60 total arms. I imagine SAASBO might gain from fitting on out of design data to fit lengthscales?

Cheers

[saitcakmak]

Hi @SebBT01. The default behavior of filtering out out-of-design observations is intended as a safety feature if those designs are significant outliers that may adversely affect model quality (this includes manually attached baseline arms that are not valid parameterizations). In your case, it sounds like the out-of-design observations are just outside of the search space, and including them would likely improve the model quality.

Adding fit_out_of_design=True to model_kwargs should solve the issue. Unfortunately, there isn't a convenient way of doing this, unless you're manually defining the GenerationStrategy. If you're using AxClient, you could update the model_kwargs on it using something like

saasbo_step = ax_client.generation_strategy._steps[1]  # Assuming GS is Sobol + SAASBO. 
saasbo_step.model_kwargs["fit_out_of_design"] = True

[SebBT01]

Thanks Sait, appreciate the insight. I too think its useful in my case, fortunately easy to test.

I am conventiently using a BOTORCH_MODULAR bridge where its easy to add it 👍

[saitcakmak]

Happy to help! I just became aware of this recent Ax-side bug fix related to fit_out_of_design: facebook/Ax#2277 You might want to use latest Ax to be on the safe side :)

[SebBT01]

Thanks will do!