How to handle different independent variables in CompositeModel?

[Julian-Hochhaus]

Hi,
During fitting XPS data with a quite complex model (a PeakModel (DoniachModel convoluted with Gaussian) and an additional BackgroundModel like Tougaard or Shirley), I stumbled upon a warning, which I do not understand.

My PeakModel uses only 'x' as an independent variable, while my BackroundModel uses 'x' and 'y' as independent variables. I, therefore, expected the CompositeModel (automatically created by CompModel=PeakModel+BackroundModel) to have two independent variables, namely 'x' and 'y' as well. However, my CompModel only has 'x' as an independent variable.
Surprisingly, it seems like the CompositeModel still works fine while fitting (it looks like the independent variable 'y' is correctly fed to my BackroundModel).
I call the fitting with:

result = CompModel.fit(y_vals, params=pars, x=x_vals, y=y_vals)

During the fitting procedure, only the following UserWarning is thrown:

model.py:990: UserWarning: The keyword argument y does not match any arguments of the model function. It will be ignored. missing_param = any(p not in params.keys() for p in self.param_names)

This would make perfect sense because my CompositeModel only knows 'x' as the independent variable.

However, if I define my Model the other way around, with CompModel=BackroundModel+PeakModel, my CompModel has the two independent variables 'x' and 'y', and additionally, the fitting now does not work and terminates with a TypeError:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-197-216a80424a28> in <module>
     56 print(pars)
     57 
---> 58 result = CompModel.fit(y_vals, params=pars, x=x_vals, y=y_vals)
     59 
     60 

~/anaconda3/lib/python3.8/site-packages/lmfit/model.py in fit(self, data, params, weights, method, iter_cb, scale_covar, verbose, fit_kws, nan_policy, calc_covar, max_nfev, **kwargs)
   1048 
   1049     def __add__(self, other):
-> 1050         """+"""
   1051         return CompositeModel(self, other, operator.add)
   1052 

~/anaconda3/lib/python3.8/site-packages/lmfit/model.py in fit(self, data, params, weights, method, nan_policy, **kwargs)
   1402         for attr in dir(_ret):
   1403             if not attr.startswith('_'):
-> 1404                 try:
   1405                     setattr(self, attr, getattr(_ret, attr))
   1406                 except AttributeError:

~/anaconda3/lib/python3.8/site-packages/lmfit/model.py in eval(self, params, **kwargs)
   1153     def eval_components(self, **kwargs):
   1154         """Return dictionary of name, results for each component."""
-> 1155         out = dict(self.left.eval_components(**kwargs))
   1156         out.update(self.right.eval_components(**kwargs))
   1157         return out

~/anaconda3/lib/python3.8/site-packages/lmfit/model.py in eval(self, params, **kwargs)
   1153     def eval_components(self, **kwargs):
   1154         """Return dictionary of name, results for each component."""
-> 1155         out = dict(self.left.eval_components(**kwargs))
   1156         out.update(self.right.eval_components(**kwargs))
   1157         return out

TypeError: can only concatenate list (not "float") to list

The reason for getting two different outcomes by simply changing the order of Models in CompositeModel seems to be, that the CompositeModel class assumes that both Models have the same independent variables and so the independent variables of the left Model (first Model) are taken (model.py, l. 1119).

# we assume that all the sub-models have the same independent vars if 'independent_vars' not in kws: kws['independent_vars'] = self.left.independent_vars

I do not get why it is necessary, that both Models have ALL independent variables in common.
And I do not think that the observed behaviour is a good solution, especially since the behaviour depends on the order of Models in the CompositeModel.
Maybe someone could enlighten me on how to find a workaround for my use case. Explanations for why the actual behaviour is intended are welcome too!

Kind regards,

Julian

[newville]

@Julian-Hochhaus I think we just didn't carefully think through the case where some sub-components would have independent parameters that others did not. Well, maybe except for "ConstantModel", which has also caused some confusion.

I think that it would probably make sense to have the composite use the union of the independent parameters of "left" and "right" when being initialized and then only pass the ones expected for each sub-component. That is, the composite might need a pair (tuple, set, dict, probably doesn't matter) of independent_vars for left and right.

PR anyone?

[Julian-Hochhaus]

I think it should be fixed by changing l. 1120 in model.py from:

if 'independent_vars' not in kws:

    kws['independent_vars'] = self.left.independent_vars

to something like:

if 'independent_vars' not in kws:  
    
    kws['independent_vars'] = list(set([*self.left.independent_vars, *self.right.independent_vars]))

or to be able to process iterable items in general and not be limited to lists:

if 'independent_vars' not in kws:

    kws['independent_vars'] = list(dict.fromkeys([*self.left.independent_vars, *self.right.independent_vars]))

I will send a push request.