Proper workflow for AutoMLForecast

[egolub22]

I am using AutoMLForecast to tune several models on several dataframes. I'm not sure about exactly what the fit function is doing and what the most computationally efficient way to train models and get proper accuracy metrics.

It would seem from the arguments to the fit function that cross-validation is used to train the final models, but the dataframe returned by the forecast_fitted_values function does not have a cutoff column and the description says the predictions are in-sample, as opposed to an out-sample cross-validation.

My solution to generate good metrics is to take each final model and run a full cross-validation function on it, then concatenate all of the outputs into one dataframe, and evaluate that dataframe. However this is very computationally expensive and takes a long time.

Is there a way to output the out of sample cross-validation used by the fit function, or even a way to just get metrics from evaluating such a dataframe without having to run a full additional cross-validation after training?

[jmoralez]

AutoMLForecast performs cross validation for each trial and reports the average across folds as the trial score. These results are available in the results_ attribute (docs), so if you just want to know the score of the best trial you can get it with: auto_mlf.results_.best_trial.value

[egolub22]

Is it possible to access the full cross-validation DataFrame from the best trial? Because I am comparing models over multiple metrics, I am also showing the graphs of individual cross-val windows to visualize model performance.

[jmoralez]

No, only the score is saved, if you want the full result you have to run cross validation again on the best configuration. Keep in mind that performing cross validation modifies the model's state, so you should do it on a separate instance using the best configuration, otherwise you'll have to fit again afterwards.