LightGBMCV with Optuna

[philipmassie]

Hello
If this already exists or is totally stupid, I apologise.
I'm struggling to figure out an efficient way to use LightGBMCV with Optuna.
I have an approach that works well, but the cv.setup step is repeated for every Optuna trial, which seems unnecessary since most of that time seems to be spent building features. For smaller sets it doesn't matter much, but when training sets become larger it adds a lot of time to tuning.

I came across the below approach somewhere which works well. I cant find the original link, but it's not mine.
nesting the setup inside the objective requires that to process on each trial.

import pandas as pd
from mlforecast.lgb_cv import LightGBMCV
import optuna

df = pd.read_csv('https://datasets-nixtla.s3.amazonaws.com/air-passengers.csv', parse_dates=['ds'])

fit_params = {
    "df": df,
    "id_col": "unique_id",
    "time_col": 'ds', 
    "target_col": 'y'
    }

fixed_params = {
    'boosting_type': 'gbdt',
    'objective': 'rmse',
    'n_estimators': 10_000,
    'verbosity': -1
    }

def objective(trial):
    params = {
        "learning_rate":    trial.suggest_float('learning_rate', 1e-3, 1e-1, step=1e-3)
    }

    cv = LightGBMCV(freq="MS", lags=[12])
    cv.setup(
        params={**fixed_params, **params},
        n_windows=2,
        h=12,
        **fit_params,
        metric="rmse"
    )

    hist = []
    eval_every = 10
    early_stopping_evals = 20
    early_stopping_pct = 0.01

    for i in range(0, fixed_params['n_estimators'], eval_every):
        val = cv.partial_fit(eval_every) 
        trial.report(val, step=i)
        rounds = eval_every + i
        hist.append((rounds, val))
        if trial.should_prune():
            raise optuna.TrialPruned()
        if cv.should_stop(hist, early_stopping_evals, early_stopping_pct):
            print(f"Early stopping at round {rounds:,}")
            best_iter = cv.find_best_iter(hist, early_stopping_evals)
            trial.set_user_attr('n_estimators', best_iter)            
            break
    return val

dbname = f"sqlite:///optuna.sqlite3"
study_name = "testing_cv2"

study = optuna.create_study(
    storage=dbname,
    study_name=study_name,
    direction='minimize')

study.optimize(objective, n_trials=5, n_jobs=1)

I've tried:

• updating the cv.items[i][1].params in each trial but the model keeps training from where it left off
• I've not found a way to re-initialise the existing cv object before updating parameters
• I thought perhaps I could simply re-instantiate a preexisting 'base' cv object in each trial but
  - copy.deepcopy() doesn't appear copy the integrated lightgbm booster in it's entirety.
  - cloudpickle similarly doesn't save the cv object in its entirety, and I'm not smart enough to figure out how to change that.
• I also tried redefining just the booster and adding that to the cv.items tuple via a tuple>list>tuple conversion but no joy there

So I'm not smart enough to figure this out, and wondering if some else already has.
Thanks

[philipmassie]

As it happens I just now came across this: #318

which got me thinking a little differently...

and I came up with this. Its a little rough and I don't know what needs a deep copy... but it appears to work.
Any thoughts are most welcome, I'm on very shaky ground here.

from functools import partial
import copy
import lightgbm as lgb
import pandas as pd
from mlforecast.lgb_cv import LightGBMCV
import optuna

df = pd.read_csv('https://datasets-nixtla.s3.amazonaws.com/air-passengers.csv', parse_dates=['ds'])

fit_params = {
    "df": df,
    "id_col": "unique_id",
    "time_col": 'ds', 
    "target_col": 'y'
    }

fixed_params = {
    'boosting_type': 'gbdt',
    'objective': 'rmse',
    'n_estimators': 10_000,
    'verbosity': -1
    }

cv = LightGBMCV(freq="MS", lags=[12])
cv.setup(
    params={**fixed_params},
    n_windows=2,
    h=12,
    **fit_params,
    metric="rmse"
)
def objective(trial):
    params = {
        "learning_rate":    trial.suggest_float('learning_rate', 1e-3, 1e-1, step=1e-3)
    }

    new_items = []
    for fold_item in cv.items:
        ts = copy.deepcopy(fold_item[0])
        ds = fold_item[1].train_set
        valid = copy.deepcopy(fold_item[2])
        bst = lgb.Booster({**fixed_params, **params, "num_threads": cv.bst_threads}, ds)
        bst.predict = partial(bst.predict, num_threads=cv.bst_threads)
        new_items.append((ts, bst, valid))
    cv.items = new_items

    hist = []
    eval_every = 10
    early_stopping_evals = 20
    early_stopping_pct = 0.01

    for i in range(0, fixed_params['n_estimators'], eval_every):
        val = cv.partial_fit(eval_every) 
        trial.report(val, step=i)
        rounds = eval_every + i
        hist.append((rounds, val))
        if trial.should_prune():
            raise optuna.TrialPruned()
        if cv.should_stop(hist, early_stopping_evals, early_stopping_pct):
            print(f"Early stopping at round {rounds:,}")
            best_iter = cv.find_best_iter(hist, early_stopping_evals)
            trial.set_user_attr('n_estimators', best_iter)            
            break
    return val

dbname = f"sqlite:///optuna.sqlite3"
study_name = "testing_cv2"

study = optuna.create_study(
    storage=dbname,
    study_name=study_name,
    direction='minimize')

study.optimize(objective, n_trials=5, n_jobs=1)
print("Fin")

[jmoralez]

Hey. I believe that example is from this notebook and it does the setup inside the trial so that you can also tune features, but if you're only tuning the model parameters then it's definitely very wasteful. I shared the following example with one of our users over slack a while ago (no longer available because we're on the free tier), it's very similar to what you currently have but since the data won't change you only need to re-initialize the LightGBM boosters.

from functools import partial

import lightgbm as lgb
import optuna
from mlforecast.lag_transforms import RollingMean
from mlforecast.lgb_cv import LightGBMCV
from mlforecast.target_transforms import Differences
from mlforecast.utils import generate_series

series = generate_series(10, n_static_features=1, min_length=1000, max_length=2000)
cv = LightGBMCV(
    freq = 'D',
    target_transforms=[Differences([1,7])],
    lag_transforms={
        1: [RollingMean(28)]
    },
    lags=range(1, 8),
    date_features=['month', 'day'],
    num_threads=2,
)
# run the setup once
cv.setup(
    series,
    n_windows=2,
    h=2,  
    id_col='unique_id',
    time_col='ds',
    target_col='y',
    static_features=['static_0'],
    params={'verbosity': -1},
)

def objective(trial, cv):
    params = {
        'bagging_freq': trial.suggest_int('bagging_freq', 3, 8),
        'learning_rate': trial.suggest_float('learning_rate', 0.001, 0.061891899489054726),
        'num_leaves': trial.suggest_int('num_leaves', 10, 20),
        'lambda_l1': trial.suggest_float('lambda_l1', 0.8, 1.0),
        'lambda_l2': trial.suggest_float('lambda_l2', 0.4, 1.0),
        'bagging_fraction': trial.suggest_float('bagging_fraction', 0.90, 1.0),
        'feature_fraction': trial.suggest_float('feature_fraction', 0.5, 1.0),
        'n_estimators':  trial.suggest_int('n_estimators', 1000,2500),
        'verbose': 0,
        'force_col_wise' : True,
    }

    # override booster object
    for i, (ts, orig_bst, valid) in enumerate(cv.items):
        bst = lgb.Booster({**params, "num_threads": cv.bst_threads}, train_set=orig_bst.train_set)
        bst.predict = partial(bst.predict, num_threads=cv.bst_threads)
        cv.items[i] = [ts, bst, valid]

    # modify these for your use case
    verbose_eval = True
    early_stopping_evals = 2
    early_stopping_pct = 0.01
    eval_every = 10    
    
    hist = []
    num_iterations = params['n_estimators']
    for i in range(0, num_iterations, eval_every):
        metric_value = cv.partial_fit(eval_every)
        rounds = eval_every + i
        hist.append((rounds, metric_value))
        if verbose_eval:
            print(f"[{rounds:,d}] {cv.metric_name}: {metric_value:,f}")
        if cv.should_stop(hist, early_stopping_evals, early_stopping_pct):
            print(f"Early stopping at round {rounds:,}")
            break
    best_iteration = cv.find_best_iter(hist, early_stopping_evals)
    hist = hist[: best_iteration // eval_every]
    return hist[-1][1]

study = optuna.create_study(direction='minimize')
study.optimize(partial(objective, cv=cv), n_trials=10)

[philipmassie]

Amazing, thank you so much. yep, similar to where I ended up, but much more refined!
Also, thank you for that link, I couldn't find it for the life of me. It has been do useful to me, that approach. Yeah, for this I'm only tuning model params.
Love MLForecast, it's really so great.