How to choose the model parameters (RandomizedSearchCV, .GridSearchCV) or manually

Question

Faced with the task of selecting parameters for the lightgbm model, the question accordingly arises, what is the best way to select them? I used the RandomizedSearchCV method, within 10 hours the parameters were selected, but there was no sense in it, the accuracy was the same as when manually entering the parameters at random. +/- the meaning of the parameters is clear, which ones are responsible for retraining, which ones are for the accuracy and speed of training, but it’s not entirely clear if you select manually one at a time or in pairs, or even more options?

Below is an example of how I implemented the selection of parameters:

SEED = 4 
NFOLDS = 2
kf = KFold(n_splits= NFOLDS, shuffle=False)

    parameters = {
          'num_leaves': np.arange(100,500,100),
          'min_child_weight': np.arange(0.01,1,0.01),
          'feature_fraction': np.arange(0.1,0.4,0.01),
          'bagging_fraction':np.arange(0.3,0.5,0.01),
          'min_data_in_leaf': np.arange(100,1500,10),
          'objective': ['binary'],
          'max_depth': [-1],
          'learning_rate':np.arange(0.001,0.02,0.001),
          "boosting_type": ['gbdt'],
          "bagging_seed": np.arange(10,42,5),
          "metric": ['auc'],
          "verbosity": [1],
          'reg_alpha': np.arange(0.3,1,0.2),
          'reg_lambda':  np.arange(0.37,0.39,0.001),
          'random_state': [425],
          'n_estimators': [500]}

model = lightgbm.LGBMClassifier()
RSCV = RandomizedSearchCV(model,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)

Answer 1

Thanks for the clarification. You can configure the parameters once or twice at a time by re-instantiating the RSCV object each time, passing different parameter dictionaries each time. For example:

SEED = 4 
NFOLDS = 2
kf = KFold(n_splits= NFOLDS, shuffle=False)

    parameters = {
          'num_leaves': np.arange(100,500,100),
          'min_child_weight': np.arange(0.01,1,0.01),
    }

model = lightgbm.LGBMClassifier()

RSCV = RandomizedSearchCV(model,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)

    parameters = {
          'feature_fraction': np.arange(0.1,0.4,0.01),
          'bagging_fraction':np.arange(0.3,0.5,0.01),
          'min_data_in_leaf': np.arange(100,1500,10),
    }

RSCV = RandomizedSearchCV(RSCV.best_estimator_,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)

By passing the RSCV.best_estimator_ instead of model to the second time, it will automatically use the best values for num_leaves and min_child_weight that it identified in the first run and effectively "freeze" those as they are, finding the best combination of feature_fraction, bagging_fraction and min_data_in_leaf under those constraints.

My understanding, however, is that the best approach is to do as you are currently doing and simply include all the parameters in one search. I generally set n_estimators to some lowish value (200 or so) and learning_rate to some reasonably high value whilst doing the tuning, and then beef them up later. This will generally drastically reduce the time taken to perform the tuning. It does necessitate a refit afterwards to take account of the new values.

model = lightgbm.LGMBClassifier(n_estimators=100, learning_rate=0.2

params = {
    'num_leaves':np.arange(100, 500 ,100)
    ...
}

RSCV.fit(train, label)

model = RSCV.best_estimator_
model.set_params(n_estimators=1000, learning_rate=0.001)
model.fit(train, label)

The literature on Randomised Search is usually quoted as being this paper, which I recommend reading: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf. One key takeaway is the conclusion that roughly 60 trials are needed to find the optimal parameter space with a reasonably high probability, so I'd recommend increasing the n_iter parameter of RSCV to 60.

Finally, the use of np.arange() results in each of your lists being uniformly distributed. For example, the tuning algorithm will select values for feature_fraction between 0.1 and 0.4 with equal likelihood. You can also use Scipy's distribution functions to define other distributions; normal and log-normal for example that targets the search on a particular area (for example, you could make the optimiser more likely to try values closer to 0 than values close to 1 for a the feature_fraction parameter.

Hope all that helps.