0
$\begingroup$

Faced with the task of selecting parameters for the lightgbm model, the question accordingly arises, what is the best way to select them? I used the RandomizedSearchCV method, within 10 hours the parameters were selected, but there was no sense in it, the accuracy was the same as when manually entering the parameters at random. +/- the meaning of the parameters is clear, which ones are responsible for retraining, which ones are for the accuracy and speed of training, but it’s not entirely clear if you select manually one at a time or in pairs, or even more options?

Below is an example of how I implemented the selection of parameters:

SEED = 4 
NFOLDS = 2
kf = KFold(n_splits= NFOLDS, shuffle=False)

    parameters = {
          'num_leaves': np.arange(100,500,100),
          'min_child_weight': np.arange(0.01,1,0.01),
          'feature_fraction': np.arange(0.1,0.4,0.01),
          'bagging_fraction':np.arange(0.3,0.5,0.01),
          'min_data_in_leaf': np.arange(100,1500,10),
          'objective': ['binary'],
          'max_depth': [-1],
          'learning_rate':np.arange(0.001,0.02,0.001),
          "boosting_type": ['gbdt'],
          "bagging_seed": np.arange(10,42,5),
          "metric": ['auc'],
          "verbosity": [1],
          'reg_alpha': np.arange(0.3,1,0.2),
          'reg_lambda':  np.arange(0.37,0.39,0.001),
          'random_state': [425],
          'n_estimators': [500]}

model = lightgbm.LGBMClassifier()
RSCV = RandomizedSearchCV(model,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)
$\endgroup$
3
  • $\begingroup$ I'm a little unclear what your question is. You want to know which parameters affect the accuracy and speed of the model? Or whether GridSearchCV is superior to RandomSearchCV? $\endgroup$
    – Dan Scally
    Sep 4, 2019 at 13:43
  • $\begingroup$ @DanScally Can I configure the parameters one or two at a time, or use multiple parameters as in the code example? $\endgroup$ Sep 4, 2019 at 13:59
  • $\begingroup$ @DanScally and what are the recommendations for the selection of parameters? $\endgroup$ Sep 4, 2019 at 14:00

1 Answer 1

2
$\begingroup$

Thanks for the clarification. You can configure the parameters once or twice at a time by re-instantiating the RSCV object each time, passing different parameter dictionaries each time. For example:

SEED = 4 
NFOLDS = 2
kf = KFold(n_splits= NFOLDS, shuffle=False)

    parameters = {
          'num_leaves': np.arange(100,500,100),
          'min_child_weight': np.arange(0.01,1,0.01),
    }

model = lightgbm.LGBMClassifier()

RSCV = RandomizedSearchCV(model,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)

    parameters = {
          'feature_fraction': np.arange(0.1,0.4,0.01),
          'bagging_fraction':np.arange(0.3,0.5,0.01),
          'min_data_in_leaf': np.arange(100,1500,10),
    }

RSCV = RandomizedSearchCV(RSCV.best_estimator_,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)

By passing the RSCV.best_estimator_ instead of model to the second time, it will automatically use the best values for num_leaves and min_child_weight that it identified in the first run and effectively "freeze" those as they are, finding the best combination of feature_fraction, bagging_fraction and min_data_in_leaf under those constraints.

My understanding, however, is that the best approach is to do as you are currently doing and simply include all the parameters in one search. I generally set n_estimators to some lowish value (200 or so) and learning_rate to some reasonably high value whilst doing the tuning, and then beef them up later. This will generally drastically reduce the time taken to perform the tuning. It does necessitate a refit afterwards to take account of the new values.

model = lightgbm.LGMBClassifier(n_estimators=100, learning_rate=0.2

params = {
    'num_leaves':np.arange(100, 500 ,100)
    ...
}

RSCV.fit(train, label)

model = RSCV.best_estimator_
model.set_params(n_estimators=1000, learning_rate=0.001)
model.fit(train, label)

The literature on Randomised Search is usually quoted as being this paper, which I recommend reading: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf. One key takeaway is the conclusion that roughly 60 trials are needed to find the optimal parameter space with a reasonably high probability, so I'd recommend increasing the n_iter parameter of RSCV to 60.

Finally, the use of np.arange() results in each of your lists being uniformly distributed. For example, the tuning algorithm will select values for feature_fraction between 0.1 and 0.4 with equal likelihood. You can also use Scipy's distribution functions to define other distributions; normal and log-normal for example that targets the search on a particular area (for example, you could make the optimiser more likely to try values closer to 0 than values close to 1 for a the feature_fraction parameter.

Hope all that helps.

$\endgroup$
1
  • $\begingroup$ Thanks for the detailed explanation, very informative. $\endgroup$ Sep 4, 2019 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.