In the tutorial boosting from existing prediction in lightGBM R, there is a init_score parameter in function setinfo. I am wondering what init_score means? In the help page, it says:

init_score: initial score is the base prediction lightgbm will boost from

Another thing is what does "boost" mean in lightGBM?


Omitting the technical details, boosting is a statistical technique where we train many additional weak models to attempt to correct the errors of the previous models we have learned so far. In each iteration of gradient boosting, a new model is trained (usually a decision tree) which tries to fit to the residual of the prediction made by the existing set of models.

The init_score value represents the prediction that you are trying to correct with the first boosting iteration. By default this probably predicts the majority class for all examples or a randomly selected class, but you can input the prediction that was outputted by any other model here if you like.

The first tree that is learned by LightGBM will try to correct the errors of this initial prediction. In the example that you linked, they train a LightGBM model for one iteration, and use the score to continue training another LightGBM model. So in practice, what they have done in this example is kind of pointless because they could have just trained a single LightGBM from the start, but I suppose it is just to illustrate the concept and how it works.

  • 1
    $\begingroup$ So would running one LightGBM model from the start with 6 rounds inherently have the behavior of running one round, then using the init_score to run 5 more? And in that case, init score is only really useful if you want to run a svm for instance and then pass the init score to a lightGBM model to finish the job? Or can the first model only be tree-based? $\endgroup$ Jan 27 '19 at 0:38

The init score in LGB(LightGBM) is similar to the base score in XGB(XGBoost).

The function of this value is used to initialize the gradient boosting algorithm. This is mainly done to speed up convergence.

If you have enough time to wait for your model to converge, you do not have to use this parameter. In my personal experience, set this one to your training label average values, you will save time and get a no bad result.

Here is a comparison among default settings, using the base score and using focal loss.

# Focal loss
[100]   fit's focal_loss: 0.00191792    val's focal_loss: 0.00359477
[200]   fit's focal_loss: 0.00082206    val's focal_loss: 0.00287949
[300]   fit's focal_loss: 0.000403044   val's focal_loss: 0.00262565
[400]   fit's focal_loss: 0.000212066   val's focal_loss: 0.00258044
[500]   fit's focal_loss: 0.000117509   val's focal_loss: 0.00265192

Test's ROC AUC: 0.98168
Test's logloss: 0.00242

# Base score 
[100]   fit's logloss: 0.00191083   val's logloss: 0.00358371
[200]   fit's logloss: 0.000825181  val's logloss: 0.00286873
[300]   fit's logloss: 0.000403679  val's logloss: 0.00262094
[400]   fit's logloss: 0.000212998  val's logloss: 0.00257289
[500]   fit's logloss: 0.0001183    val's logloss: 0.00262522

Test's ROC AUC: 0.98170
Test's logloss: 0.00242

# default
[100]   fit's focal_loss: 0.203631  val's focal_loss: 0.203803
[200]   fit's focal_loss: 0.0710043 val's focal_loss: 0.0712822
[300]   fit's focal_loss: 0.0263409 val's focal_loss: 0.0267795
[400]   fit's focal_loss: 0.0103281 val's focal_loss: 0.011038
[500]   fit's focal_loss: 0.00422448    val's focal_loss: 0.00539362

Test's ROC AUC: 0.95715
Test's logloss: 0.00688

Among performance on AUC and log loss, Focal loss or base score is better than the default one.

The minimal example CoLab Notebook(focal_loss_on_init_score.ipynb)is inspired on this post


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.