In the tutorial boosting from existing prediction in lightGBM R, there is a init_score parameter in function setinfo. I am wondering what init_score means? In the help page, it says:
init_score: initial score is the base prediction lightgbm will boost from
Another thing is what does "boost" mean in lightGBM?
Omitting the technical details, boosting is a statistical technique where we train many additional weak models to attempt to correct the errors of the previous models we have learned so far. In each iteration of gradient boosting, a new model is trained (usually a decision tree) which tries to fit to the residual of the prediction made by the existing set of models.
The init_score value represents the prediction that you are trying to correct with the first boosting iteration. By default this probably predicts the majority class for all examples or a randomly selected class, but you can input the prediction that was outputted by any other model here if you like.
The first tree that is learned by LightGBM will try to correct the errors of this initial prediction. In the example that you linked, they train a LightGBM model for one iteration, and use the score to continue training another LightGBM model. So in practice, what they have done in this example is kind of pointless because they could have just trained a single LightGBM from the start, but I suppose it is just to illustrate the concept and how it works.
The init score in LGB(LightGBM) is similar to the base score in XGB(XGBoost).
The function of this value is used to initialize the gradient boosting algorithm. This is mainly done to speed up convergence.
If you have enough time to wait for your model to converge, you do not have to use this parameter. In my personal experience, set this one to your training label average values, you will save time and get a no bad result.
Here is a comparison among default settings, using the base score and using focal loss.
# Focal loss
[100] fit's focal_loss: 0.00191792 val's focal_loss: 0.00359477
[200] fit's focal_loss: 0.00082206 val's focal_loss: 0.00287949
[300] fit's focal_loss: 0.000403044 val's focal_loss: 0.00262565
[400] fit's focal_loss: 0.000212066 val's focal_loss: 0.00258044
[500] fit's focal_loss: 0.000117509 val's focal_loss: 0.00265192
Test's ROC AUC: 0.98168
Test's logloss: 0.00242
# Base score
[100] fit's logloss: 0.00191083 val's logloss: 0.00358371
[200] fit's logloss: 0.000825181 val's logloss: 0.00286873
[300] fit's logloss: 0.000403679 val's logloss: 0.00262094
[400] fit's logloss: 0.000212998 val's logloss: 0.00257289
[500] fit's logloss: 0.0001183 val's logloss: 0.00262522
Test's ROC AUC: 0.98170
Test's logloss: 0.00242
# default
[100] fit's focal_loss: 0.203631 val's focal_loss: 0.203803
[200] fit's focal_loss: 0.0710043 val's focal_loss: 0.0712822
[300] fit's focal_loss: 0.0263409 val's focal_loss: 0.0267795
[400] fit's focal_loss: 0.0103281 val's focal_loss: 0.011038
[500] fit's focal_loss: 0.00422448 val's focal_loss: 0.00539362
Test's ROC AUC: 0.95715
Test's logloss: 0.00688
Among performance on AUC and log loss, Focal loss or base score is better than the default one.
The minimal example CoLab Notebook（focal_loss_on_init_score.ipynb）is inspired on this post。
