Lightgbm confidence interval

Question

I want to compute a confidence interval for each sample for a lightgbm model I've trained.

If the model was a random forest, it'd be quite easy, just take all the trees and compute the standard deviation of the predictions.

I wonder if a similar think can be done with lightgbm. In particular:

• Does it make sense to compute the standard deviation of a boosting model by taking the predictions of all trees, or something similar?
• Can it be done in scikit-learn?
• Are there alternatives to obtain confidence intervals for lightgbm predictions, appart from the alternative that would be training quantile models?

Answer 1

If you are looking for a statistical trick, I don't know, but

Recently Andrew NG team recently published about NGBoost.

NGBoost is a new boosting algorithm, which uses Natural Gradient Boosting, a modular boosting algorithm for probabilistic predictions.

In this Towards Data Science toy example you can see how to use the Python API:

Quoting the TDS author:

NGBoost’s one of the biggest differences from other boosting algorithms is can return probabilistic distribution of each prediction. This can be visualized by using pred_dist function.

In their web they say "NGBoost can be used with any base learner, any family of distributions with continuous parameters, and any scoring rule." So it seems pretty flexible to do whatever you consider.

Answer 2

If you are using lightgbm to do classification, in that specific context you could use Venn Aber's calibration technique.

Venn Aber calibration, fits 2 isotonic regression model on positive and negative classes, hence as a side product, gives us a probability interval for each calibrated probability.

This can be seen as a prediction interval, than a confidence interval.

Hope this helps.