Random Forest VS LightGBM

Question

Random Forest VS LightGBM

Can somebody explain in-detailed differences between Random Forest and LightGBM? And how the algorithms work under the hood?

As per my understanding from the documentation:

LightGBM and RF differ in the way the trees are built: the order and the way the results are combined. It has been shown that GBM performs better than RF if parameters tuned carefully.

Random Forest: RFs train each tree independently, using a random sample of the data. This randomness helps to make the model more robust than a single decision tree, and less likely to overfit on the training data

My questions are

1. When would one use Random Forests over Gradient Boosted Machines?
2. What are the advantages/disadvantages of using Gradient Boosting over Random Forests?

Answer 1

RandomForest advantage compared to newer GBM models is that it is easy to tune and robust to parameter changes. It is robust for most use cases although the peak performance might not be as good as a properly-tuned GBM. Another advantage is that you do not need to care a lot about parameter. You can compare the number of parameter for randomforest model and lightgbm from its documentation. In sklearn documentation the number of parameter might seem a lot, but actually the only parameter you need to care about(ordered by importance) are max_depth, n_estimators, and class_weight, and the other parameters are better to be left as is. So for me, I would most likely use random forest to make baseline model.

GBM is often shown to perform better especially when you comparing with random forest. Especially when comparing it with LightGBM. A properly-tuned LightGBM will most likely win in terms of performance and speed compared with random forest.

GBM advantages :

• More developed. A lot of new features are developed for modern GBM model (xgboost, lightgbm, catboost) which affect its performance, speed, and scalability.

GBM disadvantages :

• Number of parameters to tune
• Tendency to overfit easily

Please bear in mind that increasing the number of estimators for random forest and gbm implies different behaviour. High value of n_estimators for random forest will affect it robustness, where as for GBM model will improve the model fit with your training data (which if too high will cause your model to overfit).