3
$\begingroup$

I have a dataset of 3500 observations x 70 features which is my training set and I also have a dataset of 600 observations x 70 features which is the test set.

The target is to classify observations correctly either as 0 or 1. 2000 observations of the training set are 0 and the rest 1600 of them are 1.

I aim at the highest possible recall for precision>=90%.

I did grid search for ensemble algorithms only in relation to number of trees (from 50 to 650 trees). Analytically the best recall results for precision >= 90% for each of the algorithms are the following:

Random Forest (375 trees)

from sklearn.ensemble import RandomForestClassifier
classifier = RandomForestClassifier(random_state=0, n_estimators=375, class_weight='balanced')
classifier.fit(X_train, y_train)
  • Precision: 90%
  • Recall: 24%

Xgboost (550 trees)

from xgboost import XGBClassifier
classifier = XGBClassifier(n_estimators=n_trees, seed=0, scale_pos_weight=1.5)
classifier.fit(X_train, y_train, eval_metric='map')
  • Precision: 90%
  • Recall: 15%

Why Xgboost is performing so much worse than the Random Forest?

||improve this question
$\endgroup$
  • 1
    $\begingroup$ Different algorithms, different parameters (depth, number of nodes, minimum samples in node, number of samples to consider split, etc.), different ways of handling imbalance. $\endgroup$ – user2974951 Sep 20 '18 at 13:36
  • 2
    $\begingroup$ Slightly tangential, but did I understand correctly that you grid-searched over the number of trees to use? This is extremely inefficient in XGBoost. I strongly recommend creating an evaluating set and then using early stopping. Also, surely you found, when using a random forest, that your results just get better and better (tailing off asymptotically) as you increase the number of trees? $\endgroup$ – gazza89 Sep 21 '18 at 10:00
  • $\begingroup$ I am not sure that number of trees "is extremely inefficient in XGBoost". But in any case, only the fact that xgboost needs so much tuning in order to reach closer, if it finally does, to the performance of a way less tuned Random Forest is a sign that it is not probably the right algorithm for this case. $\endgroup$ – Outcast Sep 21 '18 at 10:49
  • 1
    $\begingroup$ Yes, I meant grid search over the other parameters and use early stopping to determine the number of trees. What it means, is that you won't have iterations where you use high numbers of trees, which take a long time to train, and give you terrible results. $\endgroup$ – gazza89 Oct 18 '18 at 10:41
  • 1
    $\begingroup$ @gazza89, I have actually performed some very deep grid searches (without early stopping) with both Random Forest and Xgboost and for now I get 37% & 28% recall respectively for precision 90% (at around 400 trees for both). Therefore, still things are more or less the same in terms of the comparative performance of these algorithms. (Please keep in mind that my aim is to maximise recall for precision>=90%). But I am going to get some new data sometime soon and see which one is really performing better in a entirely new test set because finally we may that Xgboost is significantly better. $\endgroup$ – Outcast Oct 18 '18 at 10:53
3
$\begingroup$

Different algorithms need to be tuned in different ways. For example, one important parameter in boosting is ETA or learning rate. This determines the change in weights after each boosting step. This parameter is important to reduce overfitting in boosting.

To understand the different parameters you can refer to this.

Another important thing to look at is the objective function. This will determine what your algorithm is trying to optimize.

Lastly, please make sure you are selecting the model that actually maximises the metric that you care about. You can do this by selecting an appropriate eval_metric. In this case, you probably want to use something like F1-Score or precision at a fixed recall.

||improve this answer
$\endgroup$
  • $\begingroup$ Thanks for your answer @amanbirs (upvote). I have added a small part of my source code to my post to show that I have picked some reasonable parameters for the training so probably the problem is not that I have not specified the right metric etc. In any case, finally, it may not be so unreasonable that (sometimes) the random forest performs so much better than Xgboost but I just wanted the opinion of other data scientists too. $\endgroup$ – Outcast Sep 21 '18 at 9:34
  • 1
    $\begingroup$ from the code you attached, it looks like you didn't tune the learning rate or the regularisation parameters. You also didn't add any elements of bagging (e.g. column subsampling or row subsampling). $\endgroup$ – gazza89 Sep 21 '18 at 9:56
  • 1
    $\begingroup$ Furthermore, I'd have to think through the details of this a bit more, but maybe your lower recall (albeit not your equal precision) could be affected by the class weight you used in the random forest. What precision and recall do you get if you don't set the class weight parameter (i.e. all examples get weight 1) ? $\endgroup$ – gazza89 Sep 21 '18 at 10:02
  • 1
    $\begingroup$ @Poete Maudit, you have left the parameters as the default value. The default is not always the appropriate value. I would recommend understanding how these algos and parameters work a bit better and then trying to tune them more effectively. $\endgroup$ – amanbirs Sep 21 '18 at 10:14
  • 2
    $\begingroup$ Why do you think that you can't reach the Random Forest performance, no matter how much you tune? The learning rate in particular can pretty drastically affect model performance. XGBoost is a more complicated model than a random forest and thus can almost always outperform a random forest on training loss, but likewise is more subject to overfitting. Agree with amanbirs above, try reading some blogs about hyperparameter tuning in xgboost and get a feel for how they interact with one and other. And as a said above, I highly recommend early stopping $\endgroup$ – gazza89 Sep 21 '18 at 10:42
1
$\begingroup$

You can refer to the following link XGBOOST is slower than Random Forest on the Xgboost Github. Its a weakness of GBT's in general when there are many classes.

The reason is that gradient boosting requires that you train [number of iterations] * [number of classes] trees, whereas random forest only requires [number of iterations] trees.

||improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.