1
$\begingroup$

I've read and heard about the mighty XGBoost which is one of the most famous models people are using today to solve Kaggle challenge. This makes me interest to develop my own intuition about the model so I decided to try XGBoost on my dataset, but unfortunately, I experienced a lot of dependency related problems when I tried to install XGBoost on mac :(

However, I did a bit more researches and found out another related library called LightGBM from Microsoft which they claim to achieve better result (or at least, equivalent) over XGBoost. Therefore, I decided to go with the LightGBM instead since I didn't encounter any problems during the library installation.

I expected that the model should outperform, or at least, perform similarly compared to other models on the same dataset. My dataset contains 14k of text documents and has 0 or 1 values which refer to positive and negative sentiment respectively as target variable.

As a result showing below, LightGBM with count vectorizer as input got the 6th place with an accuracy of 94.60%. I'm quite curious since the accuracy of LightGBM is not only lower than SVC but also lower than Extra Tree model (300 esitimators).

I don't have enough foundation and solid understanding behind these models so I couldn't come up with an answer about these outcomes. Can someone please give me an idea or assumption according to the result?

PS. All models was performed with 5-fold cross validation 

model             score
--------------  -------
extraT           0.9528
svc              0.9514
sgd_elas         0.9481
extraT_tfidf     0.9476
svc_tfidf        0.9473
lightGBM         0.9460
sgd_elas_tfidf   0.9458
lightGBM_tfidf   0.9420
randomF          0.9409
randomF_tfidf    0.9345
mulNB            0.9307
berNB            0.9087
berNB_tfidf      0.9087
mulNB_tfidf      0.9036
Improve this question
$\endgroup$
4
  • $\begingroup$ have you set different parameter of lightGBM $\endgroup$
    – tktktk0711
    Commented Jul 14, 2017 at 9:08
  • $\begingroup$ @tktktk0711Yes, I did grid search on LightGBM model. The result was 0.1 learning rate, 150 num_tree and 100 num_leaves. The acc of LightGBM above is the result of the model with these parameters $\endgroup$
    – Satjapong Meeklai
    Commented Jul 14, 2017 at 9:11
  • $\begingroup$ I found that the parameter( just my parameter: max_depth=3, learning_rate=0.1, n_estimators=1000) have significantly influence on error when I used it for regression. Sorry I haven't used the mode for classifier $\endgroup$
    – tktktk0711
    Commented Jul 14, 2017 at 9:15
  • $\begingroup$ Try lowering the learning rate to 0.01 and increase the number of estimators to 3000. Be sure to include early stopping to not to overfit. Also, plot and study learning curves to understand whether to increase the number of estimators even more. $\endgroup$
    – tuomastik
    Commented Jul 14, 2017 at 10:23

1 Answer 1

Reset to default
2
$\begingroup$

Xgboost usually gives a higher accuracy(this is just an observation, not a fact). But the trick with xgboost is people dont know how to best use it. The most common mistake made using xgboost is not letting it train enough. Xgboost is an algorithm with overfits more than the other algorithms(well overfit is not the right term, it should be xgboost has high variance than the others). Do not tune the other parameters until you have found a converging nrounds for appropriate learning rate(0.01 or 0.1 or 0.3 etc). If you see that the cross val accuracy is much much lower than train accuracy then tune gamma(the second order derivative) first, then other parameters. If the cross val and train accuracy have a significant but not a lot difference(lets say train acc goes from 90 to 99 and cross val is stuck at 85 after significant iterations) then try reducing the max depth of trees. If the cross val and train accuracy are running side by side then just let the algorithm converge for a particular eta and then tune the other parameters(all except gamma).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi, thank you so much for your informative answer. I also got another followed up question. In lightGBM, there're original training API and also Scikit API to use with Scikit (I believe xgboost also got the same things). The lightGBM result above is from the Scikit version one. However, the result which trained on the original training API with the same parameters is significantly different to Scikit API result. The result was only about 82.74% compared to 94.60% above. It would be highly appreciated if you could explain me why in the above case. $\endgroup$
    – Satjapong Meeklai
    Commented Jul 17, 2017 at 6:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.