I have predicted the probability of loss using different features. Now when I used this with a non-important feature to predict the probability of loss. The first one is very close. logloss was close to 0.11. However, I have few more other features, I wanted to know if the features are important or not. So, I used the new features with this predicted probability. I found volatile behavior. Not only did the performance (logloss) drop to 0.14, but the model didn't pick the predicted probability as an important feature.

My Primary Questions:

  • What is the reason behind this outcome?
  • Should I dump every single feature into one model, then see which feature is important ?
  • 3
    $\begingroup$ You may want to add some more detail to your question: what features you are using, what you are predicting. Sometimes a minimal reproducible example is useful (see stackoverflow.com/questions/5963269/…) and will help others give you more instructive feedback and useful answers. $\endgroup$ – Brandon Loudermilk Apr 30 '16 at 0:31
  • $\begingroup$ @Kappatel Patel I don't think dumping those features will give you a better result because xgboost is very powerful and is immune to such things. Dropping features will hardly make any difference as per my opinion $\endgroup$ – enterML Oct 27 '16 at 18:42
  • $\begingroup$ Kaggle Quora? (Based on the logloss you mention). Give some examples, and consider that boosting requires some degree of variation, i.e use a different model or set of features. $\endgroup$ – GrimSqueaker Jun 5 '17 at 12:45
  • $\begingroup$ @Nain XGBoost and LightGBM are powerful tree algorithms, yet they are not robust, they are rather sensitive to overfitting, useless features etc. Their power emerges from their computational efficiency enabling the framework to ensemble models. Dump useless(if you are sure they are useless) features, be careful with XGBoost. $\endgroup$ – Ugur MULUK Nov 28 '18 at 11:18

You could try the next thing:, get some top k predictions made with xgboost, and use only those as features and feed them into an LogisticRegression model. I am currious if that will help in case you want to give a feedback on my proposed solution

  • 2
    $\begingroup$ There's no reason to think that important features in a tree based model are linearly related to the target. $\endgroup$ – Matthew Drury Mar 2 '18 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.