1
$\begingroup$

I have predicted the probability of loss using different features. Now when I used this with a non-important feature to predict the probability of loss. The first one is very close. logloss was close to 0.11. However, I have few more other features, I wanted to know if the features are important or not. So, I used the new features with this predicted probability. I found volatile behavior. Not only did the performance (logloss) drop to 0.14, but the model didn't pick the predicted probability as an important feature.

My Primary Questions:

  • What is the reason behind this outcome?
  • Should I dump every single feature into one model, then see which feature is important ?
Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ You may want to add some more detail to your question: what features you are using, what you are predicting. Sometimes a minimal reproducible example is useful (see stackoverflow.com/questions/5963269/…) and will help others give you more instructive feedback and useful answers. $\endgroup$
    – Brandon Loudermilk
    Apr 30, 2016 at 0:31
  • $\begingroup$ @Kappatel Patel I don't think dumping those features will give you a better result because xgboost is very powerful and is immune to such things. Dropping features will hardly make any difference as per my opinion $\endgroup$
    – enterML
    Oct 27, 2016 at 18:42
  • $\begingroup$ Kaggle Quora? (Based on the logloss you mention). Give some examples, and consider that boosting requires some degree of variation, i.e use a different model or set of features. $\endgroup$
    – GrimSqueaker
    Jun 5, 2017 at 12:45
  • $\begingroup$ @Nain XGBoost and LightGBM are powerful tree algorithms, yet they are not robust, they are rather sensitive to overfitting, useless features etc. Their power emerges from their computational efficiency enabling the framework to ensemble models. Dump useless(if you are sure they are useless) features, be careful with XGBoost. $\endgroup$
    – Ugur MULUK
    Nov 28, 2018 at 11:18

1 Answer 1

Reset to default
1
$\begingroup$

You could try the next thing:, get some top k predictions made with xgboost, and use only those as features and feed them into an LogisticRegression model. I am currious if that will help in case you want to give a feedback on my proposed solution

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ There's no reason to think that important features in a tree based model are linearly related to the target. $\endgroup$
    – Matthew Drury
    Mar 2, 2018 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.