I have predicted the probability of loss using different features. Now when I used this with a non-important feature to predict the probability of loss. The first one is very close. logloss was close to 0.11. However, I have few more other features, I wanted to know if the features are important or not. So, I used the new features with this predicted probability. I found volatile behavior. Not only did the performance (logloss) drop to 0.14, but the model didn't pick the predicted probability as an important feature.
My Primary Questions:
- What is the reason behind this outcome?
- Should I dump every single feature into one model, then see which feature is important ?