2
$\begingroup$

I'm trying to do some feature selection using XGBoost, but the feature importance chart just spits out the features in order of appearance. The feature that is in the first column in the xtrain data is by far most important and then second is second, etc.

It seems like a sign that the model is not working properly as its not really learning anything... any advice on what could be going wrong?

UPDATE: Correlation Matrix https://ibb.co/3shDJjD

Model Code:

params = {
  'subsample':0.5,
  'learning_rate': 0.3,
  'max_depth':8,
  'num_parallel_trees' : 20,
  'objective': 'reg:squarederror',
  'verbosity':0,
  }
watchlist = [(train, 'train'), (test, 'val')]
reg = xgb.train(params, train, num_boost_round=5, early_stopping_rounds=5, evals=watchlist)

Results:

[0] train-rmse:0.274535 val-rmse:0.27431
Multiple eval metrics have been passed: 'val-rmse' will be used for early stopping.

Will train until val-rmse hasn't improved in 5 rounds.
[1] train-rmse:0.273472 val-rmse:0.273653
[2] train-rmse:0.272796 val-rmse:0.27341
[3] train-rmse:0.272318 val-rmse:0.27334
[4] train-rmse:0.271943 val-rmse:0.273346
[5] train-rmse:0.271604 val-rmse:0.273374
[6] train-rmse:0.271218 val-rmse:0.273442
[7] train-rmse:0.270927 val-rmse:0.273529
[8] train-rmse:0.270641 val-rmse:0.273561
Stopping. Best iteration:
[3] train-rmse:0.272318 val-rmse:0.27334

Feature importance (note that 0 and 1 are first). If I change the order of the columns in the xtrain, the feature importance will also change and first two columns will always be two most important features. https://ibb.co/QcHwbNg

$\endgroup$
3
  • $\begingroup$ A MWE (code and data, if possible) would be really helpful here. How's the model performing? Are the features correlated, with each other or the target? What hyperparameters are you using? $\endgroup$ – Ben Reiniger Jun 20 '20 at 1:06
  • 1
    $\begingroup$ Adding big update including correlation matrix for features, model code, results, and feature importance. $\endgroup$ – xxanissrxx Jun 20 '20 at 11:46
  • $\begingroup$ Very interesting. I could see the importances being unstable for a less-than-great model, but why the first features always get inflated beats me. Maybe try a different measure of feature importance (shap, permutation), and/or column subsampling? $\endgroup$ – Ben Reiniger Jun 22 '20 at 13:26
-1
$\begingroup$

You can use this chunk of code to plot the feature importance of your data. It is also possible that your data is columned with decreasing importance.

from xgboost import XGBClassifier
from xgboost import plot_importance
from matplotlib import pyplot
# load data
X = 
y = 
model = XGBClassifier()
model.fit(X, y)
plot_importance(model)
pyplot.show()
$\endgroup$
5
  • $\begingroup$ ya I've been using that. The result is that the feature importance is perfectly correlated with the position of that column in xtrain. If I rearrange the columns in xtrain and rerun the model, the feature importance chart perfectly matches the new order of the columns. So XGBoost is just using the first feature in my xtrain and nothing else really. $\endgroup$ – xxanissrxx Jun 19 '20 at 20:15
  • $\begingroup$ Is your input features linearly independent? If not,then it is possible that correleted features importance will be calculated the way they were encountered. Make sure your input columns are linearly independent. $\endgroup$ – SrJ Jun 19 '20 at 20:21
  • $\begingroup$ all features are independent of each other $\endgroup$ – xxanissrxx Jun 19 '20 at 20:26
  • $\begingroup$ You should see correlation matrix of your dataset to confirm that the correlation of your columns are less than 0.8 or around for each of them.If it is then you might have some error in your code or dataset. $\endgroup$ – SrJ Jun 19 '20 at 20:32
  • $\begingroup$ no correlations > 50% $\endgroup$ – xxanissrxx Jun 19 '20 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.