1
$\begingroup$

I have a binary classification problem, let's say people can buy or not buy a certain product. Now unlike a standard prediction task, I only want to find which features are the most important for the person's decision to buy.

Which metric should I use to optimize the algorithm? Maximize out of sample accuracy like when I would be interested in making the best prediction? Or maximize fit and don't care about overfitting? A mixture of both?

I am using xgboost.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Ideally you should select features in the same way you select the best model hyperparameters, with a validation set.

You are interested in how the features work on unseen data, not your training data.

Depending on your feature selection algorithm, feature search will be intractable with larger data. Naive feature selection is O(2^n) AND some model hyperparameters could be dependent on your choice of features.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I have the feeling that this answer is about feature selection. However, I think this setting migh be slightly different, my task is not to select features for a modeling purpose but to say which features are the most important for the outcome. Imagine that the features are different actions taken by the consumer on a webpage. I want to know which actions are the biggest predictor of buying the product. $\endgroup$
    – František Kaláb
    Commented Jun 13, 2018 at 23:13
  • $\begingroup$ 1. Drop a feature, 2. Train 3. Score on validation set 4. Repeat for all features, putting back in the feature previously dropped (LeaveOneOut) 5. The feature which, when dropped, caused the score to drop the most is the most important. 6. Leave that feature out and repeat. As Bert says though, this can become basically intractable for large feature sets / slow training models. $\endgroup$
    – Ken Syme
    Commented Jul 13, 2018 at 20:30
0
$\begingroup$

A potential approach:

  1. Build the most accurate model possible, using most of the data: parameter tuning using cross-validation. Accuracy should be tested on out-of-sample data. Alternatively use the default xgboost parameters and proceed to step 2.
  2. Assess feature importance using the tuned parameters: build multiple models on random subsets of data (using the same set of tuned parameters from (1) in each model) and extract feature importance. The more aggressive you are in bagging, the more models you will need to build. That is: smaller subsamples -> more models.
  3. Combine / aggregate feature importance measures from models to obtain the most important features.
Improve this answer
$\endgroup$
2
  • $\begingroup$ Could you elaborate on why to do 2 & 3? What is achieved by that over just doing 1 and knowing this is the most out-of-sample accurate model so I can take the feature importance from there? $\endgroup$
    – František Kaláb
    Commented Aug 14, 2018 at 13:17
  • $\begingroup$ My thinking is the following: xgboost has randomness inherent in it. Unless the random seed is set, multiple runs using the same set of parameters may result in slightly different results. Running the process multiple times and aggregating the feature importance scores might give you more stable results. Further, using different subsets of your data set may also reduce over-fitting. $\endgroup$
    – bradS
    Commented Aug 16, 2018 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.