14
$\begingroup$

I would like to run some machine learning model like random forest, gradient boosting, or SVM on my dataset. There are more than 200 predictor variables in my dataset and my target classes are a binary variable.

Do I need to run feature selection before the model fitting? Does it affect the model performance significantly or is there not much difference if I directly fit the model using all predictor variables?

Improve this question
$\endgroup$
5
  • $\begingroup$ How big is your dataset? If you have thousands of samples and 200 predictor variables, chances are quite high that with a model like Random Forests you will be able to already achieve quite a high performance. Further feature selection will then further improve your performance. $\endgroup$
    – Archie
    Jan 4, 2017 at 9:05
  • $\begingroup$ @Archie Yes, my dataset size is similar as what you mentioned. 'Further feature selection', do you mean to conduct the feature selection before the model fitting and it can do favor for the model performance? $\endgroup$
    – LUSAQX
    Jan 4, 2017 at 9:07
  • $\begingroup$ I mean I would first have a go with all features, Random Forests would be a great classifier to start with. If you then want to push the performance higher, I would look at for example the feature importances to select the most significant features. $\endgroup$
    – Archie
    Jan 4, 2017 at 9:29
  • $\begingroup$ Ok. That is what I have done so far. I will try some feature selection methods before the model fitting to see if there is any improvement by then. $\endgroup$
    – LUSAQX
    Jan 4, 2017 at 9:50
  • $\begingroup$ A short answer from my recent practice, the feature selection is necessary for model comparison. Some algorithms would work better on some set of features whilst some other algorithms on another set. $\endgroup$
    – LUSAQX
    Sep 11, 2017 at 22:39

3 Answers 3

Reset to default
12
$\begingroup$

Feature selection might be consider a stage to avoid. You have to spend computation time in order to remove features and actually lose data and the methods that you have to do feature selection are not optimal since the problem is NP-Complete. Using it doesn't sound like an offer that you cannot refuse.

So, what are the benefits of using it?

  1. Many features and low samples/features ratio will introduce noise into your dataset. In such a case your classification algorithm are likely to overfit, and give you a false feeling of good performance.
  2. Reducing the number of features will reduce the running time in the later stages. That in turn will enable you using algorithms of higher complexity, search for more hyper parameters or do more evaluations.
  3. A smaller set of feature is more comprehendible to humans. That will enable you to focus on the main sources of predictability and do more exact feature engineering. If you will have to explain your model to a client, you are better presenting a model with 5 features than a model with 200 features.

Now for your specific case: I recommend that you'll begin in computing the correlations among the features and the concept. Computing correlations among all features is also informative. Note that there are many types of useful correlations (e.g., Pearson, Mutual information) and many attributes that might effect them (e.g., sparseness, concept imbalance). Examining them instead of blindly go with a feature selection algorithm might save you plenty of time in the future.

I don't think that you will have a lot of running time problems with your dataset. However, your samples/features ratio isn't too high so you might benefit from feature selection.

Choose a classifier of low complexity(e.g., linear regression, a small decision tree) and use it as a benchmark. Try it on the full data set and on some dataset with a subset of the features. Such a benchmark will guid you in the use of feature selection. You will need such guidance since there are many options (e.g., the number of features to select, the feature selection algorithm) an since the goal is usually the predication and not the feature selection so feedback is at least one step away.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks a lot. But for a non-linear classifier like random forest, does it also require the predictor variables independent from each other? I guess not but could be wrong. How the correlation guide the feature selection? $\endgroup$
    – LUSAQX
    Jan 4, 2017 at 20:19
  • $\begingroup$ Random forest is a collection of trees that cope well with dependent variable. That is since in each node in the tree, the dataset is conditioned by all the variables above it. The problem is that the tree growing is heuristic so the choice of the above variable might have not been optimal. $\endgroup$
    – DaL
    Jan 5, 2017 at 7:57
  • $\begingroup$ The correlation only compare pairs of variables and therefore cannot give you the full picture. On the other hand, you get the result in O(n^2) and not O(2^n)... The guidance is usually specific to the dataset so I find it hard to explain it this way. Some examples is the removal of variables that are redundant (very correlated to other variables). Examining the strength of the correlation might indicate whether you can use a small model or will be needed to use many weak learners. Identifying subset that are not too correlated with each other might indicate the co-training will be useful. $\endgroup$
    – DaL
    Jan 5, 2017 at 8:03
6
$\begingroup$

I've posted a very similar question on Cross Validated few months ago and got a very large number of responses. Read the responses and the comments.

https://stats.stackexchange.com/questions/215154/variable-selection-for-predictive-modeling-really-needed-in-2016

Improve this answer
$\endgroup$
1
  • $\begingroup$ Great question! $\endgroup$
    – aerin
    Oct 12, 2017 at 18:29
-1
$\begingroup$

Yes, feature selection is one of the most crucial task for machine learning problems, after performing data wrangling and cleaning. you can find the functions implementing the feature selection process using XGBOOST feature importance here.

https://github.com/abhisheksharma4194/Machine-learning

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.