I am working on a classification problem. I have 1000+ features in this dataset. I don't know how to select the right variables/ features that can actually contribute to predicting the output. What are the different methods through which I can identify the important variables that can be used out of these 1000+ variables.
Two approaches come to mind:
- Use all of them and perform feature selection to identify indicative features. Algorithms include Information Gain and Mutual Information.
- Hand-pick some features you deem indicative intuitively, or delete the ones you deem non-indicative.
Generally, you have some sort of hypothesis that you are testing where you attest the capability of (a subset of) your features (and the algorithm you'll use) to uniquely identify the label/class they belong to. The features you use are one of the most important things in the classification task. So it is good to spend time to intelligently find out which features contribute.
For this task, you do have a lot of possible features so the automatic feature selection approach might be better.
The general approach in feature selection is to get a score of each feature in the data set and select top features. We can run algorithms like GBM or Random Forest on all the variables simply to get a ranking of variable importance. We can also use χ² (chi-squared) statistic with cross validation to select a user-specified percentile of features with the highest scoring. But, the disadvantage with these approaches is not detecting correlations between features.
We can also use backward elimination: features are tested one by one and the ones that are not statically significant are deleted from data set. In forward selection that starts with out any variable in dataset and then adds variables that are statically significant. Hope this helps.
You can perform logistic regression (if your dependent variable has two classes) that penalizes based on the L1 norm. You can choose the correct sparsity parameter (typically $\lambda$) that chooses how strongly to seek sparsity based on cross-validation. The model will force many non-informative features to be 0. This is a form of feature selection. See here: http://scikit-learn.org/stable/modules/linear_model.html#logistic-regression