Feature selection and classification accuracy relation

Question

One of the methodology to select a subset of your available features for your classifier is to rank them according to a criterion (such as information gain) and then calculate the accuracy using your classifier and a subset of the ranked features.

For example, if your features are A, B, C, D, E, and if they are ranked as follow D,B,C,E,A, then you calculate the accuracy using D, then D, B then D, B, C, then D, B, C, E... until your accuracy starts decreasing. Once it starts decreasing, you stop adding features.

In example1 (above), you would pick features F, C, D, A and drop the other features as they decrease your accuracy.

That methodology assumes that adding more features to your model increases the accuracy of your classifier until a certain point after which adding additional features decreases the accuracy (as seen in example 1)

However, my situation is different. I have applied the methodology described above and I found that adding more features decreased the accuracy up until a point after which it increases.

In a scenario such as this one, how do you pick your features? Do you only pick F and drop the rest? Do you have any idea why the accuracy would decrease and then increase?

Answer 1

feature selection involves several approaches just like methods for machine learning. Idea is to keep most relevant but not redundant feature for predictive model that can yield optimal accuracy.

In your case, I can not see which method you are using for feature selection but assuming that you are not taking account of multivariate nature of feature dependency. Say you have N features, likely reason that your model accuracy drops after n top feature(s) but improves by adding n+k (where n < k < N when features are in descending order based on information gain) is due to inter-dependency (more relevance and less redundancy) of top n and k features. Univariate feature selection does not necessarily get optimal model accuracy when features are inter-dependent and not mutually exclusive. From philosophical point of view, set of optimal features is analogous to a quote by Aristotle: "The whole is greater than the sum of its parts"!

For optimal feature selection, I often is Caret package in R language where one may do feature selection using recursive feature elimination (RFE) among several other approaches. There is also a package called mRMRe to do feature selection based on maximum relevance, minimal redundancy.

Best,
Samir

Answer 2

About the specific question

You should not expect a specific behavior (increase and then decrease of accuracy) while you select subset of features, since this will be totally dependent on the problem (and each model)

When you calculate the variable importance of features, you are taking into account the contribution of all the the features at the same time. Once you select a subset of features and build a new model, you will get a different representation or modelling of the problem (which does not take into account the other features - informative or not -).

Now, you want to select the best number of features. This will also depend from your problem and the characteristics or conditions you need to fulfill. If you actually need to have the fewer features possible while optimizing prediction accuracy, you can select the lowest number of features that achieves the lowest error... and, if you have different cases with very similar errors, then pick a threshold, observe the top cases whose pairwise difference of errors is lower than the threshold, and select one (for example the one with lower number of features - since the errors are marginally the same -).

Consider Recursive Feature Elimination

The method you are using might not be the most stable approach. You should consider trying something like recursive feature elimination (RFE), a wrapper method where you build the classifier, rank all the features, remove the worst and rebuild the model on the remaining features. Then you repeat the method again. This will tend to be more stable...and you should expect different ranking everytime.

Variance is also a critical factor

Beyond the actual error (or accuracy) the model is giving you with each subset, you should consider to build each model through a cross-validation procedure and take into account both the mean error of the folds, and the standard deviation of these errors. If the standard deviation is high, then the selected subset of features is not stable, and will tend to vary plenty when testing with unseen data. This is important to evaluate the expected generalization capabilities of the model, and could be helpful for deciding between models (built with different subsets).

Answer 3

You need to remove both redundant and irrelevant features from your data set. It can be seen that there are irrelevant and redundant features in your data set.

I recommend you to look at the minimum Redundancy Maximum Relevance Feature Selection(MRMR) algorithm. It is a very popular and powerful filter before your train the model.

"However, my situation is different. I have applied the methodology described above and I found that adding more features decreased the accuracy up until a point after which it increases"

It is also possible, but this will lead to more complex model.

Answer 4

Generally there are three classes of feature selection algorithms.

• Filter methods which analyze the intrinsic properties of the data and assign a score to each feature, not involving any model. Some examples are fold change, student t-test.

• Wrapper methods which different subsets of features are selected through the specific algorithms. We then fit into the classification or regression model to evaluate each selection and pick the one with best fitness value. Some examples are Genetic Algorithm for feature selection, Monte Carlo optimization for feature selection, forward/backward stepwise selection.

• Embedded methods which allows the model itself to pick the features having best contribution to the fitness of the model. Typical ones are LASSO, ridge regression.

Here is a great article in details of introduction to feature selection.