I am working with a dataset with hundreds of features. I wish to create a simple machine learning model using 7-10 features from the original dataset. My question is this:

What quantitative metrics can I use to determine that a feature will be useful to the learning model?

I have been comparing the distribution of the target mean over the feature groups, to the target mean of the dataset as a whole. For example, take a binary feature X and a binary target. Let's say the target has a mean of 0.10 when taken over the entire dataset.

To analyze the feature X, I take the target mean for each group within feature X.

mean (X=0) = 0.07
mean (X=1) = 1.15

In this way, I can observe the effect of a feature on the target.

I know that there must be some stronger metrics which people use to determine the strength of a feature. In school I used p tests to determine the statistical significance of a variable. Is there an analog in DS/ML?


I suggest taking a look at this page for some more ideas:

Feature Selection

That being said a couple of ideas that come to mind quickly, is to:

  1. use a tree based method (like Random Forest) and look at your feature importances. Scikit Learn has a handy class for doing just that see the link above.
  2. Use some sort of regularization/penalty like L1 or L2 regularization. That will force non-useful features to have parameters close to zero.
  3. Recursively remove variables and see what the resulting output is and cross-validate. Again sklearn has a method for this.

Generally, these methods will be "expensive" as you are fitting multiple models to get you where you need to go.

| improve this answer | |
  • $\begingroup$ This link is tremendous. I expected to see chi squared and other correlation tests pop up in ML. Thank you Ryan! $\endgroup$ – fermi May 30 '18 at 17:57
  • $\begingroup$ I am a bit surprised to see the removal of low variance features. A Boolean variable with low variance could still contain important information about the target, no? Your boolean variable may be 99% = 1 and 1 % = 0 but I would think that the 1% could still indicate a strong correlation with the target. $\endgroup$ – fermi May 31 '18 at 15:09
  • $\begingroup$ You are right, I don't think that it is "the best" method, simply "a" method that sklearn provides. It is also the easiest to method to understand conceptually, and frankly a little naive. Hence, I didn't mention it in my answer. $\endgroup$ – Ryan May 31 '18 at 15:23

You could stepwise (backwards or forward) remove or add features to your feature subset. For the Feature Selection procedure, you need a metric to measure which features should be included in the reduced data set of your available data. One important entropy measure is Mutual Information.

Mutual information is a measure between two (possibly multi-dimensional) random variables X and Y, that quantifies the amount of information obtained about one random variable, through the other random variable. The mutual information is given by


where p(x,y) is the joint probability density function of X and Y, and where p(x) and p(y) are the marginal density functions. The mutual information determines how similar the joint distribution p(x,y) is to the products of the factored marginal distributions. If X and Y are completely unrelated (and therefore independent), then p(x,y) would equal p(x)p(y), and this integral would be zero.

If we assume that X is one Feature and Y is the target variable then we could measure their Mutual Information. We would like to keep the features with the highest mutual information between them and the target variable.

Apart from the stepwise algorithms for selecting the appropriate features there are also some greedy methods also trying to maximize the Mutual Information between the joint distribution and the target variable.

Below you can find some indicative links

A review of feature selection methods based on mutual information

Feature selection using Joint Mutual Information Maximisation

| improve this answer | |
  • $\begingroup$ I like this idea because mutual information is so simple to calculate. I would think that mutual information is related to a univariate statistical hypothesis tests such as chi-squared, so both tests would give me similar information about the feature/target correlation. $\endgroup$ – fermi May 31 '18 at 15:03

There are many ways to estimate how good a feature is, in predicting y_i. One of the good methods is to build a proper ML model using just the feature you wanna check its importance. In this case, we will build a logistic regression model using only features which you wanna check if it is important or not.
Do remember that if it a Categorical Feature encodes it to vector form based on the model which you are using, for example (one-hot encoded) works best for linear models and Response Coding works well for tree-based Models.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.