Determine useful features for machine learning model

Question

I am working with a dataset with hundreds of features. I wish to create a simple machine learning model using 7-10 features from the original dataset. My question is this:

What quantitative metrics can I use to determine that a feature will be useful to the learning model?

I have been comparing the distribution of the target mean over the feature groups, to the target mean of the dataset as a whole. For example, take a binary feature X and a binary target. Let's say the target has a mean of 0.10 when taken over the entire dataset.

To analyze the feature X, I take the target mean for each group within feature X.

mean (X=0) = 0.07
mean (X=1) = 1.15

In this way, I can observe the effect of a feature on the target.

I know that there must be some stronger metrics which people use to determine the strength of a feature. In school I used p tests to determine the statistical significance of a variable. Is there an analog in DS/ML?

Answer 1

I suggest taking a look at this page for some more ideas:

Feature Selection

That being said a couple of ideas that come to mind quickly, is to:

1. use a tree based method (like Random Forest) and look at your feature importances. Scikit Learn has a handy class for doing just that see the link above.
2. Use some sort of regularization/penalty like L1 or L2 regularization. That will force non-useful features to have parameters close to zero.
3. Recursively remove variables and see what the resulting output is and cross-validate. Again sklearn has a method for this.

Generally, these methods will be "expensive" as you are fitting multiple models to get you where you need to go.

Answer 2

You could stepwise (backwards or forward) remove or add features to your feature subset. For the Feature Selection procedure, you need a metric to measure which features should be included in the reduced data set of your available data. One important entropy measure is Mutual Information.

Mutual information is a measure between two (possibly multi-dimensional) random variables X and Y, that quantifies the amount of information obtained about one random variable, through the other random variable. The mutual information is given by

I(X;Y)=∫∫p(x,y)logp(x,y)/p(x)p(y)dxdy,

where p(x,y) is the joint probability density function of X and Y, and where p(x) and p(y) are the marginal density functions. The mutual information determines how similar the joint distribution p(x,y) is to the products of the factored marginal distributions. If X and Y are completely unrelated (and therefore independent), then p(x,y) would equal p(x)p(y), and this integral would be zero.

If we assume that X is one Feature and Y is the target variable then we could measure their Mutual Information. We would like to keep the features with the highest mutual information between them and the target variable.

Apart from the stepwise algorithms for selecting the appropriate features there are also some greedy methods also trying to maximize the Mutual Information between the joint distribution and the target variable.

Below you can find some indicative links

A review of feature selection methods based on mutual information

Feature selection using Joint Mutual Information Maximisation

Answer 3

There are many ways to estimate how good a feature is, in predicting y_i. One of the good methods is to build a proper ML model using just the feature you wanna check its importance. In this case, we will build a logistic regression model using only features which you wanna check if it is important or not.
Do remember that if it a Categorical Feature encodes it to vector form based on the model which you are using, for example (one-hot encoded) works best for linear models and Response Coding works well for tree-based Models.