How to understand ANOVA-F for feature selection in Python. Sklearn SelectKBest with f_classif

Question

I am trying to understand what it really means to calculate an ANOVA F value for feature selection for a binary classification problem.

As I understand from the calculation of ANOVA from basic statistics, we should have at least 2 samples for which we can calculate the ANOVA value. So does this mean in the Sklearn implementation that these samples are taken from within each feature? What exactly do these samples represent in the case of feature selection for this problem?

I tried to setup a very simple example which I have listed below, but I am still struggling to understand what the ANOVA value really means here? I'm also struggling to understand how to calculate this by hand, which usually helps me see what is happening on the inside.

In this example, repayment status 0 means the loan is repaid and 1 means it defaulted. I have only supplied 5 rows of data to keep it simple.

The code and results are as follows:

Answer 1

Intuition

We have two classes and we want to find a score for each feature saying "how well this feature discriminates between two classes". Now look at the figure bellow. There are two classes red and blue and two features on $x$ and $y$ axes.

$x$ feature is a better separator than $y$ because if we project data on $x$ axis we get two completely separated classes but if we project data onto $y$, two classes have overlap in the middle of axis (comment if we need more clarification).

What makes $x$ better than $y$? As you see in the figure above:

• According to $x$, two classes are far from each other.
  • Math Translation: The distance between means of class distributions on $x$ is more than $y$.
• According to $x$, the scatter of classes do not fall on each other but according to $y$ they do. It means that according to $x$, classes are more compact so more probable to not have an overlap with another class.
  • Math Translation: The variance of each single class according to $x$ is less than those of $y$.

Now we can easily say $\frac{distance\_between\_classes}{compactness\_of\_classes}$ is a good score! Higher this score is, better the feature discriminates between classes.

Now we know, according to this definition, what $good$ and $bad$ features mean. Let's find a math formulation to quantize it.

Mathematics (to do on paper)

Let's formulate our two criteria:

• Distance between means of class distributions is the numerator. Population is taken into account, I assume for statistical significance (needs a reference from a statistician!).
• A concept similar to sample variances of classes is the denominator. Here instead of dividing sum of squares by $(sample\_population -1)$, we sum up all $(sample\_population -1)$s and divide the final value by them.

Now Back To Your Data

To calculate the above you calculate sum of between-class distances and sum of within-class variations for each feature according to different classes. I do it for only one feature. Let's choose Loan.

Class 1: [5000, 18000]
Class 2: [47500, 45600, 49500]

Mean of all points: (47500 + 45600 + 49500 + 5000 + 18000) / 5 = 33120
Mean 1: (5000 + 18000) / 2 = 11500
Mean 2: (47500 + 45600 + 49500) / 3 = 47533
Numerator: 2 x (11500 - 33120)^2 + 3 x (47533 - 33120)^2 = 1,558,052,507

For denominator we go with Sum of Squares Within class (it is simply the numerator in formulation of sample variance):

SSW 1: (5000 - 11500)^2 + (18000 - 11500)^2 = 84,500,000 
SSW 2: (47500 - 47533)^2 + (45600 - 47533)^2 + (49500 - 47533)^2 = 7,606,667
Na = 2, Nb = 3 --> (Na - 1) + (Nb - 1) = 1 + 2 = 3
Denominator: (84,500,000 + 7,606,667)/3 = 30,702,222 

Now the F-Score for feature Loan is:

F-Score: 1,558,052,507 / 30,702,222 = 50.74

as you see with your calculation in Python.

Note

• I tried to explain it in a simple way. For example, the denominator of sample variance is called degree of freedom but I skipped those terms for simplicity.
• Just understand the main idea. Further the means and smaller the within variances, better the feature is. You can formulate it yourself as well (however you will not have p-values anymore ;) )
• Finding P-values and understanding what it means is another story, which I skipped.

Hope it helped. Good Luck!

Answer 2

F-score calculated by f_classif can be calculated by hand using the following formula shown in the image: Reference video

Intuitively, it is the ratio of (variance in output feature(y) explained by input feature(X) and variance in output feature(y) not explained by input feature(X)).

Example :

Using sklearn code for f_classif

from sklearn.feature_selection import f_classif
import numpy as np
X = np.array([5000, 18000, 47500, 45600, 49500]).reshape(-1,1)
y = np.array([1,1,0,0,0])
F,pval = f_classif(X,y)
print(F,pval)

[50.74816155] [0.00569324]

Validation using above formula

X = np.array([5000, 18000, 47500, 45600, 49500])
X1 = np.array([5000, 18000])
X2 = np.array([47500, 45600, 49500])
mu = np.mean(X)                                   # overall mean
mu1 = np.mean(X1)                                 # mean of X1
mu2 = np.mean(X2).                                # mean of X2
SSm = np.sum(((X-mu)**2))                         # SS(mean)
SSf = np.sum((X1-mu1)**2) + np.sum((X2-mu2)**2)   # SS(fit)
p_fit = 2
p_mean = 1
F = ((SSm - SSf)*(X.shape[0]-p_fit))/((p_fit - p_mean)*SSf)
print(F)

50.74