7
$\begingroup$

I am trying to understand what it really means to calculate an ANOVA F value for feature selection for a binary classification problem.

As I understand from the calculation of ANOVA from basic statistics, we should have at least 2 samples for which we can calculate the ANOVA value. So does this mean in the Sklearn implementation that these samples are taken from within each feature? What exactly do these samples represent in the case of feature selection for this problem?

I tried to setup a very simple example which I have listed below, but I am still struggling to understand what the ANOVA value really means here? I'm also struggling to understand how to calculate this by hand, which usually helps me see what is happening on the inside.

In this example, repayment status 0 means the loan is repaid and 1 means it defaulted. I have only supplied 5 rows of data to keep it simple.

enter image description here

The code and results are as follows:

enter image description here

$\endgroup$
14
$\begingroup$

Intuition

We have two classes and we want to find a score for each feature saying "how well this feature discriminates between two classes". Now look at the figure bellow. There are two classes red and blue and two features on $x$ and $y$ axes.

enter image description here

$x$ feature is a better separator than $y$ because if we project data on $x$ axis we get two completely separated classes but if we project data onto $y$, two classes have overlap in the middle of axis (comment if we need more clarification).

What makes $x$ better than $y$? As you see in the figure above:

  • According to $x$, two classes are far from each other.
    • Math Translation: The distance between means of class distributions on $x$ is more than $y$.
  • According to $x$, the scatter of classes do not fall on each other but according to $y$ they do. It means that according to $x$, classes are more compact so more probable to not have an overlap with another class.
    • Math Translation: The variance of each single class according to $x$ is less than those of $y$.

Now we can easily say $\frac{distance\_between\_classes}{compactness\_of\_classes}$ is a good score! Higher this score is, better the feature discriminates between classes.

Now we know, according to this definition, what $good$ and $bad$ features mean. Let's find a math formulation to quantize it.

Mathematics (to do on the paper)

Let's formulate our two criteria:

  • Distance between means of class distributions is the numerator. Population is taken into account, I assume for statistical significance (needs a reference from a statistician!). enter image description here
  • A concept similar to sample variances of classes is the denominator. Here instead of dividing sum of squares by $(sample\_population -1)$, we sum up all $(sample\_population -1)$s and divide the final value by them. enter image description here

Now Back To Your Data

To calculate the above you calculate sum of between-class distances and sum of within-class variations for each feature according to different classes. I do it for only one feature. Let's choose Loan.

Class 1: [5000, 18000]
Class 2: [47500, 45600, 49500]

Mean of all points: (47500 + 45600 + 49500 + 5000 + 18000) / 5 = 33120
Mean 1: (5000 + 18000) / 2 = 11500
Mean 2: (47500 + 45600 + 49500) / 3 = 47533
Numerator: 2 x (11500 - 33120)^2 + 3 x (47533 - 33120)^2 = 1,558,052,507

For denominator we go with Sum of Squares Within class (it is simply the numerator in formulation of sample variance):

SSW 1: (5000 - 11500)^2 + (18000 - 11500)^2 = 84,500,000 
SSW 2: (47500 - 47533)^2 + (45600 - 47533)^2 + (49500 - 47533)^2 = 7,606,667
Na = 2, Nb = 3 --> (Na - 1) + (Nb - 1) = 1 + 2 = 3
Denominator: (84,500,000 + 7,606,667)/3 = 30,702,222 

Now the F-Score for feature Loan is:

F-Score: 1,558,052,507 / 30,702,222 = 50.74

as you see with your calculation in Python.

Note

  • I tried to explain in a simple way. For example the denominator of sample variance is called degree of freedom but I skipped those terms for simplicity.
  • Just understand the main idea. Further the means and smaller the within variances, better the feature is. You can formulate it yourself as well (however you will not have p-values anymore ;) )
  • Finding P-values and understanding what it means, is another story which I skipped.

Hope it helped. Good Luck!

$\endgroup$
6
  • $\begingroup$ This is amazing! Thank you so much I completely understand now! $\endgroup$ – JuniorDataGuyAKL May 20 '20 at 1:09
  • $\begingroup$ You are welcome my friend! Welcome to the community :) $\endgroup$ – Kasra Manshaei May 20 '20 at 7:22
  • $\begingroup$ Does this mean applying this to categorical/mixed type features doesn't make much sense ?. Since there is really a distance metric between categorical values $\endgroup$ – imantha Feb 15 at 18:20
  • $\begingroup$ Right. Between cat features, you may use chi-squared. But between num and cat features, you can fund some correlation by seeing the histogram of num values according to cat values $\endgroup$ – Kasra Manshaei Feb 16 at 17:12
  • $\begingroup$ @KasraManshaei : does the described approach work also for heavily imbalanced classification problems? I see no reason in your explanation why it shouldn't. $\endgroup$ – AlexGuevara Apr 20 at 12:42
1
$\begingroup$

F-score calculated by f_classif can be calculated by hand using the following formula shown in the image: Reference video Screenshot from video

Intuitively, it is the ratio of (variance in output feature(y) explained by input feature(X) and variance in output feature(y) not explained by input feature(X)).

Example :

Using sklearn code for f_classif

from sklearn.feature_selection import f_classif
import numpy as np
X = np.array([5000, 18000, 47500, 45600, 49500]).reshape(-1,1)
y = np.array([1,1,0,0,0])
F,pval = f_classif(X,y)
print(F,pval)

[50.74816155] [0.00569324]

Validation using above formula

X = np.array([5000, 18000, 47500, 45600, 49500])
X1 = np.array([5000, 18000])
X2 = np.array([47500, 45600, 49500])
mu = np.mean(X)                                   # overall mean
mu1 = np.mean(X1)                                 # mean of X1
mu2 = np.mean(X2).                                # mean of X2
SSm = np.sum(((X-mu)**2))                         # SS(mean)
SSf = np.sum((X1-mu1)**2) + np.sum((X2-mu2)**2)   # SS(fit)
p_fit = 2
p_mean = 1
F = ((SSm - SSf)*(X.shape[0]-p_fit))/((p_fit - p_mean)*SSf)
print(F)

50.74

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.