3
$\begingroup$

I was attempting to determine whether a feature is important or not base on its kde distribution for target variable. I am aware how to plot the kde plot and guess after looking at the plots, but is there a more formal doing this? Such as can we calculate the area of non overlapping area between two curves?

When I googled for the area between two curves there are many many links but none of them could solve my exact problem.

NOTE:
The main aim of this plot is to find whether the feature is important or not. So, please suggest me further if I am missing any hidden concepts here.

What I am trying to do is set some threshold such as 0.2, if the non-overlapping area > 0.2, then assert that the feature is important, otherwise not.

MWE:

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

df = sns.load_dataset('titanic')

x0 = df.loc[df['survived']==0,'fare']
x1 = df.loc[df['survived']==1,'fare']

sns.kdeplot(x0,shade=1)
sns.kdeplot(x1,shade=1)

Output

enter image description here

Similar links

$\endgroup$
2
$\begingroup$

There are different ways to measure the similarity between two functions.

One option is to define the overlap between both functions as their dot-product:

# ensure both functions are normalized (self-overlap = 1)
x0 /= np.dot(x0, x0)
x1 /= np.dot(x1,x1)
overlap = np.dot(x0,x1)

Instead of multiplying the individual function values as above, you may calculate their difference and take, for example, the mean. This is similar to a loss function in machine learning:

d = np.absolute(x0 - x1)
mae = np.mean(d)    # mean absolute error
mse = np.mean(d**2) # mean square error

If the data is represented on different grid this approach won't work. But you can interpolate your functions and represent them on a new, common grid. A basic example is available in the SciPy documentation. The interpolated data can then be used in the above code snippets.

$\endgroup$
5
  • $\begingroup$ It gives ValueError: shapes (549,) and (342,) not aligned: 549 (dim 0) != 342 (dim 0) $\endgroup$
    – usermay14
    Jun 14 '20 at 17:32
  • $\begingroup$ Well, the data needs to be represented on the same grid of course. If it is not, you can try interpolation. $\endgroup$
    – Feodoran
    Jun 14 '20 at 17:43
  • $\begingroup$ So, how to calculate the overlap given in the question? $\endgroup$
    – usermay14
    Jun 14 '20 at 17:45
  • $\begingroup$ I don't know your data and the relations of their grids, so I just linked to the SciPy documentation on 1D-interpolation. $\endgroup$
    – Feodoran
    Jun 14 '20 at 17:58
  • $\begingroup$ that's ok, I appreciate your effort. $\endgroup$
    – usermay14
    Jun 14 '20 at 18:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.