# How modelling is affected by similar feature distributions across classes?

I have grouped my dataset according to labels (good and bad customers), in an attempt to test how each feature is distributed within. Along this, I found some feature has almost exact distribution in both groups and

I'm wondering what insight can I learn from this? Can it mean that the feature isn't that significant to data prediction and can be dropped?

• Which plot is this and how did you conclude so ? – 10xAI Dec 3 '20 at 16:37
• I used dataframe.hist from pandas. My assumption was if a feature has a strong influence on label, then the distribution of the values of given feature should differ. For example a feature is study hours (0,1,2,3,4) are it's values and label is pass/fail. Then shouldn't we find more of class 5,4,3 in label 1 and more 0,1,2 in label 2 – Raed Tabani Dec 4 '20 at 0:05
• Raed, just a note that the y axis scales differ in the two plots. – hH1sG0n3 Dec 4 '20 at 11:55
• I assumed its because the data is imbalanced, label1 is twice as many as label2 – Raed Tabani Dec 4 '20 at 13:57

## EDA

What you are doing there, falls under Exploratory Data Analysis (EDA). A better way to investigate how your features between them and are distributed across classes is a correlogram, sometimes referred to as pairplot.

A correlogram helps with

• investigating relationships between pairs of numerical features, via scatter plots for each pair of features
• inspecting the distribution of each feature, using a histogram or a density plot in the diagonal of the pairplot.

You can easily create one of those in seaborn:

# library & dataset
import seaborn as sns
import matplotlib.pyplot as plt

# Basic correlogram
sns.pairplot(df, kind="scatter")
sns.plt.show()

# Annotate classes with in different colours
sns.pairplot(df, kind="scatter", hue="species")

# Use regression instead of scatter
sns.pairplot(df, kind="reg", hue="species")


You can see how classes may or may not form clusters which an algorithm could potentially draw hyperplanes by combining and transforming features.

Can it mean that the feature isn't that significant to data prediction and can be dropped?

The y axis maximum value differs between the two plots you have posted, and so it is not easy visually inspect fairly.

One could potentially assume that the model may not "work" with that particular feature to begin with, should there be other variables in that dataset that are easier to separate. This assumption however (a) can be considered naïve given how ML algorithms work to transform the feature space in order to make classes separable and therefore (b) does not constitute grounds for dropping this feature.

A less useful feature in your dataset can still increase the performance of your model. In the case of unbearably large feature space you can use dimensionality reduction techniques (PCA, kernel PCA, autoencoders).

Dropping features usually relates to how well these relate to the ground truth e.g. noise, or how they may affect feature importance and model stability e.g. multicollinearities.

• Thanks for pointing out the EDA , as for answer i felt you were saying that droping features is not favourable because of "transformation of feature space" unless we have large dimension – Raed Tabani Dec 4 '20 at 0:11
• Well, it is absolutely fine to compare distributions of a feature across classes and potentially derive a level of confidence as to whether they are significantly different, however that is not necessarily part of a ML modelling pipeline. Even in the simplest forms of statistical modelling e.g. regression, the model (and not the modeller) will decide what features to used and which ones to penalise (see regularisation). Dropping features usually relates to how well these relate to the ground truth e.g. noise, or how they may affect feature importance and stability e.g. multicollinearities. – hH1sG0n3 Dec 4 '20 at 11:13