1
$\begingroup$

I am aware of the fact that the Pandas Dataframe's Statistical description can easily be obtained using df.describe(). I am having 2 dataframes of the same dimensions (i.e. 102 columns and 800000 rows for both the dataframes). I would like to depict the fact visually that the 2 dataframes are very similar/have a statistically similar distribution.

I am not sure of how this can be done visually/graphically in Python. The df.describe method provides count, mean, std, min, 25 %, 50 %, 75 % and max values for the dataframes, but it is difficult to infer from in the first instance. I would prefer a visual/graphical method of any type (I am not sure if Boxplot can help?). Any suggested method along with a Minimum Workable Example would be highly appreciated. Cheers!

Presently,

# df1 is dataframe 1
# df2 is dataframe 2
df1.describe() # this gives statistical values of df1 dataframe
df2.describe() # this gives statistical values of df2 dataframe
$\endgroup$
4
$\begingroup$

You can use the Kolmogorov-Smirnov Test. From Wikipedia

In statistics, the Kolmogorov–Smirnov test (K–S test or KS test) is a nonparametric test of the equality of continuous, one-dimensional probability distributions that can be used to compare a sample with a reference probability distribution (one-sample K–S test), or to compare two samples (two-sample K–S test). It is named after Andrey Kolmogorov and Nikolai Smirnov.

Good for us, there is already an implementation for this in Scipy

You can use this dummy code to test it

from scipy import stats
p_value = 0.05
rejected = 0
for col in range(103):
    test = stats.ks_2samp(df1.ix[col,], df2.ix[col,])
    if test[1] < p_value:
         rejected += 1
print("We rejected",rejected,"columns in total")

I made the assumption that the columns that you want to compare are on the same index in both dataframes. If this is not true, you need to find another way. Maybe if they have the same name? You can do it like that.

from scipy import stats
p_value = 0.05
rejected = 0
for col in df1:
    test = stats.ks_2samp(df1[col], df2[col])
    if test[1] < p_value:
         rejected += 1
print("We rejected",rejected,"columns in total")

If the K-S statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same.

Edit: I haven't tried it before, but maybe you can do something like that

Boxplot

from scipy import stats
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#create 2 dataframes with random integers. I don't have data to simulate your case.

df1 = pd.DataFrame(np.random.randint(0,100,size=(10000, 102)), columns=range(1,103))
df2 = pd.DataFrame(np.random.randint(0,100,size=(10000, 102)), columns=range(1,103))

#apply the Kolmogorov-Smirnov Test

p_value = 0.05
p_values = []
for col in range(103):
    test = stats.ks_2samp(df1.iloc[col,], df2.iloc[col,])
    p_values.append(test[1])

#create the box plot

plt.boxplot(p_values)
plt.title('Boxplot of p-values')
plt.ylabel("p_values")
plt.show()

Boxplot

Heatmap

Another way is the heatmap.

import matplotlib.patches as mpatches

plt.rcParams["figure.figsize"] = 5,2
x = range(1,104)
y = np.array(p_values)
fig, (ax,ax2) = plt.subplots(nrows=2, sharex=True)

extent = [x[0]-(x[1]-x[0])/2., x[-1]+(x[1]-x[0])/2.,0,1]
im = ax.imshow(y[np.newaxis,:], cmap="plasma", aspect="auto", extent=extent)
ax.set_yticks([])
ax.set_xlim(extent[0], extent[1])

values = np.unique(np.round(y.ravel(),2))

colors = [ im.cmap(im.norm(value)) for value in values]
# create a patch (proxy artist) for every color 
patches = [ mpatches.Patch(color=colors[i], label="Level {l}".format(l=values[i]) ) for i in range(len(values)) ]
# put those patched as legend-handles into the legend
plt.legend(handles=patches, bbox_to_anchor=(1.05, 2.2), loc=2, borderaxespad=0. )

ax2.plot(x,p_values)

plt.tight_layout()
plt.show()

Heatmap

Code for heatmap: stackoverflow

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the answer. Yes, the columns of both dataframes are having same name and indices. I wonder if there is a way to present this visually/graphically as I mentioned in the question and can you please add a sample code example for plotting this? Cheers $\endgroup$ – JChat Apr 1 '19 at 11:46
  • $\begingroup$ Can you check the edited answer now? $\endgroup$ – Tasos Apr 2 '19 at 6:47
  • $\begingroup$ Thanks for that. Is there a way to describe the colors in the heatmap using a legend or something? $\endgroup$ – JChat Apr 2 '19 at 7:51
  • 1
    $\begingroup$ Not the best implementation. Maybe you could try to group the values. But it's the best I can do it the short time :) $\endgroup$ – Tasos Apr 2 '19 at 8:18
  • $\begingroup$ Thanks! :) It is a helpful answer! $\endgroup$ – JChat Apr 2 '19 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.