I have the following three datasets.


data_a is real data and the other two are the simulated ones. Here I am trying to check which one (data_b or data_c) is closest or closely resembles to data_a. Currently I am doing it visually and with ks_2samp test (python).


I graphed the cdf of real data vs cdf of simulated data and try to see visually that which one is the closest.

cdf of data_a vs data_b

Above is the cdf of data_a vs cdf of data_b enter image description here

Above is the cdf of data_a vs cdf of data_c

So by visually seeing it one might can say that data_c is more closer to data_a then data_b but it is still not accurate.

Kolmogorov-Smirnov (KS) Test

Second method is the KS-test where I tested data_a with data_b as well as data_a with data_c.

>>> stats.ks_2samp(data_a,data_b)
Ks_2sampResult(statistic=0.5923076923076923, pvalue=0.02134674813035231)
>>> stats.ks_2samp(data_a,data_c)
Ks_2sampResult(statistic=0.4692307692307692, pvalue=0.11575018162481227)

From above we can see that statistic is lower in when we tested data_a with data_c so data_c should be closer data_a than data_b. I didn't consider the pvalue as it would not be appropriate to think of it as a hypotheses test and use the p-value obtained because the test is designed with the null hypothesis predetermined.

So my question here is that if I am doing this correctly and also is there any other better way to do it? Thank You

  • 1
    $\begingroup$ Are you sure those are CDF plots of the data? Empirical CDF plots of that data are these. It appears that you are plotting the values on y-axis over some standard x-axis. Are the values in your variable observations or points on the CDF? $\endgroup$
    – Edmund
    Jun 9 '19 at 11:22
  • $\begingroup$ @Fatemehhh Hello I am not plotting these two plots in a single one as there are hundreds of other datasets I need to compare with the real dataset and find out which one is the closest to the real dataset. Basically I am trying to approximate a parameter which is generating different simulated datasets at different values and when the simulated dataset is closest to the real dataset then there is the right parameter! $\endgroup$ Jun 9 '19 at 16:46
  • $\begingroup$ @Edmund I think you are correct, those values are the variable observation and I am plotting the values on y-axis over some standard x-axis. x_points=np.asarray(list(range(0,len(data_a)))) >>> x_points=x_points/len(data_a) >>> plt.plot(x_points,data_a) >>> x_points=np.asarray(list(range(0,len(data_b)))) >>> x_points=np.asarray(list(range(0,len(data_c)))) >>> x_points=x_points/len(data_c) >>> plt.plot(x_points,data_c) This is the code. But my question is how one can find the closeness between the two datasets $\endgroup$ Jun 9 '19 at 16:48
  • 2
    $\begingroup$ Wouldn't it be better simply to fit a distribution to the observed data? It seems a bit convoluted and computationally expensive to go about it by guessing parameters, simulating results, and testing similarity. $\endgroup$
    – Edmund
    Jun 11 '19 at 0:30
  • 1
    $\begingroup$ This feels a lot like an XY Problem. Please share the problem you are attempting to solve instead of this part of the solution to the problem. $\endgroup$
    – Edmund
    Jun 11 '19 at 11:16

You could take an Information Theory approach by finding the lowest Kullback–Leibler divergence between the distributions. There is a KL divergence option within SciPy's entropy function.

>>> from scipy.stats import entropy

>>> p = [0.21,0.24,0.36,0.56,0.67,0.72,0.74,0.83,0.84,0.87] # Data removed to make equal sizes: [0.91,0.94,0.97]
>>> q_1 = [0.13,0.21,0.27,0.34,0.36,0.45,0.49,0.65,0.66,0.90]
>>> print(entropy(p, q_1)) 

>>> q_2 =[0.14,0.18,0.19,0.33,0.45,0.47,0.55,0.75,0.78,0.82]
>>> print(entropy(p, q_2))

The second simulated distribution is closer than the first simulated distribution to the real distribution.

If you are interested in inference, you could run many simulations and compute p-values. That process is a variation of permutation testing.

  • $\begingroup$ Thank You for your answer but why would you remove the data from the real distribution. I understand that the scipy method would only work with equal datasets but which elements of the data to remove or why? Here i see you removed the last ones. Also isn't there anyway to make it work on unequal datasets? $\endgroup$ Jun 22 '19 at 5:51
  • 3
    $\begingroup$ I arbitrarily removed data to get the code sample working. If it was my project, I would not remove real data. I would simulate data to be the same size as the real data. Handling unequal datasets would not be necessary because I would generate data of equal sizes. Since this is your project, you can choose which data to remove if you choose to use SciPy's KL divergence method. $\endgroup$ Jun 22 '19 at 15:29
  • $\begingroup$ The Wasserstein distance in the answer below this one is by far the better measure as explained in the following link containing an illustrated example that shows KL-divergence to be indifferent to the location of a distribution's mode, whereas Wasserstein distance is much more detailed and would give a different meaningful measurement regarding how far apart different datasets are: stats.stackexchange.com/questions/295617/… $\endgroup$
    – develarist
    Nov 8 '20 at 11:46

Consider using the Earth Mover's Distance (i.e., the Wasserstein-1 distance), which (similar to the KL-divergence) can be used to compute the "distance" between sets of points (or rather the empirical distribution induced by them). There is a method in scipy for it, as well as this library.


  • You do not need to have the same number of points in each set (the EMD allows "splitting" mass).
  • An advantage over the KL-divergence is that the KLD can be undefined or infinite if the distributions do not have identical support (though using the Jensen-Shannon divergence mitigates this). Further, estimating entropies is often hard and not parameter-free (usually requiring binning or KDE), while one can solve EMD optimizations directly on the input data points.
  • An advantage over simple statistics (e.g., comparing means and covariances, or norms) is that they tend to lose information. E.g., matching the first two moments does not force the third moment to match; or, two datasets can have the same norm despite being very different (for $n$ points, every point on the $n$-hyper-sphere of the same radius has identical norm). In contrast, the EMD must consider the relation of every point in one set to every point in the other.

I consider using the KS test perfectly reasonable. See also this post. One caveat is that its use of the supremum is a little extreme. For instance, one distribution has a large CDF deviation $\delta$ at some point and is very close the rest of the time vs another that deviates by $\delta-\epsilon$ for some tiny $\epsilon$ many times - the KS statistic will prefer the former. It is up to you whether that makes sense.


As we should not remove any data...we can use vector norm from the origin(l2 norm) given that data_a, data_b, data_c are arrays.

 import numpy as np    
 import pandas as pd
 from numpy.linalg import norm


2.619885493680974 1.5779100101083077 1.6631897065578538.

as l2_a, l2_c values are closer, data_a and data_c are close to each other.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.