I'm writing a data science report, I want to find an exist distribution to fit the sample. I got a good looking result cdf pdf, but when I use KS-test to test the model, I got a low p-value,1.2e-4, definitely I should reject the model.

I mean, whatever what distribution/model you use to fit the sample, you cannot expect to have a perfect result, especially working with huge amount of data. So what does KS-test do in a data science report? Does it means only if we got high p-value in KS-test then the model is correct?

  • 1
    $\begingroup$ this must be a stupid question, in real world, do we always be able to find a fitting curve with high enough p-value? And what if we can't, then we have to disprove our assumption? $\endgroup$ – Carl Feb 12 at 20:27
  • $\begingroup$ In my sole opinion: I do use to measure the discriminatory power of the model, In other words if my model does distinguish between events and non-events(looking at first 4 deciles). In this way KS I do use for model selection aka what model does perform better for my problem/task I try to solve (not only logit models). And I do use on the large amount of data (hundreds of millions of obs) ... on what u do show is to test if your data comes from same distribution and I do not see usage on validating model performance ... if u agree With my approach I will post detailed answer. $\endgroup$ – n1tk Feb 13 at 16:17
  • $\begingroup$ There is no contradiction. Your chosen distribution is a pretty good fit. The low p-value (loosely speaking) says that it is not a perfect fit. If you have a fairly large sample size, then you have the sensitivity to detect even small deviations from the fitted distribution. You might be interested in this discussion over at the statistics Stack: stats.stackexchange.com/q/2492/247274. $\endgroup$ – Dave Apr 14 at 10:13

In your case, the null hypothesis $H_0$ is that your sample follows your the distribution that your model has learned. The alternative hypothesis $H_1$ is that it follows some other distribution. Assuming you have fixed your significance level $\alpha$ to be $0.05$ (the most common choice for $\alpha$, but up to you if you want to go lower), getting a p-value lower than that means you should reject the null hypothesis.

The p-value can be interpreted as the probability of a type I error, in other words a false positive: the probability that you reject the null hypothesis when it is in fact true. In your case, rejecting the hypothesis means stating that there is statistically significant evidence that the distribution your model has learned is not the underlying distribution of the sample. So yes, you would like as large a p-value as possible.

You are using a Kolmogorov-Smirnov test to compare your sample to a reference distribution, in this case, so it's a one-sample KS test. The way I would put it is that getting a high p-value means that: "it is highly unlikely that your model has learned a wrong distribution". In other words, it is highly likely it has learned a pretty good approximation of the underlying distribution. However, nothing is certain when doing statistical hypothesis testing!

I'm not sure what you're showing on your plots though, since there doesn't seem to be an empirical cumulative distribution function on them (lines look smooth).

  • $\begingroup$ Yeah, I think I understand the meaning of p-value. To clarify, I used two Pareto distribution to form an empirical distribution and find the parameters by maximum likelihood, and the blue line in first plot is the sample's CDF, and the red line is the fitting CDF. What confused me is, even if I used log scale when plotting, this empirical distribution seem fit the sample great, but I still got a pretty low p-value. So in this case, I should change my assumption to find other distribution to fit the sample? $\endgroup$ – Carl Feb 12 at 19:40
  • $\begingroup$ The test statistic of the KS test is the Kolmogorov Smirnov statistic, which is the greatest distance between your empirical and predicted distributions. The test statistic of a test is itself assumed to follow a distribution "in the wild", here the Kolmogorov distribution. The p-value is defined (here, for a one-sided right-tail test) as the probability of something more extreme (i.e. distance between the two distributions even greater). One weakness of the KS test is few & far between data points, which leads to large KS statistic which leads to low p-value. How many data points do you have? $\endgroup$ – David Cian Feb 13 at 0:09
  • $\begingroup$ When you're saying "empirical distribution" do you mean you sampled the two Pareto distributions to give a discrete set of samples or just used them as is? $\endgroup$ – David Cian Feb 13 at 0:10

The p-value is interpreted as the probability of a type I error. In other words a false positive: the probability that you reject the null hypothesis when it is in fact true.

You are invoking a Kolmogorov-Smirnov test.

" when I use KS-test to test the model, I got a low p-value,1.2e-4, definitely I should reject the model." Answer - Your low p-value does not indicate that the observed distribution does not fit expected distribution. p value simply indicates the chance for comiting type - 1 error which is quite low in your case. The low value of p i.e. alpha implies that your model predicts very well. The null hypothesis of no difference between two distributions (observed and predicted) is accepted. In nutshell, the test confirms validity of your model.

  • $\begingroup$ -1 The p-value, loosely speaking, is the probability of getting the observations you got if the null hypothesis is true. Thus, the low p-value is evidence against the null hypothesis. $\endgroup$ – Dave Apr 14 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.