2
$\begingroup$

I have a large data set of property values at two points in time. I have calculated the change in value between those two points. I want to assess which distribution most closely matches the variation in change in property values. What are some possible ways of selecting the closest type of distribution?

I plan to use python, scipy and pandas to do this, although my question is more about what are the correct statistical tests to do to determine whether one distribution is a better fit to the data than another.

$\endgroup$
2
$\begingroup$

My understanding of your problem: you have realizations of $X$ (property value at first point in time) and $Y$ (value at second point in time). Want to fit a distribution for $Y-X$--more generally, a "model for the data."

One way of doing this would be plot a histogram of these values and then come up with plausible distributions that might fit the data, and then use maximum likelihood estimation to find the optimal parameters. If the models are nested, then you can use, e.g., the likelihood ratio test to determine which is a better fit to the data.

If not, then things are trickier. One good option is to hold out some of the data for testing, fit the models on the training data, predict the test data using each model, and see which one performs better.

$\endgroup$
1
$\begingroup$

Cool question. My understanding is that the Kolmogorov Smirnov test is what you're after here. You could use the distribution functions in scipy to generate various kinds of distributions and use the K-S test to assess the similarity between your distribution of value variances and each of the distributions you select for a test.

K-S test documentation here: https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.kstest.html

Here's an example:

import scipy.stats as stats
import numpy as np

target = stats.norm.rvs(0, 1, size=1000) # The target has standard normal distribution

a = stats.kstest(target, stats.norm(0, 1).cdf) # Testing target against a standard normal distribution)
b = stats.kstest(target, stats.chi2(1).cdf) # Testing target against a chi2 distribution

print("""
Target x Normal Distribution: test-stat {0}, p-value {1}
Target x Chi2 Distribution: test-stat {2}, p-value {3}
""".format(a[0], a[1], b[0], b[1]))

And the output is:

Target x Normal Distribution: test-stat 0.022869389923164785, p-value 0.6723583002262676
Target x Chi2 Distribution: test-stat 0.488, p-value 5.343066131071921e-220

The null hypothesis is that the two distributions are identical, so you can see we can confidently reject that hypothesis when testing against the Chi2 distribution but not against the normal distribution.

One potential problem I can see with this approach is that changing the mean and standard deviation parameters of the target or b does result in the test statistic changing to the point that the KS test no longer considers them the same:

stats.kstest(target, stats.norm(10, 1).cdf))

KstestResult(statistic=0.9999999999962224, pvalue=0.0)

Which is fair in a way, given they no longer overlap in the slightest:

enter image description here

What should happen here depends on whether you want those two distributions to be considered similar or not I guess. If so you'd have to perform some transformation before the test, or use the mean and standard deviation of your target distribution as parameters for stats.norm(). For example we might scale both distributions by removing the mean and dividing by standard deviation, and that way the KS test sees their similarity again.

test = stats.norm.rvs(10, 1, size=1000)

sns.distplot((target - target.mean() / target.std()))
sns.distplot((test - test.mean()) / test.std())

print(stats.ks_2samp((target - target.mean() / target.std()), (test2 - test2.mean()) / test2.std()))

enter image description here

Ks_2sampResult(statistic=0.03200000000000003, pvalue=0.6785103823828891)

$\endgroup$
5
  • $\begingroup$ This doesn't answer what was asked. K-S tests whether two distributions are the same, not whether one distribution is a better fit to the data than another. $\endgroup$ Aug 22 '19 at 19:58
  • $\begingroup$ @SheridanGrant I'm not sure I understand the distinction. The best fit to the data is the distribution from which the data is drawn. The K-S tests allows you to determine which distribution that is. $\endgroup$
    – Dan Scally
    Aug 22 '19 at 21:09
  • $\begingroup$ I see now what you're going for, but it isn't the right approach. We don't know the distribution of the data--that's why we're trying to find the best model for it. K-S would require some reference distribution to test against, which could be the empirical distribution or a KDE distribution. But if TaxpayersMoney has many data points, then basically any model that is difference from this reference will show up as difference. How do you decide what model is best? The largest p-value? Too ad-hoc--this situation is what the LRT or other model comparison techniques were designed for. $\endgroup$ Aug 24 '19 at 0:55
  • $\begingroup$ Also, you'll never know the exact distribution the data came from, or be able to determine it with a statistical test. Statistical tests can only tell you if you should be reasonably sure your hypothesis is wrong, not guarantee for you that it's right. $\endgroup$ Aug 24 '19 at 1:00
  • $\begingroup$ @SheridanGrant I missed this response previously (or more likely forgot to reply to it), sorry. Yeah I agree it's a little ad-hoc of an approach, but in the absence of knowledge about LRT (which I'd not heard of previously) it seemed like a reasonable way forward. Thanks for the links; interesting lunch time reading. $\endgroup$
    – Dan Scally
    Sep 2 '19 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.