# When are p-values deceptive?

What are the data conditions that we should watch out for, where p-values may not be the best way of deciding statistical significance? Are there specific problem types that fall into this category?

• Snarky answer: almost always. There's a huge incentive to create Type 1 errors (i.e., "false alarms") when analysts examine data, so almost all p-values you'll encounter are "too" small. Commented May 14, 2014 at 23:07
• Just throwing this out there, but wouldn't this sort of question best be posed on Cross Validated? Commented May 14, 2014 at 23:47
• @buruzaemon: Maybe. I did a search, this is the closest match: stats.stackexchange.com/questions/67320/… There don't seem to be more than a handful of questions that touch on this. Commented May 15, 2014 at 8:55
• goodness of fit (p value) measure of raw data and measure of difference between two statistics e.g. mean values - statistical significance are two different concepts. Commented Feb 21, 2021 at 23:00

You are asking about Data Dredging, which is what happens when testing a very large number of hypotheses against a data set, or testing hypotheses against a data set that were suggested by the same data.

In particular, check out Multiple hypothesis hazard, and Testing hypotheses suggested by the data.

The solution is to use some kind of correction for False discovery rate or Familywise error rate, such as Scheffé's method or the (very old-school) Bonferroni correction.

In a somewhat less rigorous way, it may help to filter your discoveries by the confidence interval for the odds ratio (OR) for each statistical result. If the 99% confidence interval for the odds ratio is 10-12, then the OR is <= 1 with some extremely small probability, especially if the sample size is also large. If you find something like this, it is probably a strong effect even if it came out of a test of millions of hypotheses.

• While Bonferroni is definitely old-school it is still pretty popular. Related to it is a method called Šidák correction ( en.wikipedia.org/wiki/%C5%A0id%C3%A1k_correction ). I am calling it out, because in a large scale targeting advertising system I worked on we were able to implement this approach as a UDF in Hive. However this only works better when you have independence between tests. If not you have to fall back to Bonferroni or another method. Commented May 19, 2014 at 23:38

You shouldn't consider the p-value out of context.

One rather basic point (as illustrated by xkcd) is that you need to consider how many tests you're actually doing. Obviously, you shouldn't be shocked to see p < 0.05 for one out of 20 tests, even if the null hypothesis is true every time.

A more subtle example of this occurs in high-energy physics, and is known as the look-elsewhere effect. The larger the parameter space you search for a signal that might represent a new particle, the more likely you are to see an apparent signal that's really just due to random fluctuations.

One thing you should be aware of is the sample size you are using. Very large samples, such as economists using census data, will lead to deflated p-values. This paper "Too Big to Fail: Large Samples and the p-Value Problem" covers some of the issues.