What is the correct meaning and interpretation of p-values?

Question

I’m posting this question, and an answer, to help dispel a few misunderstandings about what p-values are. As a hiring manager interviewing mid-level and senior data scientists, I have noticed these misunderstandings many times. I have also noticed several posts here on DS.SE where the poster has misinterpreted, or misunderstood, p-values, so rather than point it out every time with details, I thought it better to make a question and answer on the topic. The intention is to hopefully create a "canonical" Q&A that the community can refer to, if/when appropriate. I would welcome any comments in case I have made any mistakes.

What follows is very heavily based on the paper by Haller and Krauss (2002) who investigated misconceptions about null hypothesis significance testing (NHST) among three groups: psychology students, scientific psychologists, and methodology instructors teaching statistics to Psychology students German universities. They used a survey consisting of six statements about the interpretation of a p-value derived from a t-test. Participants were asked to classify each statement as "true" or "false".

---

A questionnaire:

Suppose you have a treatment that you suspect may alter performance on a task. You compare the means of your control and experimental groups (say 20 subjects in each sample). Then, you use an independent means t-test and your result is (t = 2.7, d.f. = 18, p = 0.01). Please choose "true" or "false", and note that several or none of the statements may be correct.

1. You have disproved the null hypothesis (that is, there is no difference between the population means).
2. You have found the probability of the null hypothesis being true.
3. You have proved your experimental hypothesis (that there is a difference between the population means).
4. You can deduce the probability of the experimental hypothesis being true.
5. You know, if you decide to reject the null hypothesis, the probability that you are making the wrong decision.
6. You have a reliable experimental finding in the sense that if, hypothetically, the experiment were repeated a great number of times, you would obtain a significant result on 99% of occasions.

I would encourage everyone reading this to think deeply about these questions and come up with their answers, before reading my answer which follows. I would also encourage people to read the Haller and Krauss (2002) paper on which this is based (there is a link to the pdf in the References section)

References

• Haller, H., & Krauss, S. (2002). Misinterpretations of significance: A problem students share with their teachers. Methods of Psychological Research, 7(1), 1–20. PDF

Answer 1

TL;DR:

Use Bayesian methods when we wish to obtain results that allow us to directly estimate the probability of hypotheses given the data (or most of the other wishful thinking that abounds on the topic).

---

Understanding P-Values and Addressing Misconceptions

Definition of a P-Value

The p-value is defined as:

$$ \mathcal{P}(|\mathcal{T}| \geq |t| \mid \mathcal{H_0}), $$

where:

• $\mathcal{T}$ is the test statistic, a random variable, which follows a known probability distribution under the null hypothesis ($\mathcal{H_0}$). For an independent means t-test, $\mathcal{T}$ follows a t-distribution under $\mathcal{H_0}$.
• $t$ is the observed value of the test statistic, calculated from the sample data.
• $\mathcal{H_0}$ is the null hypothesis.

In the specific scenario described in the question, the research question does not specify a directional hypothesis (eg., whether performance increases or decreases due to treatment). Therefore, a two-sided test is conducted. This means the p-value considers both tails of the distribution and represents the probability of observing a test statistic at least as extreme as the observed value ( $t$ ), in either direction, under $\mathcal{H_0}$.

If the research question had specified a directional hypothesis (eg., whether treatment improves performance), a one-sided test could have been conducted. However, for the given study design, the two-sided test is appropriate.

---

Addressing the Questionnaire Statements

1. False. A p-value cannot "prove" or "disprove" $\mathcal{H_0}$. It quantifies the evidence against $\mathcal{H_0}$ but is inherently probabilistic. For example, consider flipping a fair coin. Even if you observe an extreme outcome, such as ten heads in a row, this result does not "disprove" the fairness of the coin. It only suggests the outcome is unlikely under $\mathcal{H_0}$. Similarly, a low p-value suggests that the observed data is unlikely under $\mathcal{H_0}$, but it does not provide certainty.

2. False. The p-value reflects $\mathcal{P}(|\mathcal{T}| \geq |t| \mid \mathcal{H_0})$, not $\mathcal{P}(\mathcal{H_0} \mid \mathcal{D})$ (the probability of the null hypothesis being true given the data). For example, consider testing for a rare disease where the null hypothesis ($\mathcal{H_0}$) is "no disease." Even if the test has a low false-positive rate (eg., 1%), the disease's rarity (eg., 1 in a billion people) makes it much more likely that a positive test result is a false positive rather than a true positive. This highlights that the p-value does not directly measure the probability of $\mathcal{H_0}$.

3. False. A p-value does not "prove" the alternative hypothesis ($\mathcal{H_1}$). If you cannot disprove $\mathcal{H_0}$ (see (1)), you cannot "prove" $\mathcal{H_1}$. Logically, proving $\mathcal{H_1}$ would imply disproving $\mathcal{H_0}$, which the p-value does not achieve. The two hypotheses are complementary in classical hypothesis testing, but neither is proved or disproved definitively.

4. False. Frequentist methods cannot calculate $\mathcal{P}(\mathcal{H_1} \mid \mathcal{D})$ without incorporating priors. Using the rare disease example from (2), knowing the false-positive rate does not tell you how likely it is that someone who tests positive actually has the disease. This probability depends on the disease’s prevalence in the population. For instance, if the disease is prevalent in 100% of the population, the probability of the experimental hypothesis ($\mathcal{H_1}$) being true is 100%. If it is present in 0%, the probability is 0%. Priors matter.

5. False. The p-value does not directly measure the Type I error rate ($\alpha$) or the probability of making a wrong decision in rejecting $\mathcal{H_0}$. The p-value is a conditional probability: the probability of observing data as extreme or more extreme than what was observed, assuming $\mathcal{H_0}$ is true. It does not reflect the error rate across multiple tests or decisions.

6. False. A p-value of 0.01 does not imply that 99% of replications would yield a significant result. Replicability depends on several factors, including effect size, sample size, and variability. For example, consider an experiment where participants are given tools (eg., formulas) as part of the "treatment" to solve very easy problems. In this setup, every replication might yield a significant result simply because the task structure guarantees success, not because of the p-value. This shows that the p-value alone does not determine replicability.

---

Results from Haller and Krauss (2002)

As already mentioned, Haller and Krauss (2002) surveyed psychology students, practising psychologists, and statistics instructors using these six statements. The results revealed pervasive misconceptions across all groups, as shown by the percentage of participants who endorsed at least one false statement:

• Psychology students (1st year): 100%
• Practising psychologists: 98%
• Statistics instructors: 80%

A detailed breakdown of the specific misconceptions revealed:

1. "You have disproved the null hypothesis":
   • Endorsed by 62% of students, 47% of psychologists, and 42% of instructors.
2. "You have found the probability of the null hypothesis being true":
   • Endorsed by 62% of students, 53% of psychologists, and 47% of instructors.
3. "You have proved your experimental hypothesis":
   • Endorsed by 84% of students, 77% of psychologists, and 67% of instructors.
4. "You can deduce the probability of the experimental hypothesis being true":
   • Endorsed by 68% of students, 53% of psychologists, and 50% of instructors.
5. "You know the probability of making a wrong decision in rejecting the null hypothesis":
   • Endorsed by 53% of students, 47% of psychologists, and 42% of instructors.
6. "You have a reliable experimental finding in the sense that 99% of replications would yield significant results":
   • Endorsed by 74% of students, 65% of psychologists, and 58% of instructors.

These findings demonstrate that even experienced professionals, including those who teach statistics, frequently misinterpret p-values and NHST. The results underline the need for targeted educational interventions to correct persistent myths and clarify the proper interpretation of p-values.

Roots of Misconceptions

1. Textbook Errors: Many textbooks propagate incorrect interpretations of p-values, eg., equating them with $\mathcal{P}(\mathcal{H_0} \mid \mathcal{D})$. For example, the book "Introduction to Statistics for Psychology and Education" by Nunally (1975) amazingly contains all of these mistakes:
   • "The improbability of observed results being due to error."
   • "The probability that an observed difference is real."
   • "If the probability is low, the null hypothesis is improbable."
   • "The statistical confidence... with odds of 95 out of 100 that the observed difference will hold up in investigations."
   • "The degree to which experimental results are taken 'seriously.'"
   • "The danger of accepting a statistical result as real when it is actually due only to error."
   • "The degree of faith that can be placed in the reality of the finding."
   • "The investigator can have 95% confidence that the sample mean actually differs from the population mean."
2. Simplistic Teaching: Overemphasising calculations leads to poor conceptual understanding.
3. Language Ambiguity: Phrases like "reject $\mathcal{H_0}$" may imply certainty where none exists.
4. Hybridisation of Paradigms: Combining Fisher’s and Neyman-Pearson frameworks creates confusion (Gigerenzer, 1993).

---

Expanding the Bayesian Perspective

Using Bayes’ theorem, the posterior probability of $\mathcal{H_0}$ is:

$$ \mathcal{P}(\mathcal{H_0} \mid \mathcal{D}) = \frac{\mathcal{P}(\mathcal{D} \mid \mathcal{H_0}) \mathcal{P}(\mathcal{H_0})}{\mathcal{P}(\mathcal{D})}, $$

where:

• $\mathcal{P}(\mathcal{H_0})$ is the prior probability of $\mathcal{H_0}$,
• $\mathcal{P}(\mathcal{D} \mid \mathcal{H_0})$ is the likelihood of the data under $\mathcal{H_0}$,
• $\mathcal{P}(\mathcal{D})$ is the marginal probability of the data.

Frequentist p-values do not incorporate priors or marginal probabilities, which makes $\mathcal{P}(\mathcal{H_0} \mid \mathcal{D})$ inaccessible without Bayesian tools.

---

Non-Technical Explanation

Explaining to a non-technical audience what a p-value is can be quite challenging. I have had some success with the following:

The p-value is a measure of how surprising your data would be if the null hypothesis ($H_0$) were true, and can be thought of as the probability of obtaining the data that you got, or data more extreme, if $H_0$ is true. Thus, the low p-value is evidence against the null hypothesis.

---

References

• Haller, H., & Krauss, S. (2002). Misinterpretations of significance: A problem students share with their teachers. Methods of Psychological Research, 7(1), 1–20. PDF
• Gigerenzer, G. (1993). The Superego, the Ego, and the Id in Statistical Reasoning. Handbook for Data Analysis in the Behavioral Sciences. Hillsdale, NJ: Erlbaum.
• Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2), 129–133.

Answer 2

TLDR

Confidence/fiducial intervals and p-values are not a measure of evidence in the way that a Bayesian posterior is. However, they are used as such, in the sense that they aid in decision rules about deciding whether a certain observation is sufficient support for a particular hypothesis or not.

They do provide a method to create a criterium for hypothesis testing and interval estimates. Passing the criterium can be seen as proof. Not by the p-value by itself, but by the paradigm where a certain p-value/significance is chosen as a sufficient criterium.

Not the p-value, but the observation of t=2.7 is proof that the null hypothesis is false.

The p-value is like meta-data that describes our decision about the t=2.7 being proof or not.

---

1. You have disproved the null hypothesis (that is, there is no difference between the population means).

That first statement is arguably correct. It is a bit semantical however.

Is a significant result 'disproving the null hypothesis' or is it 'demonstrating that the null hypothesis is false'? A lot of people accept the use of 'disproof'. For example RA Fisher in his 'The Design of Experiments' (section 8 'The Null Hypothesis').

In relation to any experiment we may speak of this hypothesis as the “null hypothesis”, and it should be noted that the null hypothesis is never proved or established, but is possibly disproved, in the course of experimentation.

(emphasis mine)

The bigger debate/misinterpretation is about the difference proving versus disproving (or accepting/rejecting) as in the 3rd statement.

• Why do statisticians say a non-significant result means "you can't reject the null" as opposed to accepting the null hypothesis?

• Why do we need alternative hypothesis?

• Is it possible to accept the alternative hypothesis?

• Why can't we accept the null hypothesis, but we can accept the alternative hypothesis?

---

Obie2.0 makes a good point in the comments by giving an example of a flimsy experiment

Suppose I roll a 20-sided die that I purchased at a game store, set my null hypothesis to be that the die is fair, such that p(20) = 1/20, and my alternative hypothesis to be that the die is weighted such that I am at least a little bit more likely to roll a 20. I set my significance level at 0.05, roll the die, and get a 20 (p = 0.05 under the null hypothesis).

With the single die roll it will probably not change much in the believes/probability about the null hypothesis being true or not

It is true that the p-value is not the same as the probability of the null hypothesis being true (the 2nd statement in the questionnaire).

Instead, it depends on the context what this exact probability is. For example, the previous believes about the hypothesis, the probability of obtaining a certain p-value given different hypotheses.

Considering all that and the differences in different powered tests, a low p-value does mean less plausibility that the null is true. Asking about the statement 'you have disproved the null', some information about the required significance level is left out but typically a p=0.01 result is considered as an acceptable level for disproving a null hypothesis.

And in a way, people also use low p-values as as proof in favour of the alternative hypothesis. A newsarticle headline like "scientists find new proof for ... " is only wrong when you scrutinize it based on an exact technical interpretation.

So you can also argue that 3 is correct.

3. You have proved your experimental hypothesis (that there is a difference between the population means).

Interpreting 1 and 3 depends on the context that is being filled in by the person that makes the questionnaire. They do not imagine the dice roll example and consider instead their own practices.

It isn't 100% certain that the experimental hypothesis is correct, and you can not proof something with absolute certainty. But, in many experimental settings experimenters use a significance level as a criterium for considering proof of an effect. The proof might not be water tight, but it can be an acceptable proof. In time these criteria levels can change and different fields use different criteria: e.g. Origin of "5$\sigma$" threshold for accepting evidence in particle physics?

So if you ask practicing psychologists and statistics instructors such questions, then it is not weird to get some of those people to consider 'passing a certain significance level' as proof of an alternative or disprove of the null.