Timeline for What is the correct meaning and interpretation of p-values?
Current License: CC BY-SA 4.0
22 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 22 at 21:33 | vote | accept | Robert Long | ||
| Nov 22 at 7:20 | comment | added | Obie 2.0 | @SextusEmpiricus - I mean to disprove in the sense in which I did the other ones, giving an easy, intuitive counterexample. That is a little harder than in the other cases. | |
| Nov 22 at 7:14 | comment | added | Sextus Empiricus | @Obie2.0 it was a response to "I think (6) is by far the trickiest to disprove". There isn't really much tricky to disprove as it is just a confusion of significance and power. | |
| Nov 21 at 19:56 | comment | added | Obie 2.0 | @SextusEmpiricus - Sure. I'm not sure why you are addressing me, though. All I did was give a concrete example of the statement being false. | |
| Nov 21 at 11:10 | comment | added | Sextus Empiricus | @Obie2.0 the 6-th questionnaire statement is confusing type-II error rate with type-I error rate. In the statement "if, hypothetically, the experiment were repeated a great number of times, you would obtain a significant result on x% of occasions." The number x is the power and it is not 100% minus the p-value. | |
| Nov 19 at 16:46 | comment | added | Robert Long | @LarsSeme Also, I would not shy away from including the formal definition too - otherwise I would worry that it could lead to problems understanding the topic down the road. Obviously I don't know what age/level/ability your students are at. I have only ever taught postgraduate students and undergraduate medical students, but almost all of them arrived at the class with many misconceptions. Almost all of them had the wrong understanding. For the PGs, I gave them a quiz, part of which was the Q&A I have used here, on Day 1 with the addition of questions on confidence intervals and interactions. | |
| Nov 19 at 16:37 | comment | added | Robert Long | @LarsSeme Thanks for your comment :) I think that describing a p-value as the proportion of all possible samples under $H_0$ that would lead to a test statistic as or more extreme than observed, can be a useful introduction to the topic, but it's important to clarify that this proportion is not an empirical proportion observed from actual data but rather a theoretical one, derived from the sampling distribution of the test statistic under $H_0$. In other words, it is based on what would happen if we were to repeat the sampling process indefinitely while assuming $H_0$ is true. I hope it helps! | |
| Nov 19 at 15:27 | comment | added | Lars Seme | This is a great question/answer! I have had some success in Intro-level Stats reframing $p$ as--assuming $\mathcal{H}_0$ is true--the proportion of all samples which lead to a test statistic as large (small, whatever). My students seem to often interpret "probability" in this context as some combination of magic and bad choices made by the researcher vs. having "proportion" being somehow more concrete. I know there are potential technical issues here (what do we mean proportion of "all possible samples"...) but find this useful language, especially on first introducing the topic. | |
| Nov 19 at 15:15 | comment | added | Robert Long | @Obie2.0 I have updated the section titled Addressing the Questionnaire Statements to address the very interesting points raised in your comments. I was deliberating about whether to post a new answer, which I might still do in case there are other follow-on questions (though it might be better to ask new question if that is the case), Anyway, I hope it helps :) | |
| Nov 19 at 15:04 | history | edited | Robert Long | CC BY-SA 4.0 |
Updated the section titled "Addressing the Questionnaire Statements" to address the comments by Obie, and added a "TL;DR" at the top.
|
| Nov 18 at 13:12 | comment | added | Stephan Kolassa | Your thread is already winding its way through at least one psychology department... Re my comment, try comparing log-normally distributed populations with parameters $\mu_1=0, \sigma_1=2$ and $\mu_2=1.5, \sigma_2=1$: both have equal means of $\exp(2)$, but my simulations find that $P(p<.05)\approx .21$. And of course there are tons of papers on how susceptible various methods, like ANOVA, are against violations of all these "implicit" assumptions. | |
| Nov 18 at 12:48 | comment | added | Obie 2.0 | Thanks. I don't know what happened to it, but I also pointed out that (3) and (4) have even easier counterexamples in the case of non-exhaustive hypotheses, though that was not the case in the question setup. | |
| Nov 18 at 12:47 | comment | added | Robert Long | @StephanKolassa Nice to see you over here :) And thanks for your thoughtful comment - your point is very interesting and I will give it some thought a bit later today | |
| Nov 18 at 12:45 | comment | added | Robert Long | @Obie2.0 thanks very much for your comments. I will give them some attention later today when I should have some time, | |
| Nov 18 at 12:43 | comment | added | Obie 2.0 | Assuming pre-filtering for basic math skills or whatever. | |
| Nov 18 at 12:42 | comment | added | Obie 2.0 | Incidentally, I think (6) is by far the trickiest to disprove, but there is an easy counterexample. Suppose your experiment involves seeing whether the groups give the right answers on a test that is very easy with the right formulas, and nearly impossible without (questions like "what is a plus one" with and without the value of a), and the "treatment" involves giving them the formulas necessary to solve it (perhaps some questions are impossible to solve even with the formulas, to get whatever t-statistic you want). Then you should expect a significant result every time! | |
| Nov 18 at 12:26 | comment | added | Stephan Kolassa | +1. Micro-nitpick: "the null hypothesis" here not only encompasses equality of means, but much more, which is usually not mentioned explicitly. For instance, to have a $t$ distribution, we also require equality of variances and normality of observations within each group, unless we want to rely on asymptotics. Once you have all this in mind, it's quite strange how $p<.05$ is always taken as evidence against equality of means only, when it could equally be taken as evidence against all the other (unspoken) assumptions. | |
| Nov 18 at 12:22 | comment | added | Obie 2.0 | ("Believe (1) should be "believe (1) is wrong"). For (4), one can go back to the disease example and point out that knowing how often the test gives false positives cannot tell you how likely it is that someone has a condition if you don't know whether one in a billion people have it, or everyone has it, and point out that if you know everyone has a condition beforehand, then the probability of the experimental hypothesis being true is not 99% as one might suspect, but rather 100%. | |
| Nov 18 at 12:18 | comment | added | Obie 2.0 | For (1), I think the standard way is to talk about flipping coins and to point out that no matter how extreme the result, there is a chance that a fair coin flip could generate it. For (3), just point out that if they believe (1), it can't be correct. If you can't disprove the null hypothesis, you cannot prove the experimental hypothesis, since if doing so were possible, it would also disprove the null hypothesis. | |
| Nov 18 at 12:09 | comment | added | Obie 2.0 | A very nice answer. Since it is also intended for non-technical audiences, perhaps it could benefit from examples to illustrate why some of the misinterpretations are wrong? For instance, for (2), the typical example is testing someone for a very rare disease and taking them not having the disease as your null hypothesis. Even if the probability of a positive test result given that they do not have the disease is only 1/100, if only one in a billion people has the disease, it is still much more likely that you got a false positive, so 1% is not the probability of the null hypothesis. | |
| Nov 17 at 13:51 | comment | added | Shawn Hemelstrand | This is an excellent, really well thought out answer (+1). For those interested, I have quite a few references at the end of the CV answer here if you want to dive into the history, misinterpretations, and rationale of $p$ values. | |
| Nov 17 at 13:40 | history | answered | Robert Long | CC BY-SA 4.0 |