Model selection by Interpreting p-value of anova function

Question

I am trying to interpret the p-values for model selection. Here is a sample code taken from a book (An Intro. to stat. Learning, page 290, by Gareth James et al.)

 Null-hypothsis: Model M1 is sufficient to explain the data against the alternative hypothesis that a more complex model M2is required.

With this statement, I think p-value should be high so that null-hypotheis can be rejected and a model M2 is selected. Keeping this in mind, I think we should select Model 5 as it is having highest p-value, but authors mention either of Model 3 or Model 4 is a good fit. I don't understand why Model 5 is not a good fit.

Answer 1

To understand why, we should try to understand what we're doing here.

Let's start with the first p-value: 2e-16, what does that mean? This is the p-value under the null hypothesis that the linear model and the second polynomial model are statistically identical. We say that they are the same if the extra coefficient in the polynomial is statistically zero. This is exactly what the p-value is telling you. It reads like this: There is 2e-16 probability that the null hypothesis of the linear model and the polynomial are identical. This is a very small probability, so you can conclude that they are not identical. This means, the second-order polynomial is a better model than the linear model with a significance level of 5%.

(PS: a more correct statistical interpretation of the p-value relates to false-positive rejection, but let's not go into that deep)

Now, using the same logic, you can conclude the third-order polynomial fits better than the second-order polynomial.

Next, the p-value comparing the third and fourth order polynomial is about 0.05. You can reject it or you don't want to reject it, this is up to you. But If was you, I'd simply fail to reject it because it's larger than 0.05.

Finally, the final p-value is about 0.37 which is too high. This means although the fifth order polynomial fits better than the fourth order, the loss in your RSS is insufficient to justify the loss of the degree of freedom. Therefore, we say the fifth order polynomial is statistically no better than the fourth-order.

Conclusion: It's "bad" to have a large p-value because you really want to reject a null hypothesis. Statistically, we do it to control the false positive rate.

PS: In your example, R uses the F-test to compare the two models.