0
$\begingroup$

I am currently going through Hastie and Tibshirani's 'Introduction to Statistical Learning' textbook and I have come across something I don't understand on page 77. I have two questions.

The author states that if we had 100 variable predictors (with respective coefficients $\beta_i$) and supposed that the null hypothesis, $$H_0 : \beta_1 = ... = \beta_{100} = 0$$ were true, then roughly $5\%$ of the $p$-values would fall below $0.05$ by chance, and therefore we might wrongly conclude that certain predictors are related to the response. Why would this happen by chance? Is this simply a mathematical truth?

In addition, the author then goes on to state that the $F$-statistic is a better measure because "if $H_0$ were true, then there is only a $5\%$ chance that the $F$-statistic would result in a $p$-value below $0.05$". I don't understand the difference - could somebody explain a bit more clearly?

$\endgroup$
1
$\begingroup$

The example given in your textbook proposes a multiple linear regression with 100 predictors, all of which have a "true" regression coefficient of 0. In other words, your independent variables have no statistical association with your dependent variable.

When you calculate the $p$-value of an individual coefficient, you're looking at the magnitude of the coefficient, the standard error of the coefficient, making some distributional assumptions about it, and asking the following question: "what is the probability of seeing a coefficient value this extreme if the true value is actually zero?"

If our distributional assumptions are correct, any given coefficient with a true value of 0 will report a <0.05 $p$-value approximately 5% of the time. That's not so much of a problem if we only have one predictor, but by the law of large numbers, if we have lots of predictors, we'd expect 5% of them to report a $p$-value this low. This makes the $p$-values in high-dimensional regressions hard to interpret.

The $F$-test is different. Instead of evaluating every single coefficient for statistical significance, it applies a single test to the entire regression. So instead of having 100 chances to throw up an erroneous $p$-value, it only has one chance. This makes the $F$-test useful for evaluating whether or not there is a regression effect for high-dimensional regressions.

$\endgroup$
  • $\begingroup$ Thank you for your help. So, just to clarify one other thing, supposing we have $n$ predictors the $p$-value, say for $\beta_1$, is calculated via the hypothesis: $H_0: \beta_1 = 0, \ \ H_a: \beta_1 \neq 0$, correct? i.e. $H_0$ would be the case in which only $\beta_1$ is zero and the rest are non-zero? $\endgroup$ – Johnny Breen Mar 23 '17 at 14:02
  • 1
    $\begingroup$ When you calculate the $p$-value for a single predictor, you are calculating it independently of all the other predictors. So your null hypothesis in this case (call it $H_0^{\beta_1}$) would simply be the case in which $\beta_1 = 0$. It doesn't tell you anything about your other predictors. $\endgroup$ – R Hill Mar 23 '17 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.