3
$\begingroup$

In backward elimination, I heard the steps of fitting the model by keep removing the highest p-value(a.k.a. insignificant independent variable) each time like below

  1. Select a significance level to stay in the model(e.g. SL = 0.05)
  2. Fit the full model with all possible predictors
  3. Consider the predictor with the highest P-Value(P > SL)
  4. Remove the predictor
  5. Fit model without this variable (Repeat step 3-5 until P <= SL)

But the part which I don't get is why is having higher p-value makes the corresponding independent variable insignificant. Doesn't having high p-value mean it's more close to the null hypothesis so that that variable is more significant?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Actually, the Null Hypothesis is that the predictor is not significant. Taken from the book Introduction to Statistical Learning:

  • Null Hypothesis: There is no relationship between X and Y
  • Alternative Hypothesis: There is some relationship between X and Y

If we have a high p-value, we have an expressive result showing that the null hypothesis is correct and therefore the estimated coefficient come from a normal distribution around zero and can be discarded. I hope this answer your question, please let any comment if you need more help.

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ Also, as an extra comment: I think you should also calculate other metrics as well and not only p-values, such as MSE/MAE, R_squared, BIC, etc. This is because that when variables are interacting some can have high/low p-values due chance, if you have a significant amount of predictors. I highly recommend that you read chapters 3 and 6 from ISLR book, the authors offer a free pdf version on their website: www-bcf.usc.edu/~gareth/ISL $\endgroup$
    – Victor Oliveira
    Commented Mar 5, 2019 at 22:24
  • 1
    $\begingroup$ In this pizza example we have a Null Hypothesis that states the average delivery time is 30 minutes or less, that is, you will sample some deliveries, calculate the mean and test your results to see if this calculated_mean comes from a distribution which the mean is 30 minutes. Then you get a result of 0.001, strongly indicating that we have a low probability of seeing a value calculated_mean coming from a distribution where the original mean is 30 minutes and we then reject the null hypothesis. $\endgroup$
    – Victor Oliveira
    Commented Mar 6, 2019 at 12:09
  • 1
    $\begingroup$ The part you are getting confused is here: - "Doesn't having high p-value mean it's more close to the null hypothesis so that that variable is more significant?" The Null Hypothesis on regression case is that the variable is not significant, that is, zero. And not that the variable is significant. Can you see this? Please, if you have any question don't hesitate to keep commenting, let's clarify that. :) $\endgroup$
    – Victor Oliveira
    Commented Mar 6, 2019 at 12:11
  • 1
    $\begingroup$ No, in regression your Null Hypothesis is that the coefficients are zero, not the mean. Null hypothesis is not something fixed, you can define anything to test against. In the pizza case, it could be other way around, that is, the Null Hypothesis could be that the average delivery time is more than 30 minutes. Don't mix both tests, in the regression case you are testing to check if your coefficient is zero, then a high p-value indicates that this is true. In the pizza case you want to check if the average is 30 minutes, then a high p-value indicates that is true (But of course, we had a low p) $\endgroup$
    – Victor Oliveira
    Commented Mar 7, 2019 at 11:28
  • 1
    $\begingroup$ Oh, thank you very much! Thanks to you, I think I got to grasp the concept :D $\endgroup$
    – Seungho Lee
    Commented Mar 18, 2019 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.