2
$\begingroup$

I am very new to the machine learning field and have been practicing logistic regression on few sample data sets. I have built a Model using the logistic regression algorithm. Few of the coefficients have a p-value of more than 0.05 (which is the alpha I am considering).

This is the snippet of my dataset

R-Code for building the model and a summary of the model is given below.

model.bank.1 <- glm(y~., data=bankfull, family ="binomial")

summary(model.bank.1)

enter image description here

Now, before considering AIC, Residual/Null Deviance, confusion matrix and ROC for evaluating. I have observed that the p-values for some of the independent variables is more than 0.05(age has very high p-value, what does this mean?). In such a case, what should be done? Should i straight away remove all such predictors from my model ? Is there any way by which i can make p-value of these predictors less than 0.05 ?

What are the things needed to be checked before going on to evaluate the model using the AIC,Deviance, Confusion Matrix and ROC measures ?

Edit 1: I've tried standardizing the numerical columns, but there is no change in the model at all.

$\endgroup$

1 Answer 1

1
$\begingroup$

The p-value summarises a statistical test for a coefficient not to be statistically different from zero. So basically, when the p-value is > 5%, the estimated coefficient can be positive or negative (the confidence intervall includes positive and negative values).

Often this is interpreted in a way that some variable does not make a reliable contribution to a model. This usually is relevant in causal modelling. But this does not mean that the variable does not contribute to a model and should be excluded. Omitted variable bias can be a massive problem!!! (see: Econometric Theory and Methods, Davidson/Mackinnon, Chapter 3.2 for the OLS case - very interesting stuff). Also in joint significance tests, a non-significant variable can still be relevant.

Especially if you are interested in predictions, the p-value is not so much of a problem. In tendency you can say that overspecification of a model is less harmful than underspecification. In doubt, keep the non-significant variable in the model (it may only be a weak predictor).

However, if you have weak predictors you may also look towards Lasso or Ridge-Regression. In this regression types, variables with little or no contribution to prediction are "shrunken". This is a really cool thing to do and might be the first next thing for you to look at: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html#log

I like Lasso/Ridge, because model selection is (at least to some extent) data driven in this case.

Also have a look at "An Introduction to Statistical Learning with Applications in R" (Chapter 6.2). You may find a copy of the book online. The R code examples in the book are really instructive.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.