Logistic Regression - Coefficients have p-value more than alpha(0.05)

Question

I am very new to the machine learning field and have been practicing logistic regression on few sample data sets. I have built a Model using the logistic regression algorithm. Few of the coefficients have a p-value of more than 0.05 (which is the alpha I am considering).

R-Code for building the model and a summary of the model is given below.

model.bank.1 <- glm(y~., data=bankfull, family ="binomial")

summary(model.bank.1)

Now, before considering AIC, Residual/Null Deviance, confusion matrix and ROC for evaluating. I have observed that the p-values for some of the independent variables is more than 0.05(age has very high p-value, what does this mean?). In such a case, what should be done? Should i straight away remove all such predictors from my model ? Is there any way by which i can make p-value of these predictors less than 0.05 ?

What are the things needed to be checked before going on to evaluate the model using the AIC,Deviance, Confusion Matrix and ROC measures ?

Edit 1: I've tried standardizing the numerical columns, but there is no change in the model at all.

Answer 1

The p-value summarises a statistical test for a coefficient not to be statistically different from zero. So basically, when the p-value is > 5%, the estimated coefficient can be positive or negative (the confidence intervall includes positive and negative values).

Often this is interpreted in a way that some variable does not make a reliable contribution to a model. This usually is relevant in causal modelling. But this does not mean that the variable does not contribute to a model and should be excluded. Omitted variable bias can be a massive problem!!! (see: Econometric Theory and Methods, Davidson/Mackinnon, Chapter 3.2 for the OLS case - very interesting stuff). Also in joint significance tests, a non-significant variable can still be relevant.

Especially if you are interested in predictions, the p-value is not so much of a problem. In tendency you can say that overspecification of a model is less harmful than underspecification. In doubt, keep the non-significant variable in the model (it may only be a weak predictor).

However, if you have weak predictors you may also look towards Lasso or Ridge-Regression. In this regression types, variables with little or no contribution to prediction are "shrunken". This is a really cool thing to do and might be the first next thing for you to look at: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html#log

I like Lasso/Ridge, because model selection is (at least to some extent) data driven in this case.

Also have a look at "An Introduction to Statistical Learning with Applications in R" (Chapter 6.2). You may find a copy of the book online. The R code examples in the book are really instructive.