Importance of variable to implement in Logistic Regression model in Python sklearn?

Question

Logistic regression using Python sklearn

In Logistic Regression model we can implement variables which meet 3 conditions:

1. independent variables have to be correlated with dependent variable
2. independent variables have NOT to be correlated with each other
3. only relevant variables can be added to the model, otherwise it does not make sense

and both 1. and 2. point we can achieve using Pearson correlation for example:

-- correlation between independent variables and dependent variable (targetVariable

1. CORREL = data.corr().sort_values('TargetVariable') 

--heatmap which shows lvl of Pearson correlation between independent variables (the closer 0 the lower the correlation)

2. sns.heatmap (data.corr(), cmap="coolwarm", annot=true) 

But when I choose appropriate variables after 1. and 2. step how can I meet the 3. step and choose which variables (which are correlated with independent variable and not correlated with other independent variable) are important and I should use it in model and which are not important and I should out them? Maybe p-value and delete all with p-value higher than 0,05? But how?

Please give me the example code to meet 3. step.

Answer 1

The problem you are describing is feature selection. Welcome to a particularly sticky problem. Your method of Pearson correlation certainly can help - simply drop those features with low and or insignificant correlation to your dependant variable. This is univariate selection. Since you are using a logistic regression, you can also inspect the model coefficients and odds ratios to determine the effect size and significance of each feature.

But what if some combination of your independent variables produces a significant effect within your model? Say the combination of: gender, age, income, and house value for a binary response related to likelihood to purchase. This is multivariate feature selection.

For logistic regression, and other models, you can use "interaction terms" to investigate the combined effect of input variables. In Data Science and Machine Learning literature this would fall under the practice of feature engineering. There are a number of other procedural and rule of thumb methods to select inputs for multivariate models. For instance, one can use a decision tree method to evaluate important features across repeat random samples of the data, or one can simply use domain and conceptual knowledge of the process being modeled. In most cases you should use a variety of techniques and choose the inputs that make both mathematical and theoretical sense.