I am working on an Insurance domain use case to predict if an existing customer will buy a second insurance policy or not. I have a few personal details saved under different categories like Marital status, Smoker (Yes or No), Age (Young, Adult, Senior Citizen), Gender (Male/Female) and few are continuous variables like Premium Paid, Sum Insured.

My target is to use this mix set of categorical and continuous variables and predict the class ( 1 - Will buy a second policy, 0 - Will not buy a second policy). So how can I find/compute the correlation in this dataset and pick only the significant ones to use in Logistic Regression formula for classification?

Will appreciate if someone can provide articles, link to a similar piece of work done in Python.

  • $\begingroup$ Thanks for the response, I am looking for a similar use case in Python, the questions highlighted have examples from R. Can you point me to some implementations in Python. $\endgroup$
    – tanmay
    Nov 15 '19 at 12:33

Regarding your question about Python implementations of the given R examples: SKlearn has ready to use implementations for feature selection as they were described under the linked question in R (see here).

Here is an example for categorical input and output data: With SelectKBest you can select the K features with the highest corelation, e.g. based on a chi squared test.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

# load the famous iris data set for which X.shape is (150, 4) and y.shape (150,)
iris = load_iris()
X, y = iris.data, iris.target

# Add exponentially distributed noise (20 new attributes)
rng = np.random.RandomState()
noise = rng.exponential(size=(len(iris.data), 20))
# While X.shape is (150, 4), X_noisy has shape (150, 24)
X_noisy = np.hstack([iris.data, noise])

# Select 4 features based on chi squared test
selector = SelectKBest(chi2, k=4)
selector.fit(X_noisy, y)
X_selected = selector.transform(X_noisy)

Checking the shapes gives the following:

Out[113]: (150, 4)

Out[114]: (150, 24)

Out[115]: (150, 4)

You can also check which features were selected:

[False False  True  True False False False False False  True False False
 False False False False False False False False False False  True False]

As you can see 2 of the initial 4 features (which were not noise) did get selected. The feature selectors also have attributes to check for examples the p values (see here for SelectKBest).

The book 'Introduction to machine learning with Python' by Mueller and Guido has a section about it too (their example is very close the one above).

However, the example of a CHI squared test I gave is applicable to categorical independent and dependent variables. For a mix of categorical and continuous independent variables you might need to discretize the continuous variables or check other methods for features selection, e.g. model-based.

  • $\begingroup$ Thanks, @Sammy, this was helpful, I will read the book. Just one confirmation. In my dataset, the target variable is categorical (value of 0 and 1) and independent variables have a mix of categorical (value of 0 and 1) and continuous variables. Statistically speaking, is it fine to run SelectKBest function for all the independent variables and are the results acceptable ? $\endgroup$
    – tanmay
    Nov 18 '19 at 4:20
  • $\begingroup$ Good that you are asking since my answer was targeted to provide you Python code and libraries for the discussed examples with R in the linked question "How to get correlation between two categorical variable and a categorical variable and continuous variable?". However, the CHI squared test that I applied in my example is for categorical data (regarding the dependent and independent variables). But for mixed independent variables you might need to switch from univariate to other methods like model-based feature selection! $\endgroup$
    – Sammy
    Nov 18 '19 at 12:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.