How to identify if there is a relationship between 5 categorical independent variables to a binary dependent variable?

Question

My dataset has 5 independent variables, each with a value of either Large, Medium or None and a binary dependent variable. The dataset has 67 rows with a split of 17:50.

I would like to identify if there is a relationship between the independent variables and the dependent variable and to explore which variables/set of variables have the largest impact on the result.

Most of the solutions I have checked so far (Nominal Variable Association and Cramér's V) don't support more than one independent variable.

Answer 1

One simple, tricky but probably best approach would be using a Decision Tree. After building a model, you can analyze the structure of the tree to identify the relation. Afterward, you can list the feature importance to see the effect of an independent variable on the dependent variable.

In python, you can use sklearn to achieve it. feature_importances_ will give you Gini's importance of the features and the plot_tree() function to see the structure of the tree. This article might also help to see how to do that.

Answer 2

There are multiple ways you can do this some of which are:-

1.) Use L2 Regularization to determine which out of the 5 features are contributing more in predicting the target.

2.) Use Tree based and gradient boosting based model to calculate the feature importance of each features. Here is an article that uses various models to calculate the feature importance.

3.) Use PCA to get the features with the maximum variance (although I'm not sure about this)

4.) Use mutual_info_classification from the sklearns library to calculate the feature importance.

Other than this you can use filter based, wrapper based methods to get the feature importance of the 5 features.