Feature importance and deriving rules using tree based classification models

Question

I have a dataset where I have categorical and continuous values with targets 0/1 (binary classification task). Since I need to find patterns and relationships in the occurrence of the event or target, I think I should use Decision trees. However the issue is that I have 2 categorical variables which have 700 & 150 categories respectively with the remaining variables being numerical/continuous.

My questions are as follows:

1. Can I estimate feature importance using random forests in sklearn for this set of variables? If yes, do I need to change the type of variables?

2. Can I use the variables as they are, i.e categorical with so many categories each & continuous variables, when training a decision tree using sklearn in order to be able to visualize rules from the tree?

Please feel free to suggest any other approach too. The prime objective of the task is to identify the drivers/features which impact the target & generate a pattern - if at all can be identified from the data.

Also, given these many categories for the 2 variables, how do I conduct a statistical analysis for this variables alone with the target to understand if any correlation exists or identify cause effect relationships.

Answer 1

You can use categorical features with decision trees in scikit-learn, but you'll need to encode them as numbers. If your categorical features are ordinal (such as ranking ‘bad’, ‘fair’, ‘good’), they are easy to encode in numbers that respect the underlying ordering (e.g. 0, 1, 2). For nominal features, given the high cardinality you mention, you can try some version of the target encoding or the frequency encoding. With the target encoding, make sure to use some form of regularization in order to avoid overfitting.

After the features are encoded, you can build, for example, a random forest and obtain the impact of each feature with attribute feature_importances_.