Why continuous features are more important than categorical features in decision tree models?

Question

I have both categorical and continuous features in my prediction model and want to select (and rank) most important features.

I have converted all categorical variables into dummy variables using one hot encoding (for better interpretation in my logistic regression model).

On one hand, I use LogisticRegression (sklearn) and rank the most significant features by using their coefficients. In this way, I see both categorical and continuous variables among the most important features.

On the other hand, When I want to rank the features by using Decision Tree models (SelectFromModel) they always give higher scores (feature_importances_) first to continuous features and then to categorical (dummy) variables. A completely different behavior in comparison with Logistic Regression.

Whilst the performance of Decision Tree models is much higher (about 15%) than the performance of Logistic Regression, I want to know which sorting of features (Decision Tree or Logistic Regression) is more correct? And why Decision Tree models give more priority to continuous features?

Answer 1

It could be the way that you encode categorical variables.

If you do One Hot Encoding (dummy) each encoded feature will only have two possible values [0,1]. Binary variables normally have less importance in Decision trees given how it is computed the feature weight.

Let's say per example, that you are trying to predict the condition of a patient at the hospital Alive==0, Dead ==1. Imagine that you have a feature is called Head_Shot[0,1], that is really rare, it only appears a few times in the dataset.

The linear model will assign a lot of weight to this coefficient since it is crucial for the target variable. If this happens the rest of the features has no meaning.

For the decision tree, it could do a split in just one of the tree and since it calculates the importance of a feature weighted time the number of times it appears it wouldn´t such a relevant weight.

I am assuming you are doing one-hot encoding. With other techniques, it will be different. And I also assume the way that you calculate the feature importance. So this is far from a scientific answer.

Continuous variables can have more importance in decision trees because each tree can do several splits along its way.

Sorry for the example, it is a bit drastic but I believe it makes the point.

Answer 2

In decision tree models features are ranked based on the number of splits they're involved in. Continuous features are split more often than categorical ones.