I have both categorical and continuous features in my prediction model and want to select (and rank) most important features.
I have converted all categorical variables into dummy variables using one hot encoding (for better interpretation in my logistic regression model).
On one hand, I use LogisticRegression (sklearn) and rank the most significant features by using their coefficients. In this way, I see both categorical and continuous variables among the most important features.
On the other hand, When I want to rank the features by using Decision Tree models (SelectFromModel) they always give higher scores (feature_importances_) first to continuous features and then to categorical (dummy) variables. A completely different behavior in comparison with Logistic Regression.
Whilst the performance of Decision Tree models is much higher (about 15%) than the performance of Logistic Regression, I want to know which sorting of features (Decision Tree or Logistic Regression) is more correct? And why Decision Tree models give more priority to continuous features?