0
$\begingroup$

I have trained Logistic regression and decision tree in skearn on the same standardized dataset (binary classification).

Top important coefficients for the decision tree are (sorted by tree.feature_importances_):

enter image description here

And for logistic regression (sorted by abs(logreg.coef_)):

enter image description here

'Total day charge' is the most important coefficient for decision tree, but for logistic regression it is only the 5-th. Why?

'Total day charge' s approximately 35% more important than the next coefficient for decision tree, but for logreg it is almost 50% less important than the most important coefficient.

I can't imagine such optimal separating hyperplane, at least in 3 dimensions. As I can imagine, the biggest coeff in logreg must correspond to dimension, which is the best separating dimension itself.

$\endgroup$

1 Answer 1

0
$\begingroup$

The difference in the importance of the 'Total day charge' coefficient between the decision tree and logistic regression models is due to the way that each model learns from the data. Decision trees and logistic regression are two different machine learning algorithms, and they have different strengths and weaknesses.

In general, decision trees are good at learning complex, non-linear relationships between the features and the target variable. They are non-parametric models that partition the feature space into regions that are separated by decision boundaries, based on the values of the features. They can easily capture interactions between features, and they don't require the data to be linearly separable. This makes them well-suited to handling a wide variety of data, including data with many features and complex relationships between the features and the target variable. In decision trees, feature importance is determined by how much each feature contributes to reducing the uncertainty in the target variable. This is typically measured by the amount of reduction in the Gini impurity or entropy that is achieved by splitting on a particular feature. In your case, it appears that the 'Total day charge' feature was the most effective at reducing uncertainty in the target variable, which is why it is considered the most important feature by the decision tree model.

On the other hand, logistic regression is a linear model, which means that it assumes that the relationship between the features and the target variable is linear. It is a parametric model that estimates the coefficients for each feature in order to maximize the likelihood of the observed data. This makes logistic regression less flexible than decision trees, but it also means that it can be easier to interpret and understand. Logistic regression models are typically simpler and more interpretable than decision trees, which can make them more useful for certain tasks. In logistic regression, feature importance is determined by the magnitude of the coefficients for each feature. In your case, the 'Total day charge' feature has a relatively small coefficient compared to the other features, which is why it is not considered as important by the logistic regression model.

Given this difference in the way that decision trees and logistic regression learn from the data, it's not surprising that the two models would produce different results when applied to the same dataset. It's worth noting that the importance of a particular feature can vary depending on the model that is used, and different models can have different interpretations of feature importance. This doesn't necessarily mean that one model is better than the other - it just reflects the different strengths and weaknesses of each model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.