5
$\begingroup$

I fitted a logistic regression model on a data set and got an AUC score of .70. I added some additional out-hot encoded categorical features to the model and the AUC improved slightly to .74.

How do I assess how the model improved? What plots/other analyses are used to to assess the performance gain?

I understand that the model improved, but I want to be able to explain why adding those features improved the model.

This is just a general data science question.

Thanks!

$\endgroup$
1
  • $\begingroup$ I would go back a step and inquire whether the AUC is a practical assessment for the problem that you are trying to solve. The AUC is a way of observing the true positive and false positive rates at all possible thresholds for separating the two classes. However, in many business areas it would be impractical, expensive or even illegal to allow certain levels of false positive behavior. If you look at the AUC graph, in what portion of the graph was your improvement observed? $\endgroup$ – Gabriel Alon Nov 17 '20 at 19:52
1
$\begingroup$

First, it makes sense that adding features will improve your performance, just make sure you do evaluation carefully and not overuse the same validation dataset (and if yes try to re evaluate it on an unseen independent different test set) to ensure you are not overfitted.

After that, you can use Shapley values in its aggregate look to see which features (or specific values in your encoding) impacted the model decision making

$\endgroup$
0
$\begingroup$

I don't think there is one way to do this. What I do is I analyse the correlation of the first model residuals with new features before adding them or train a model on the residuals as a target and the new features as an input. If you want to understand your features importance or impact you can use dependency-plots or use the shap value library to see the magnitude of each feature in your algorithm.

$\endgroup$
3
  • $\begingroup$ By dependency-plots, do you mean partial dependency plots? How exactly do they work for a Logistic Regression. I tried plotting these in R but they are just straight lines which I dont know how to interpret $\endgroup$ – Eisen Jul 10 '20 at 12:56
  • $\begingroup$ Also I'm looking through SHAP packages for R, is this only for XGBOOST models? $\endgroup$ – Eisen Jul 10 '20 at 12:57
  • $\begingroup$ for partial dependency plot and SHAP package, it is used for non linear machine learning algorithms which is not the case for logistic regression. Correlation with the residuals is a great way to assess feature importance and in a linear model. $\endgroup$ – mirimo Jul 10 '20 at 13:01
0
$\begingroup$

Often adding features will improve a model's performance because it increases the model's ability to predict the target.

It is common to ask if the relative to increase in performance metric is worth the increase in model complexity. AIC and BIC are Information criteria methods used to assess model fit while penalizing the number of parameters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.