Interpreting lasso logistic regression feature coefficients in multiclass problem

Question

I have a dataset with a large number of text features, where the target variable has three classes.

I have encoded the features using tf-idf. This has resulted in a dataset with an extremely large number of features.

I have tried performing classification using lasso logistic regression (in sklearn, LogisticRegression(penalty='l1')), which automatically sets some features' coefficients to 0 and thus performs some kind of feature selection for me.

The sklearn implementation of LogisticRegression() performs multiclass classification using 'one-versus-rest'. That is to say, it trains three separate classifiers which predict one particular class against the other two, and then assigns each data point to the class whose predictor gave the highest probability to.

The coefficients of some of the features from my dataset were set to $0$ in all three classifiers. Others, however, are set to non-zero numbers in all three.

Is it correct to think of those with coefficients set to $0$ by all three classifiers as uninformative, in the sense that they add no new information given the presence of all the others?

Does it make sense to compare the magnitude of the non-zero coefficients, across the three classifiers, in order to find out which features are most informative?

Answer 1

Welcome to the site!

I think you are right that feature weights = 0 across all three classifiers mean that the features are not predictive of the outcome.

In terms of comparing magnitudes, I'm not sure the absolute values are really comparable - each model is trying to predict a different target.

However, you could rank the coefficients in some way and then compare the rank of the coefficient across the three models. You would need to choose a robust way to rank the raw coefficients since they would range from positive to negative - e.g. a strongly negative coefficient such as -2 is probably more predictive than a barely positive coefficient of 0.1.