1
$\begingroup$

I am using the following LogisticRegression model using sklearn. The task requires to select one label from multi-labels, so if I provide a, b the output could be d etc:

from sklearn.linear_model import LogisticRegression
multi_class_prediction_model = LogisticRegression(solver='newton-cg', multi_class='multinomial')

x_train=[['a','b'],[...]]
y_train=[['d'],[...]]
x_train.applymap(encodings)
y_train.applymap(encodings)

encodings={'a':1,'b':2,'c':3,'d':4...}
multi_class_prediction_model.fit(x_train, y_train)

I encoded the labels manually to numeric integers as above. However, the prediction seems to be sensitive to numeric values of encodings! So if I change it following:

encodings={'a':100,'b':200,'c':300,'d':400...}

Then the prediction changes drastically.

Is this methodology correct?

$\endgroup$

1 Answer 1

2
$\begingroup$

Most ML methods assume that you properly preprocessed the data, which includes mean removal and scaling. It is so, because the underlying optimization algorthms are indeed sensitive to scaling.

So you should preprocess your data, which would undo the scaling operation you applied. You can find more about standard preprocessing techniques in the sklearn docs: https://scikit-learn.org/stable/modules/preprocessing.html

EDIT:

As you commented you are dealing with categorical features, which are not easy in general for ML algorithms. The problem is that there might be no meaningful measure among the categories, while after encoding there is, and in fact ML algorithms try to use this measure to come up with predictions.

One way to remove this measure is to use one-hot encoding (see sklearn.preprocessing.OneHotEncoder), which separates the features into binary feature vectors of size as the number of categories. But that rapidly becomes intractable as the number of categories and the number of categorical features grow. Beyond this, your results will depend on the encoding you use pretty much for any ML algorithm. When you do the encoding, you should try to assign closer numbers to the categories which are in some sense more similar if possible.

Although the results will vary for different encodings for all ML algorithms I know, that does not mean all ML algorithms are sensitive the same way to categorical variables. Linear models (like logistic regression) perform poorly, so you should try decision trees (especially random forests, gradient boosting) which will mostly use the encoding implied measure for node splitting and are able to separate the similarly encoded categorical variables which influence the prediction differently. This is what a linear model cannot do as there a weight value simply multiplies the encoding value, so closer encoding values will produce closer prediction values too.

One final note, there is no point of uniformly scaling the whole encoding as you will only introduce scaling problems as I mentioned above, and make the underlying optimization perform worse, at least for logistic regression. In fact, decision trees would be unaffected by such scaling. In both of your example, the encodings provide the same order among the categories, and that is all that matters. Of course if it make sense to say that 'b' is more similar to 'a' than to 'c', then it might make sense to reflect that in the numbers.

$\endgroup$
2
  • $\begingroup$ But I have categorial variables, so there is no notion of scaling. I have utilized various encodings and all of them yield different results. So does that mean multi-label logistic regression cant be done as its sensitive to choice of encodings? $\endgroup$ Feb 2 at 12:52
  • 1
    $\begingroup$ Ops, I overlooked that. Short answer is yes, logistic regression will perform poorly on categorical variables, but let me expand on this a bit by editing the answer... $\endgroup$
    – gabalz
    Feb 2 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.