Performance difference between decision trees and logistic regression when one of the features is a string

Question

I have a set of features, one of which is a string. I convert the string to an integer by treating the string as a base 36 number (I only use the first 13 characters). Then I can use DecisionTrees since in the sklearn implementation you need to convert it to a number. When I tried a different model, say Logistic Regression, performance drops drastically, say from 80% to 30% accuracy.

I might have accepted this result if I had been able to use the strings as such in the DecisionTrees model, but since I used the same string to integer conversion for both models, why such great difference?

I cannot go into the details, but let me provide you with an analogy. Let's say you are classifying millions of objects by their usefulness. So you say hammers are 4, screwdrivers 6, washers 10, etc. Of course you have more than one screwdriver, and sometime you forget and give it a value of 5, or something else. The model goes through millions of example, and then makes a prediction about the number for each object. I converted the names into integers, as I explained, and decision trees gives me an 80% accuracy, linear regression 30%. I assume that the problem is that linear regression tries to figure out some mathematical rule that does not exist. But why is decision trees immune from this problem?

Answer 1

String data can be either categorical (where you have e.g. more than 10 examples of each string) or free text. If it's the former, a decision tree can deal with it no problem. You don't have to convert it into a numeric.

For regression you cannot directly use categorical variables. If you want to use them in a regression, you will need to create dummy variables to encode the values. e.g. if your categories are "Red", "Yellow", "Blue" for the colour variable, you create variables "Red" (which will take a 1 or a 0) and "Yellow" (which takes a 1 or 0). If both are 0, the colour must be "Blue". There are functions in sklearn to do this automatically.

If your string is just free text then you will need a better way of grabbing information out of it. You can use text mining such as tokenizing, TF-IDF etc. to convert it into numerical and categorical information that can be fed into a classifier.

Answer 2

Decision trees can classify categorical data. Even if they treat every string as a separate (non comparable to the others) category, they are still able to detect when two strings are equal.

This is not the case with statistical methods, such as logistic regression. These need I'm interval data. This is why you need to define a string similarity metric over your strings. Unless there's a reason for defining your metric this way (notice that differences in the first character of otherwise similar strings get evaluated as much more distant than differences in the last character), probably the algorithm is confused by the values of your strings. Thus, it is possible that your regression learns something that you've introduced with your string conversion, and which does not exist in the original data. This introduced dependency could be confusing your regression and could be overshadowing a more efficient way of learning.

To validate such a hypothesis, you can try applying other string similarity metrics and compare the results. However, be cautious: different metrics might be useful in different contexts.