# How regression algorithm works on categorical features

I have a dataset with features that most of them are nominal categorical features, I have converted my model to indicator values,

Original

F1,F2,L
1 ,1 ,50
2 ,3 ,30


After indicator values

F1-1,F1-2,F2-1,F2-3,L
1   ,0   ,1   ,0   ,50
0   ,1   ,0   ,1   ,30


I used different regression algorithm (Poisson, Bayesian, Decision Tree reg, Decision Forest reg, Boosted decision tree reg, linear regression, neural network), but all of then have low performance (r2 ~ 20-30)

Then I was thinking how regression can find values, then I found something interesting : relation of data with label They are like below picture

But in most of the books and examples and samples suitable data for regression are like below

And this is the point I got confused!

So my question is how regression (or which algorithms) are suitable for predicting values in high categorical data

TL;DR: It is a convention to convert categorical features to numeric features before you can use them in regression, or any other machine learning algorithm for that matter. Technically, there is nothing that is stopping ML algorithms from working with categorical features but their software implementation would be prohibitively expensive, hence this convention in ML practice.

Turning to your problem, if you are trying to convert categorical label values into numerical values, there are various methods but it appears the best one for your data would be One-Hot encoder. Both Scikit-Learn and PySpark, and most other libraries provide handy functions for it, as it is very common operation. For example:

from sklearn.preprocessing import OneHotEncoder
one_hot = OneHotEncoder()
one_hot.fit(...your_columns...)


Once you have all data in numeric form, you can use pretty much any algorithm.

As for the last figure, it is scatter plot between x and y, and it is not obvious what x and y are. Weirdly enough, it is not one-on-one relationship because a given value of x seems to produce multiple value of y?!!! And I am not sure what you are plotting in earlier figures that is giving you straight lines!

• thanks for answer, yes I already used one-hot encoder, (see my second dataset, in azure it's called indicator values) then if I compare the label with one of the features, since all have 0 or 1 value there will be a graph which has many 0s and many 1s, and it look like my second image and it look likes the straight lines , (or may be I should have switched x and y axis, btw in that situation still there was some straight lines) this is why I got confused, because it seems I can't find a line to estimate values based on that
– Reza
Dec 31, 2017 at 16:37

The relation that your graph is showing for you categorical variable versus the numerical label does not look to be very strong as for every value of SGPriorityValue, you have all sorts of values of y it seems. For a categorical variable the relationship might not look as smooth and continuous (Like the last graph you have) but there has to be pattern for any model to fit.

usually if a categorical variable is related to a numerical value, you can expect a graph like below.

• Thanks for answer, SGPriority has 3 values 'Critical,Important,Trivial,' so I convert it to 3 numbers 7,5,3 and the I used normalization which generates these 3 new numbers, the question is either with normalization or without or just one-hot encoding categories if I draw a scatter plotter between the feature and lablel it will be something like that, because there will be always 2 (or multiple) concrete values, I don't know hot could it get to something that you drew
– Reza
Dec 31, 2017 at 16:41
• It won't, hot encode is just a representation of the data. So if your data itself does not have linear pattern with the value, no encoding will show a pattern, hence a bad R-square value. Jan 1, 2018 at 14:59
• I don't get it, consider there is only one categorical feature with values A,B, if you do one-hot encoding the values for column-A will be 0 and 1 and same for column-B, now if you do a scatter based of label and column-A, it will be some straight lines, correct?
– Reza
Jan 1, 2018 at 16:40
• Yes it will be a straight line. But the point is if it will have a slope. That slope will help you get a good regression. The Chart you have at the end has a nice slope. But if I try to do a similar trendline for Top two charts, I might not be getting a nice slope as data is scattered all over the place instead of showing a good pattern. Changing encoding will not help there. Let me know if you need more detail. I can edit my answers to include some examples. Jan 3, 2018 at 10:09
• Yeah, thanks, actually the original issue is performance (r2) on my data is very low (0.20) so I am trying to find root cause and I tought categorical features could be root cause
– Reza
Jan 3, 2018 at 14:14