Linear Regression with Category variables

Question

I'm currently learning and exploring machine learning and understand the basics of linear regression based on two numerical variables, but now I wish to go a little further and need some guidance understanding how to go about it.

Specifically, I'm now learning about linear regression with categorical variables, and I understand the gist of it: We just encode the categorical variable to some sort of numerical representation (like one-hot encoding) and put it in the model. Great.

While there are many guides on how to do various encoding methods etc. online, I haven't really found a resource that explains the use-cases of such a method: under what kind of circumstances would using categorical data to predict a numeric value be useful?

And what type of data format should I have my data in before doing the encoding? (does having two columns with one numeric results and the other the corresponding category work?)

I would also like to know the different ways we can visualize and analyse the results of our model (and its predictions), especially if we have a sizeable amount of categorical variables.

Sorry if this is too many questions in one post, I need some guidance on the concept. All the online resources are telling me how to implement the model, but not when and why to use it.

Answer 1

We just encode the categorical variable to some sort of numerical representation (like one-hot encoding)

The choice of representation matters, because it has to preserve the properties of a categorical variable: one-hot-encoding is a standard option, but directly encoding categorical values as integers is a mistake because it introduces order where there is none.

the use-cases of such a method: under what kind of circumstances would using categorical data to predict a numeric value be useful?

There are many applications for which regression from categorical data is useful. For example text data is often represented as "bag of words" (BoW), where each word is a categorical variable. Some tasks involve predicting a numerical target from some text input, for example:

• Predicting the grade of an essay
• Predicting a score representing the likelihood of plagiarism (e.g.Turnitin)
• Predicting sentiment of a text on a scale from 1 to 5

There are many other examples, and not only from text data.

And what type of data format should I have my data in before doing the encoding? (does having two columns with one numeric results and the other the corresponding category work?)

The data can have as many independent variables (features) as needed. Since the categorical variables are typically one-hot-encoded, what matters is the number of features after encoding. It's often necessary to preprocess the data in order to avoid overfitting (typically rare values should be discarded).

I would also like to know the different ways we can visualize and analyse the results of our model (and its predictions), especially if we have a sizeable amount of categorical variables.

As far as I know there's no specific method for analyzing the result of the model, it's the same principle as with numerical features. If one wants to analyze the impact of an individual feature on the performance of the model, a simple technique is to train the model with/without this feature and compare the results.

Answer 2

Suppose you have a question like: "How does car weight affect miles per gallon (mpg)?"

Load and plot the "car data". In the first plot you can clearly see that there is a (more or less) linear relation between $weight$ and $mpg$.

Now you can ask: is there a difference in time? You can add a "dummy" for the years $\leq$ 1975 to flag this years ($=1$ if true $=0$ otherwise).

When you plot the data for the two periods of time ($\leq$ 1975 vs. $>$ 1975), you can see that there is quite a difference.

library(ISLR)
library(dplyr)

# Car dara
df = ISLR::Auto
df = df %>% select(weight,mpg,year)
summary(df)
# Dummy encoding "time"
df$y70_75 = ifelse(df$year<=75, 1, 0)

# Plot
par(mfrow=c(1,2))
plot(df$weight,df$mpg,xlab="Weight", ylab="MPG", ylim=c(10,50))
 
# Plot with "time dummy"
plot(df$weight[df$y70_75==1],df$mpg[df$y70_75==1],xlab="Weight", ylab="MPG", col="red", ylim=c(10,50))
points(df$weight[df$y70_75==0],df$mpg[df$y70_75==0])

Now we can plug this into a linear regression, essentially fitting one linear line for the black dots and one linear line for the red dots (indicated by the dummy). The model looks like

$$ mpg = \beta_0 + \beta_1 weight + \beta_3 dummy_{70-75} + u$$

# Regression
reg = lm(mpg~weight,data=df)
summary(reg)

# Regression with time dummy
reg2 = lm(mpg~weight+y70_75,data=df)
summary(reg2)

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 46.325718   0.689669   67.17   <2e-16 ***
weight      -0.006985   0.000230  -30.37   <2e-16 ***
y70_75      -4.533643   0.391472  -11.58   <2e-16 ***

For the years $\leq$ 1975, the estimated $mpg$ would be $43.3 - 4.53 * 1 - 0.007 * weight$, for the remaining years ($>$ 1975), the estimated effect would be $43.3 - 4.53 * 0 - 0.007 * weight$.

Adding a dummy here simply augments the intercept term conditional on the dummy. Here this would be a different intercept term conditional on time as defined by the dummy.

We can say that (on average) $mpg$ for the years $\leq$ 1975 was lower than in later years. Or in other words, for a given $weight$, $mpg$ was about -4.5 units lower in $\leq$ 1975 compared to later years.

When you have a lot of dummies, you can end up with more variables (columns) than observations (rows) which is a "high dimensional" problem. This is - for instance - the case when you dummy encode text as a "bag of words". In this case you need to use regularization in linear models (e.g. Lasso/Ridge).