For a linear model without intercept, why does the redundent term in one-hot encoding function as intercept?

Question

In this question Elias Strehle pointed out that if we keep all the levels during one hot encoding on a linear model without an intercept, the redundant feature will function as an intercept. Why is this the case?

Isn't that in a linear model, the intercept term ($x_0$, not $\beta_0$) will always have a value of 1? Suppose the feature to encode is gender, then some of the rows of the redundant term will be 0 while others will be 1s.

Answer 1

Let's have a look how dummies work:

R Example:

# Some data
df = data.frame(y=c(30,32,28,10,11,9),gender=c(1,1,1,0,0,0), gender2=c(0,0,0,1,1,1))

# 1) Regression with constant and dummy
summary(lm(y~gender,data=df))

# 2) Regression without constant and dummy
summary(lm(y~gender-1,data=df))

# 3) Regression without constant and two dummies
summary(lm(y~gender+gender2-1,data=df))

Results:

Case 1: Since dummies generally work as "contrasts" to some base category (1 vs. 0 / "on" vs. "off") and since the base category has a mean of 10, the intercept term equals 10 and for gender = 1, the difference to the base category is identified (here 20) because the mean of category gender = 1 is 30. (Remember that a regression with only an intercept or with dummies simply gives the arithmetic mean).

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  10.0000     0.9129   10.95 0.000394 ***
gender       20.0000     1.2910   15.49 0.000101 ***

Case 2: Without a constant only gender = 1 is considered, since for gender = 0 we have $0 + 0 * \beta$ so that gender = 0 is dropped. The coefficient now is the mean of gender = 1.

Coefficients:
       Estimate Std. Error t value Pr(>|t|)   
gender   30.000      4.546   6.599   0.0012 **

Case 3: Including a dummy for both groups (denote gender = 0 from above as gender2) without adding an intercept now gives the mean for each group directly. Note that the interpretation of the coefficients is different here compared to case 1.

Coefficients:
        Estimate Std. Error t value Pr(>|t|)    
gender   30.0000     0.9129   32.86 5.11e-06 ***
gender2  10.0000     0.9129   10.95 0.000394 ***

The interesting bit is when you add some additional $x$:

Some new data, now including $x$:

df = data.frame(y=c(30,32,28,10,11,9),gender=c(1,1,1,0,0,0), gender2=c(0,0,0,1,1,1), x=c(20,22,25,28,30,29))

Regression with both dummies, no intercept:

summary(lm(y~gender2+gender-1+x,data=df))

Coefficients:
        Estimate Std. Error t value Pr(>|t|)  
gender2  19.8864    12.6285   1.575   0.2134  
gender   37.6136     9.7446   3.860   0.0307 *
x        -0.3409     0.4342  -0.785   0.4897  

is the same as...

Regression with one dummy and intercept (apart of the dummy interpretation discused above):

summary(lm(y~gender+x,data=df))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept)  19.8864    12.6285   1.575   0.2134  
gender       17.7273     3.1973   5.544   0.0116 *
x            -0.3409     0.4342  -0.785   0.4897

... so the marginal effect of $x$ is the same. This is in contrast to...

Regression with one dummy, no intercept:

summary(lm(y~gender+x-1,data=df))

Coefficients:
       Estimate Std. Error t value Pr(>|t|)    
gender 22.38736    1.41677  15.802 9.37e-05 ***
x       0.34086    0.03864   8.822 0.000911 ***

Here the marginal effect of $x$ is entirely different.

Why is this?

When you fit some new data, you will see that the fitted line for $x$ goes through (0,0) "no intercept in the model".

newdata = data.frame(gender=c(0,0,0,0,0,0), x=c(-1,0,1,2,3,4))
predict(lm(y~gender+x-1,data=df), newdata=newdata)

         1          2          3          4          5          6 
-0.3408643  0.0000000  0.3408643  0.6817286  1.0225929  1.3634572 

This happens because there are cases in which you have $0 + \beta x$ (which is 0 for $x=0$). Or as $x$-matrix (first row would be the intercept, for illustration = 0):

\begin{pmatrix} 0 & 1 & x_1\\ 0 & 1 & x_2\\ 0 & 0 & x_3\\ 0 & 0 & x_4 \end{pmatrix}

However, when you have the two dummies included, you have:

\begin{pmatrix} 0 & 1 & x_1\\ 0 & 1 & x_2\\ 1 & 0 & x_3\\ 1 & 0 & x_4 \end{pmatrix}

So there is no case in which you force $\beta x$ to be zero.

See this post for further discussion on regression without constant term.