Parameter estimation in linear regression

Question

Another test Q I couldn't answer - We have marks of students belonging to 3 sections - A,B,C and two genders - M & F. Which regression model will not be able to estimate all the parameters?

1 ) Marks = β0 + β1 Male + β2 Female + β3 section A + β4 section B + β5 section C

2. Marks = β0 + β1 Male + β4 section B + β5 section C

3. Marks = β0 + β1 Male + β5 section C

4. Marks = β0 + β1 Male

I couldn't understand this question at all. If anyone can, kindly explain. Thanks in advance.

Answer 1

This might be a trick question, if so then the answer could be the first option. The reason being that the first model estimates a coefficient for males and females separately, but the effect of one of these will already be in the intercept, so this will result in one of the male/female coefficients to be undetermined.

Answer 2

The typical way to encode a category is with a 0/1 indicator about absence or presence of that category. If you do this, then model $\#1$ results in your model matrix having three columns (the sections) that add up to a vector of all $1$, which is in the model matrix as the intercept column (ditto for the male/female columns adding up to that same vector of $1$s). This means that the model matrix lacks full rank, so the OLS solution is not unique. It is typical to refer to the parameters as not being “identifiable” in such a situation, which is quite close to the phrasing in your test question.

What’s happening is that you can add $c$ to the intercept and not change the predictions is you subtract that same $c$ from $\beta_3$, $\beta_4$, and $\beta_5$ (ditto for subtracting $c$ from $\beta_1$ and $\beta_2$).

Saying that we can’t estimate the parameters by minimizing the sum of squared residuals takes it somewhat too far. There are infinitely many solutions that minimize the sum of squared residuals! However, that is closest to what the question is asking, and there are not estimation issues with the other three models.

(Technically, there should be a “hat” on the parameter estimates discussed above to indicate that they are estimates instead of the parameters themselves.)