# Dummy Variable trap in Linear Regression

The dummy variable trap is a common problem with linear regression when dealing with categorical variables, since one hot encoding introduces redundancy, so if we have m categories in our categorical variable we usually drop one dummy variable to have m-1 dummy variables instead of m dummy variables. Now, when talking about linear regression we must do so. The reason behind this can be quoted from the book Data Preparation for Machine Learning: Data Cleaning, Feature Selection, and Data Transforms in Python by Jason Brownlee , page 245 :

For example, in the case of a linear regression model (and other regression models that have a bias term), a one hot encoding will cause the matrix of input data to become singular, meaning it cannot be inverted and the linear regression coefficients cannot be calculated using linear algebra. For these types of models a dummy variable encoding must be used instead.

So according to these words, if we use one hot encoding without removing one dummy variable, we will not be able to get the coefficients of the model, however, in many cases we can still use one hot encoding with linearregression() class provided by sklearn library and the coefficients are then found easily by this class. So my first question is can someone explain the statement mentioned in this book because it seems to me we have a contradiction.

Moreover, when using drop="first" with onehotencoding(), we cant set the handle_unknown attribute to "ignore", so in the case when we have unknown category in the test set or in any prediction we will get an error, because we don't have any options rather than to set the handle_unknown attribute to "error" . After doing a research to fix such problem, many people suggest to use one hot encoding; so this brings me to another question, how can we use one hot encoding if this encoding method introduces the dummy variable trap?

• A few clarifications. Only need to drop a one hot encoded column if fitting an intercept since the intercept columns comes in with all 1's. Also the pseudoinverse can be used instead - here and here. Most models I have seen use an intercept but some do not. Also I like to control which encoded category I drop (which means it goes in the intercept) for interpretability, suiting the data and the business problem. Oct 3, 2022 at 12:26

drop and 'handle_unknown='ignore' do work together since sklearn 1.0.2.
• Vanilla LinearRegression() does not, though Ridge() and other classes sklearn offers are still doing linear regression. Sklearn implementation wraps scipy.linalg.lstsq(), which utilizes a pseudo-inverse I believe (see stats.stackexchange.com/questions/494824, stats.stackexchange.com/questions/240573) and thus can potentially handle matrices with a perfect collinearity, too. As it was mentioned in the comments, you may also choose not to fit_intercept and thus have no bias term. Oct 4, 2022 at 9:06
• If you do it the classic way, like weights = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train, it will fail with a singular matrix. However, you can work it around using something like U, S, Vt = np.linalg.svd(X_train, full_matrices=False); weights = Vt.T @ np.clip(np.linalg.inv(np.diag(S)), 0, 1e+9) @ U.T @ y_train. I assume the actual implementation is more robust (sklearn uses scipy which uses LAPACK which I'm sadly not too knowledgeable about). Oct 4, 2022 at 14:17