2
$\begingroup$

The dummy variable trap is a common problem with linear regression when dealing with categorical variables, since one hot encoding introduces redundancy, so if we have m categories in our categorical variable we usually drop one dummy variable to have m-1 dummy variables instead of m dummy variables. Now, when talking about linear regression we must do so. The reason behind this can be quoted from the book Data Preparation for Machine Learning: Data Cleaning, Feature Selection, and Data Transforms in Python by Jason Brownlee , page 245 :

For example, in the case of a linear regression model (and other regression models that have a bias term), a one hot encoding will cause the matrix of input data to become singular, meaning it cannot be inverted and the linear regression coefficients cannot be calculated using linear algebra. For these types of models a dummy variable encoding must be used instead.

So according to these words, if we use one hot encoding without removing one dummy variable, we will not be able to get the coefficients of the model, however, in many cases we can still use one hot encoding with linearregression() class provided by sklearn library and the coefficients are then found easily by this class. So my first question is can someone explain the statement mentioned in this book because it seems to me we have a contradiction.

Moreover, when using drop="first" with onehotencoding(), we cant set the handle_unknown attribute to "ignore", so in the case when we have unknown category in the test set or in any prediction we will get an error, because we don't have any options rather than to set the handle_unknown attribute to "error" . After doing a research to fix such problem, many people suggest to use one hot encoding; so this brings me to another question, how can we use one hot encoding if this encoding method introduces the dummy variable trap?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ A few clarifications. Only need to drop a one hot encoded column if fitting an intercept since the intercept columns comes in with all 1's. Also the pseudoinverse can be used instead - here and here. Most models I have seen use an intercept but some do not. Also I like to control which encoded category I drop (which means it goes in the intercept) for interpretability, suiting the data and the business problem. $\endgroup$
    – Craig
    Commented Oct 3, 2022 at 12:26

1 Answer 1

Reset to default
1
$\begingroup$

drop and 'handle_unknown='ignore' do work together since sklearn 1.0.2.

Also, using a regularized model should ensure matrix inversion is still solvable (see e.g. https://stats.stackexchange.com/questions/559575).

Improve this answer
$\endgroup$
4
  • $\begingroup$ But Linearregression() doesn't add any regularization to the model according to the documentation, so how it can works with one hot encoding without drop="first" ? $\endgroup$
    – John adams
    Commented Oct 4, 2022 at 8:12
  • 1
    $\begingroup$ Vanilla LinearRegression() does not, though Ridge() and other classes sklearn offers are still doing linear regression. Sklearn implementation wraps scipy.linalg.lstsq(), which utilizes a pseudo-inverse I believe (see stats.stackexchange.com/questions/494824, stats.stackexchange.com/questions/240573) and thus can potentially handle matrices with a perfect collinearity, too. As it was mentioned in the comments, you may also choose not to fit_intercept and thus have no bias term. $\endgroup$
    – dx2-66
    Commented Oct 4, 2022 at 9:06
  • $\begingroup$ Yes I know these alternative solutions, but the question is if I use LinearRegression() with one hot encoding still the coefficients can be calculated , although LinearRegression() doesn't add a regularization term , and hence, the matrix of features will be singular and in this case we should not get an output. So how we still can get an output with LinearRegression() used with one hot encoded ( with no drop="first" I mean) ? Thanks a lot for your efforts ! $\endgroup$
    – John adams
    Commented Oct 4, 2022 at 12:34
  • 2
    $\begingroup$ If you do it the classic way, like weights = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train, it will fail with a singular matrix. However, you can work it around using something like U, S, Vt = np.linalg.svd(X_train, full_matrices=False); weights = Vt.T @ np.clip(np.linalg.inv(np.diag(S)), 0, 1e+9) @ U.T @ y_train. I assume the actual implementation is more robust (sklearn uses scipy which uses LAPACK which I'm sadly not too knowledgeable about). $\endgroup$
    – dx2-66
    Commented Oct 4, 2022 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.