6
$\begingroup$

While there may not be any added value in standardizing one-hot encoded features prior to applying linear models, is there is any harm in doing so (i.e., affecting model performance)?

Standardizing definition: applying (x - mean) / std to make the feature mean and std 0, 1 respectively)

I prefer applying standardization to my entire training dataset after one-hot encoding, rather than applying it only to the numerical features. I feel it would significantly simplify my pipeline.

For example, if I have a binary feature then the vector that will be provided to the model is [1,1,0,0,0,1,1].

If standardization is applied to this binary feature prior to fitting the model (subtract mean = ~0.57 and divide by std = ~ 0.49), the vector will become

[ 0.8660254 , 0.8660254 , -1.15470054, -1.15470054, -1.15470054, 0.8660254 , 0.8660254 ]

$\endgroup$
2
  • 1
    $\begingroup$ What linear model? OLS? And what language? If you use R, for instance, and you standardize, and AFTER that apply as.factor(), you are fine. Otherwise, when you do not apply a „as.factor“ function but feed raw numbers into the regression, „dummies“ should be either 1 or 0. $\endgroup$
    – Peter
    Aug 13 '20 at 21:37
  • 1
    $\begingroup$ The question is unrelated to a specific language and it is more about the theory of standardizing features which are binary. By Linear models I mean Linear regression, Lasso, Ridge, ElasticNet, Logistic regression, etc. $\endgroup$ Aug 14 '20 at 15:44
3
$\begingroup$

With unpenalized linear models, there is no difference. The coefficients will just scale to counteract the new scale of the variables, and the intercept will shift to compensate for the centering.

With penalized linear models though, there will be a difference. Since the standard deviation of a binary variable is at most $1/2$, you'll be increasing the overall scale of the variable by standardizing. That will cause the unpenalized coefficient to decrease in magnitude, which will change the balance of how the penalty applies to different features. I suspect there is no "better" approach then: sometimes the penalty improves performance when the dummies are scaled, and sometimes degrades performance.

$\endgroup$
1
$\begingroup$

It makes no sense to standardize one-hot encoded features. One-hot encoding implies the level of the measurement for a feature is nominal / categorial. Standardization implies the level of measure for a features is at least interval.

For example, if the feature is country of origin. Since that feature is categorical, one-hot encoding makes sense. A person is from a country or not. Taking the mean of the country of origin yields numbers that do not make sense.

$\endgroup$
1
  • $\begingroup$ I agree that is has a negative effect from a model interpretation point of view (which is true when you standardize any numerical feature.) My question was regarding the effect on model performance (e.g., F1, Recall, Accuracy, MSE, MAE, etc. ) $\endgroup$ Oct 5 '20 at 14:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.