2
$\begingroup$

I have two questions :

  1. Why doesn't normalization have any effect on linear regressor performance (mathematical approach is appreciated ) ?
  2. When we normalize the training set we ought to normalize the target set too . Won't it affect the performance ? I mean won't the data set change completely because the model had different Ranges of features as compared to the ranges of features in target set .

I tried googling the questions but was not able to come to conclusion . Any help would be appreciated .

Thanks !

$\endgroup$
  • $\begingroup$ Can you give the code in which implement this linear regressor? $\endgroup$ – Alireza Zolanvari Mar 16 at 9:07
  • $\begingroup$ What do you mean by "performance"? Computational performance, score performance, residuals? $\endgroup$ – gented Mar 16 at 13:38
  • $\begingroup$ I mean score performance $\endgroup$ – Apoorv Jain Mar 16 at 13:38
  • $\begingroup$ Linear regressor will be effected by the scaling for sure, so try making sure that you did it correctly. Otherwise since it assigns weights to the cols, it will just pick the ones which will help it to reach the target $\endgroup$ – Aditya Mar 16 at 15:29
3
$\begingroup$
  1. Why doesn't normalization have any effect on linear regressor performance (mathematical approach is appreciated)?

Theoretically, normalization does not influence the performance of the model. In order to understand this let us have a look at a standard linear regression. $$y_i = \boldsymbol{w}^T\boldsymbol{x}_i + b+\varepsilon_i$$

If we Introduce the scaled independent variable $\boldsymbol{z}_i=\dfrac{1}{\sigma}\left[ \boldsymbol{x}_i - \bar{\boldsymbol{x}}\right] \implies \boldsymbol{x}_i=\bar{\boldsymbol{x}}+\sigma\boldsymbol{z}_i$. This will result in $$y_i = \boldsymbol{w}^T\bar{\boldsymbol{x}}+\sigma\boldsymbol{w}^T\boldsymbol{z}_i+b+\varepsilon_i.$$

If we introduce $\tilde{b}=b+\boldsymbol{w}^T\bar{\boldsymbol{x}}$ and $\tilde{\boldsymbol{w}}^T=\sigma\boldsymbol{w}^T$ we can rewrite the equation into

$$y_i=\tilde{\boldsymbol{w}}^T\boldsymbol{z}_i+\tilde{b}+\varepsilon_i.$$

Hence, we see that the transformed independent variables just change the bias (if a translation is included) and the weights are scaled by $\sigma$. The significance of the parameters will not change only their specified values.

But if normalization does not really enhance the model, why do we still do it? The reason is more computationally inspired. If we had very large values the weights would need to be very small such that we can scale the output of the regression into a reasonable range. A large range of numerical values forces us to reserve more memory for our variables that we use. Hence, it is better to normalize our inputs, such that the parameters don't need to scale the inputs down to a reasonable amount that it fits the output.

  1. When we normalize the training set we ought to normalize the target set too. Won't it affect the performance? I mean won't the data set change completely because the model had different ranges of features as compared to the ranges of features in the target set.

Normalizing the output is not necessary, but it can also improve the numerical efficiency. You can just use the previous linear transformation on your dependent variable (output) and you will see that you can rewrite it to a standard linear regression in the new output. Just remember to transform your inputs and retransform your outputs if you want to use the original variables.

Does the significance of the parameters change?

In order to show that scaling the inputs by a constant factor $\sigma$ does not influence the significance of the parameters, we will calculate the $t$-value for a given coefficient. If the $t$-value stays invariant the $p$-value will also stay invariant.

For the linear regression $y_i = \boldsymbol{w}\boldsymbol{x}_i+\varepsilon_i$ (bias absorbed into the weight vector). The regression coefficients are given by

$$\hat{\boldsymbol{w}}=\left[\boldsymbol{X}^T\boldsymbol{X} \right]^{-1}\boldsymbol{X}^T\boldsymbol{y}\quad \text{ , in which } \quad \boldsymbol{X}=\begin{bmatrix}\boldsymbol{x}_1^T \\ \vdots \\ \boldsymbol{x}_N^T\end{bmatrix}$$

is the data matrix with an added $1$-column for the bias.

Additionally, we need the matrix $$\boldsymbol{C}=\left[\boldsymbol{X}^T\boldsymbol{X} \right]^{-1}\quad \text{ and } \quad s_e = \sqrt{\dfrac{(\boldsymbol{y}-\hat{\boldsymbol{y}})^T(\boldsymbol{y}-\hat{\boldsymbol{y}})}{N-p-1}},$$

in which $N$ is the number of observations, $p$ is the number of predictors, $\boldsymbol{y}$ is the vector of outputs and $\hat{\boldsymbol{y}}$ is the vector of predicted outputs. We saw that the predicted values will stay invariant under scaling. Hence $s_e$ is invariant under scaling.

The $t$-value for a regression weight given by

$$t_i= \dfrac{\hat{w}_i-\mathbb{E}\left[\hat{w}_i\right]}{s_e\sqrt{c_{ii}}}.$$

The $c_{ii}$-values are the corresponding diagonal values of the $\boldsymbol{C}$ matrix.

If we scale our inputs by $\sigma$ (we ignore the 1 column of the data set which is only important for the bias) then we observe

$$\boldsymbol{X}'=\sigma\boldsymbol{X} \text{ , }\hat{\boldsymbol{w}}' = \dfrac{1}{\sigma}\hat{\boldsymbol{w}} \text{ , } \mathbb{E}\left[ \hat{\boldsymbol{w}}'\right]=\dfrac{1}{\sigma}\mathbb{E}\left[ \hat{\boldsymbol{w}}\right] \quad \text{ and } \quad \boldsymbol{C}' = \dfrac{1}{\sigma^2}\boldsymbol{C}.$$ The last condition implies $$\implies c_{ii}' = \dfrac{1}{\sigma^2}c_{ii} \implies \sqrt{c_{ii}'} = \dfrac{1}{\sigma}\sqrt{c_{ii}}.$$

By these observations, we see that the $t$-value stays invariant.

$$t' = \dfrac{\hat{w}_{i}'-\mathbb{E}\left[\hat{w}_{i}' \right]}{s_e\sqrt{c_{ii}'}}=\dfrac{\hat{w}_{i}-\mathbb{E}\left[\hat{w}_{i} \right]}{s_e\sqrt{c_{ii}}}=t$$

Hence, the significance of the regression coefficients didn't change either. In theory, the same should apply individually scaling the variables but the algebra gets more complicated.

$\endgroup$
  • $\begingroup$ This is slightly incorrect. The $\sigma$ aren't necessarily the same for all components and in that case I'm not sure that eventually you can factor them out. $\endgroup$ – gented Mar 17 at 3:43
  • $\begingroup$ Yes, you can, but the coefficients you end up with are different than the original ones (not just a re-scaling if the $\sigma$ are different). Essentially you just showed that a composition of two affine maps is an affine map, which is obvious - but that has nothing to do with invariance of the performance. $\endgroup$ – gented Mar 17 at 11:21
  • $\begingroup$ The intercept changes and the coefficients are re-scaled individually: this means that the significance tests may result in different values (they may or they may not). Basically my point is that an answer to the question must prove that such tests don't change - which you haven't. Same goes for the residuals (they may or may not change, but it must be proven). A sketch of the answer is provided here: stats.stackexchange.com/questions/162399/… $\endgroup$ – gented Mar 17 at 11:34
  • $\begingroup$ @gented I added the claimed result for uniform scaling. Nonuniform scaling should work as well but the algebra is more involved. $\endgroup$ – MachineLearner Mar 17 at 13:05
  • $\begingroup$ thank you, it's now a thorough answer, +1 :) $\endgroup$ – gented Mar 17 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.