How to obtain original coefficients after performing linear regression on normalized data?

Question

I am reading data from a file using pandas which looks like this:

data.head()

   ldr1  ldr2  servo
0   971   956     -2
1   691   825   -105
2   841   963    -26
3   970   731     44
4   755   939    -69

I proceed to normalize this data to perform gradient descent:

my_data = (my_data - my_data.mean())/my_data.std()
my_data.head()

       ldr1      ldr2     servo
0  1.419949  1.289668  0.366482
1 -0.242834  0.591311 -1.580420
2  0.647943  1.326984 -0.087165
3  1.414011  0.090200  1.235972
4  0.137231  1.199041 -0.899949

I perform multivariate regression and end up with fitted parameters on the normalized data:

Thetas:  [[-3.86865143e-17,  8.47885685e-01, -5.39083511e-01]]

I would like to plot the plane of best fit on the original data and not the normalized data using the normalized thetas.

I used scipy.optimize.curve_fit to perform multivariate linear regression and come up with the optimal fitted parameters. I know that the original thetas should be close to the following:

[   0.26654135   -0.15218007 -107.79915373]

How can I get the 'original' thetas for the original data-set in order to plot, without using Scikit-Learn?

Any suggestions will be appreciated.

Answer 1

Coefficients of the linear regression for unnormalized features

If parameters in the normalized space are denoted as $(\theta_0', \theta_1', \theta_2')$, parameters in the original space $(\theta_0, \theta_1, \theta_2)$ can be derived as follows $$\begin{align*} y' &= \theta_2'x_2'+\theta_1'x_1'+\theta_0'\\ \frac{y-\mu_Y}{\sigma_Y} &= \theta_2'\frac{x_2 - \mu_{X_2}}{\sigma_{X_2}} + \theta_1'\frac{x_1 - \mu_{X_1}}{\sigma_{X_1}} +\theta_0' \\ y &= \overbrace{\left(\frac{\sigma_{Y}}{\sigma_{X_2}}\theta_2'\right)}^{\theta_2}x_2+ \overbrace{\left(\frac{\sigma_{Y}}{\sigma_{X_1}}\theta_1'\right)}^{\theta_1}x_1 + \overbrace{\sigma_Y\left(-\theta_2'\frac{\mu_{X_2}}{\sigma_{X_2}}-\theta_1'\frac{\mu_{X_1}}{\sigma_{X_1}} + \theta_0'\right) + \mu_Y}^{\theta_0} \end{align*}$$

Generalization to D features

$$\begin{align*} \theta_d = \left\{\begin{matrix} \sigma_Y \left(\theta_0' - \sum_{i=1}^{D}\theta_i'\frac{\mu_{X_i}}{\sigma_{X_i}} \right) + \mu_Y& d=0\\ \frac{\sigma_{Y}}{\sigma_{X_d}}\theta_d' & d > 0 \end{matrix}\right. \end{align*}$$

A trick

For visualization, we can plot the plane in original, un-normalized space without changing the parameters (Thetas). We only need to re-label (re-scale) the plot axes.

For example, a point $(x_1', x_2', y')$ in the normalized space corresponds to point $$(x_1, x_2, y)=(\sigma_{X_1}x_1'+\mu_{X_1}, \sigma_{X_2}x_2'+\mu_{X_2}, \sigma_{Y}y'+\mu_{Y})$$

in the original space. So you just need to rename the plot axes from $(x_1', x_2', y')$ to $(x_1, x_2, y)$.

Note that $y'=\theta_2'x_2'+\theta_1'x_1'+\theta_0'$ is still calculated using normalized features.