In my textbook, I read that whenever you reduce the mean of each feature from corresponding features in the training data and divide each feature by its standard deviation (this process is called Normalizing input data), the bias term is not significant. I don't understand this. Why is that?

To provide extra clarification I provided the following image:

(The left one is badly conditioned and the right one is the one which has been normalized).

conditioned image

  • 1
    $\begingroup$ The bias term of what? Of a regression? $\endgroup$ – kbrose Oct 19 '17 at 17:09
  • $\begingroup$ @kbrose as I have added tag, I meant neural nets $\endgroup$ – Media Oct 19 '17 at 17:35
  • $\begingroup$ It certainly can be significant. Do a regression problem where the expected output is always 1 with random inputs. Now in practice it may be less useful if you’ve normalized your inputs. As with most questions like this, it’s best to try it yourself and experiment $\endgroup$ – kbrose Oct 19 '17 at 23:23
  • $\begingroup$ @kbrose I can not add the exact text of the book but for clarity take a look at here. Although professor speaks about regularization, he says an important fact; almost all the parameters are in W rather b. I wanted to know if there is any math behind it. $\endgroup$ – Media Oct 20 '17 at 4:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.