attach "1" for the bias unit in neural networks. what does it mean?

Question

In the image below the instructor says attach a "1" for the bias unit in neural networks. what does a "bias unit"mean in neural network?

Answer 1

It allows you to account for some static offset in your learning process. To illustrate, consider the output of your first hidden layer without the bias unit.

h = sigma(W * x)

Here W is the matrix of your layer weights, and W * x is the matrix-vector multiplication between these weights and your input. sigma is your nonlinearity operating on each element of the result. Now consider what this looks like With the "bias unit".

h = sigma(W' * [x, 1]) ~ sigma(W*x + b)

Here, W' is your weight matrix with new entries associated with the "bias unit", and your new input is your original input with a 1 concatenated to it. You can think of this as being equivalent to your original matrix multiplication W * x plus some bias term b.

I will have to dig for the sources on this, but conventional wisdom is that you don't need the bias unit in modern architectures. If it's for a class though, I'd say use it if that's what you are instructed to do.

Answer 2

You always include bias. Try to visualize it from math standpoint of view. In a single layer, you're learning to fit linear function to your input.

$ f(x_{i}, W) = Wx_{i} $

By adding b you can kinda move this function around the plot up and down, so you can fit better.

$ f(x_{i}, W, b) = Wx_{i} + b $

When it comes to neural nets, we're using something called the bias trick. We combine weight matrix W and bias vector b into a single parameter. We just add additional dimension to our input data X with a constant value. Now your scoring function becomes this:

$ f(x_{i}, W) = Wx_{i} $

and it makes implementation much easier as bias becomes just another parameter we'll be optimizing.