In the image below the instructor says attach a "1" for the bias unit in neural networks. what does a "bias unit"mean in neural network?
0
$\begingroup$
$\endgroup$
It allows you to account for some static offset in your learning process. To illustrate, consider the output of your first hidden layer without the bias unit.
h = sigma(W * x)
Here W is the matrix of your layer weights, and W * x is the matrix-vector multiplication between these weights and your input. sigma is your nonlinearity operating on each element of the result. Now consider what this looks like With the "bias unit".
h = sigma(W' * [x, 1]) ~ sigma(W*x + b)
Here, W' is your weight matrix with new entries associated with the "bias unit", and your new input is your original input with a 1 concatenated to it. You can think of this as being equivalent to your original matrix multiplication W * x plus some bias term b.
I will have to dig for the sources on this, but conventional wisdom is that you don't need the bias unit in modern architectures. If it's for a class though, I'd say use it if that's what you are instructed to do.
