In the image below the instructor says attach a "1" for the bias unit in neural networks. what does a "bias unit"mean in neural network?
$\begingroup$
$\endgroup$
It allows you to account for some static offset in your learning process. To illustrate, consider the output of your first hidden layer without the bias unit.
h = sigma(W * x)
Here W is the matrix of your layer weights, and W * x is the matrix-vector multiplication between these weights and your input. sigma is your nonlinearity operating on each element of the result. Now consider what this looks like With the "bias unit".
h = sigma(W' * [x, 1]) ~ sigma(W*x + b)
Here, W' is your weight matrix with new entries associated with the "bias unit", and your new input is your original input with a 1 concatenated to it. You can think of this as being equivalent to your original matrix multiplication W * x plus some bias term b.
I will have to dig for the sources on this, but conventional wisdom is that you don't need the bias unit in modern architectures. If it's for a class though, I'd say use it if that's what you are instructed to do.
$\begingroup$
$\endgroup$
You always include bias. Try to visualize it from math standpoint of view. In a single layer, you're learning to fit linear function to your input.
$ f(x_{i}, W) = Wx_{i} $
By adding b you can kinda move this function around the plot up and down, so you can fit better.
$ f(x_{i}, W, b) = Wx_{i} + b $
When it comes to neural nets, we're using something called the bias trick. We combine weight matrix W and bias vector b into a single parameter. We just add additional dimension to our input data X with a constant value. Now your scoring function becomes this:
$ f(x_{i}, W) = Wx_{i} $
and it makes implementation much easier as bias becomes just another parameter we'll be optimizing.
