2
$\begingroup$

In the image below the instructor says attach a "1" for the bias unit in neural networks. what does a "bias unit"mean in neural network?

enter image description here

$\endgroup$
0
$\begingroup$

It allows you to account for some static offset in your learning process. To illustrate, consider the output of your first hidden layer without the bias unit.

h = sigma(W * x)

Here W is the matrix of your layer weights, and W * x is the matrix-vector multiplication between these weights and your input. sigma is your nonlinearity operating on each element of the result. Now consider what this looks like With the "bias unit".

h = sigma(W' * [x, 1]) ~ sigma(W*x + b)

Here, W' is your weight matrix with new entries associated with the "bias unit", and your new input is your original input with a 1 concatenated to it. You can think of this as being equivalent to your original matrix multiplication W * x plus some bias term b.

I will have to dig for the sources on this, but conventional wisdom is that you don't need the bias unit in modern architectures. If it's for a class though, I'd say use it if that's what you are instructed to do.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy