# attach “1” for the bias unit in neural networks. what does it mean?

In the image below the instructor says attach a "1" for the bias unit in neural networks. what does a "bias unit"mean in neural network? ## 2 Answers

It allows you to account for some static offset in your learning process. To illustrate, consider the output of your first hidden layer without the bias unit.

h = sigma(W * x)


Here W is the matrix of your layer weights, and W * x is the matrix-vector multiplication between these weights and your input. sigma is your nonlinearity operating on each element of the result. Now consider what this looks like With the "bias unit".

h = sigma(W' * [x, 1]) ~ sigma(W*x + b)


Here, W' is your weight matrix with new entries associated with the "bias unit", and your new input is your original input with a 1 concatenated to it. You can think of this as being equivalent to your original matrix multiplication W * x plus some bias term b.

I will have to dig for the sources on this, but conventional wisdom is that you don't need the bias unit in modern architectures. If it's for a class though, I'd say use it if that's what you are instructed to do.

You always include bias. Try to visualize it from math standpoint of view. In a single layer, you're learning to fit linear function to your input.

$$f(x_{i}, W) = Wx_{i}$$ By adding b you can kinda move this function around the plot up and down, so you can fit better.

$$f(x_{i}, W, b) = Wx_{i} + b$$ When it comes to neural nets, we're using something called the bias trick. We combine weight matrix W and bias vector b into a single parameter. We just add additional dimension to our input data X with a constant value. Now your scoring function becomes this:

$$f(x_{i}, W) = Wx_{i}$$

and it makes implementation much easier as bias becomes just another parameter we'll be optimizing.