2
$\begingroup$

I'm going to use PyTorch specifically but I suspect my question applies to deep learning & CNNs in general therefore I choose to post it here.

Starting at this point in this video and subsequently:

https://www.youtube.com/watch?v=JRlyw6LO5qo&t=1370s

George H. explains that the PyTorch function torch.nn.Linear with the bias parameter set to False then makes the torch.nn.Linear functionally equivalent (other than GPU support of course) to the following NumPy line:

x = np.dot(weights, x) + biases

Note that in torch.nn.Linear bias by default is set to True:

https://pytorch.org/docs/stable/generated/torch.nn.Linear.html

Here is the PyTorch documentation for the bias parameter:

bias – If set to False, the layer will not learn an additive bias. Default: True

Can anybody please explain what "additive bias" is? In other words, what additional steps is PyTorch doing if torch.nn.Linear bias parameter set to True? Surprisingly I was not able to find much on this topic upon Googleing.

$\endgroup$
1
$\begingroup$

You probably misanderstood the video, it is not said that a linear layer with Bias set to False is equivalent to :

x = np.dot(weights, x) + biases

Because that is not true, a layer without Bias is equivalent to

x = np.dot(weights, x)

The way he recreates the layer without Bias is actually with the following function :

x = x.dot(l1)  # X = W1.X  First linear layer
x = np.maximum(x, 0)  # X = ReLU(X)
x = x.dot(l2)  # X = W2.X  Second linear layer

Setting Bias=True means the layer has the second bias term it adds after multiplying the weights with the input (as in the formula you quoted) :

x = np.dot(weights, x) + biases
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.