for better understanding, I would like to represent the calculations in a neural network with one hidden layer and one output layer as a matrix calculation. The hidden layer has 3072 neurons, the output layer 768. Specifically, we are talking about the Feed Forward Neural Network, which is used in BERT's Transformer architecture. There, a vector 1x768 is fed into the FFNN and the target is an output vector with 1x768 as well.

The calculation would be as follows :

 Input      Hidden Layer   Output Layer       Output Vector  
[1x768] x [768 x 3072] x [3072 x 768] =        [1 x 768]

Is this correct? Thank you very much!

  • $\begingroup$ This would indeed be correct if you would not be using biases in you neural network. $\endgroup$
    – Oxbowerce
    Aug 24 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.