I think it is the vocab size. However I am not sure and I appreciate your help.
1 Answer
$\begingroup$
$\endgroup$
7
It is not very clear what you are referring to with "number of input neurons".
The input layer in BERT is an embedding layer, which is a table of vectors. Each of those vectors has dimensionality 768, and each vector is associated to one of the tokens in the vocabulary (so the number of vectors in the embedding table is the vocabulary size).
-
$\begingroup$ Thanks a lot @ncasas, I was thinking similar to any neural network, there should be an input layer. I found it difficult to understand how sentences with different length are feeding to the network? (is there something like zero-padding?). For example for two sentences: "I am new to BERT" and "BERT is great" the length are different but the input layer of a neural network should have a fixed number of neurons. So for the first sentence we have a 5*768 and for the second we have 3*768. $\endgroup$– AdelJan 7, 2021 at 11:22
-
$\begingroup$ You are wrong, the input of a neural network does not need to have a fixed input size. Most text-processing neural networks are perfectly capable of handling variable-length inputs. This includes LSTMs, convnets, and the Transformer (BERT's architecture). $\endgroup$– noeJan 7, 2021 at 11:33
-
$\begingroup$ The "neurons" analogy is not very useful when you move past a simple multilayer perceptron, at least to me. I think it's better to think in terms of differentiable operations over matrices. $\endgroup$– noeJan 7, 2021 at 11:34
-
$\begingroup$ About the "padding", it is used in NLP networks, but for the case when you group multiple sentences in the same mini-batch. In that case, the vocabulary contains a special
<pad>token that is used to fill up the sentences that are shorted than the longest sentence. $\endgroup$– noeJan 7, 2021 at 11:36 -
$\begingroup$ I guess I am not wrong. I know NNs can handle different input sizes, but for the sentence with different sizes, some of the neurons get 0 and therefore are not effective. $\endgroup$– AdelJan 7, 2021 at 11:40