I think it is the vocab size. However I am not sure and I appreciate your help.


It is not very clear what you are referring to with "number of input neurons".

The input layer in BERT is an embedding layer, which is a table of vectors. Each of those vectors has dimensionality 768, and each vector is associated to one of the tokens in the vocabulary (so the number of vectors in the embedding table is the vocabulary size).

  • $\begingroup$ Thanks a lot @ncasas, I was thinking similar to any neural network, there should be an input layer. I found it difficult to understand how sentences with different length are feeding to the network? (is there something like zero-padding?). For example for two sentences: "I am new to BERT" and "BERT is great" the length are different but the input layer of a neural network should have a fixed number of neurons. So for the first sentence we have a 5*768 and for the second we have 3*768. $\endgroup$ – Adel Jan 7 at 11:22
  • $\begingroup$ You are wrong, the input of a neural network does not need to have a fixed input size. Most text-processing neural networks are perfectly capable of handling variable-length inputs. This includes LSTMs, convnets, and the Transformer (BERT's architecture). $\endgroup$ – noe Jan 7 at 11:33
  • $\begingroup$ The "neurons" analogy is not very useful when you move past a simple multilayer perceptron, at least to me. I think it's better to think in terms of differentiable operations over matrices. $\endgroup$ – noe Jan 7 at 11:34
  • $\begingroup$ About the "padding", it is used in NLP networks, but for the case when you group multiple sentences in the same mini-batch. In that case, the vocabulary contains a special <pad> token that is used to fill up the sentences that are shorted than the longest sentence. $\endgroup$ – noe Jan 7 at 11:36
  • $\begingroup$ I guess I am not wrong. I know NNs can handle different input sizes, but for the sentence with different sizes, some of the neurons get 0 and therefore are not effective. $\endgroup$ – Adel Jan 7 at 11:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.