# What is the number of neurons for the input layer of the BERT?

I think it is the vocab size. However I am not sure and I appreciate your help.

It is not very clear what you are referring to with "number of input neurons".

The input layer in BERT is an embedding layer, which is a table of vectors. Each of those vectors has dimensionality 768, and each vector is associated to one of the tokens in the vocabulary (so the number of vectors in the embedding table is the vocabulary size).

• Thanks a lot @ncasas, I was thinking similar to any neural network, there should be an input layer. I found it difficult to understand how sentences with different length are feeding to the network? (is there something like zero-padding?). For example for two sentences: "I am new to BERT" and "BERT is great" the length are different but the input layer of a neural network should have a fixed number of neurons. So for the first sentence we have a 5*768 and for the second we have 3*768.
• About the "padding", it is used in NLP networks, but for the case when you group multiple sentences in the same mini-batch. In that case, the vocabulary contains a special <pad> token that is used to fill up the sentences that are shorted than the longest sentence.