I'm trying to understand the workings and output of BERT, and I'm wondering how/why each layer of BERT has a 'hidden state'.

I understand what RNN's have a 'hidden state' that gets passed to each time step, which is a representation of previous inputs. But I've read that BERT isn't a RNN - it's a CNN with attention.

But you can output the hidden state for each layer of a BERT model. How is it that BERT has hidden states if it's not a RNN?


1 Answer 1


BERT is a transformer.

A transformer is made of several similar layers, stacked on top of each others.
Each layer have an input and an output. So the output of the layer n-1 is the input of the layer n.

The hidden state you mention is simply the output of each layer.

You might want to quickly look into this explanation of the Transformer architecture : https://jalammar.github.io/illustrated-transformer/

Note that BERT use only Encoders, no Decoders.

  • 1
    $\begingroup$ Thanks for your response! I've read that a few times, but I never saw a mention of a 'hidden state'. Is it just intuitive that that's what the hidden states must be? When I google 'BERT hidden state', I see that you can get the hidden states in the BERT output, but no document or article that says that it's the output of each layer, as you explain. Basically, I'm wondering what I was missing, or how you know that that's what the hidden states refer to. $\endgroup$ Jan 22, 2020 at 0:25
  • 1
    $\begingroup$ You're right, it's not mentioned explicitly. I personally understood this notion of hidden state by looking at the source code of Bert implementations. $\endgroup$
    – Astariul
    Jan 22, 2020 at 1:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.