# What is a 'hidden state' in BERT output?

I'm trying to understand the workings and output of BERT, and I'm wondering how/why each layer of BERT has a 'hidden state'.

I understand what RNN's have a 'hidden state' that gets passed to each time step, which is a representation of previous inputs. But I've read that BERT isn't a RNN - it's a CNN with attention.

But you can output the hidden state for each layer of a BERT model. How is it that BERT has hidden states if it's not a RNN?

BERT is a transformer.

A transformer is made of several similar layers, stacked on top of each others.
Each layer have an input and an output. So the output of the layer n-1 is the input of the layer n.

The hidden state you mention is simply the output of each layer.

You might want to quickly look into this explanation of the Transformer architecture : https://jalammar.github.io/illustrated-transformer/

Note that BERT use only Encoders, no Decoders.

• Thanks for your response! I've read that a few times, but I never saw a mention of a 'hidden state'. Is it just intuitive that that's what the hidden states must be? When I google 'BERT hidden state', I see that you can get the hidden states in the BERT output, but no document or article that says that it's the output of each layer, as you explain. Basically, I'm wondering what I was missing, or how you know that that's what the hidden states refer to. – Nick Koprowicz Jan 22 '20 at 0:25
• You're right, it's not mentioned explicitly. I personally understood this notion of hidden state by looking at the source code of Bert implementations. – Astariul Jan 22 '20 at 1:34