I'm trying to understand the workings and output of BERT, and I'm wondering how/why each layer of BERT has a 'hidden state'.
I understand what RNN's have a 'hidden state' that gets passed to each time step, which is a representation of previous inputs. But I've read that BERT isn't a RNN - it's a CNN with attention.
But you can output the hidden state for each layer of a BERT model. How is it that BERT has hidden states if it's not a RNN?