Please help me understand the implications of initialising the bert model from huggingface with
is_decoder parameter set to
According to the documentation:
The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in Attention is all you need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the is_decoder argument of the configuration set to True; an encoder_hidden_states is expected as an input to the forward pass.
I know how Bert is just the encoder and there is no causal masking which is the hallmark of a decoder. Can you please me understand how
is_decoder changes the implementation of the bert to make it usable as a decoder?