Please help me understand the implications of initialising the bert model from huggingface with is_decoder parameter set to True

According to the documentation:

The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in Attention is all you need by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the is_decoder argument of the configuration set to True; an encoder_hidden_states is expected as an input to the forward pass.

I know how Bert is just the encoder and there is no causal masking which is the hallmark of a decoder. Can you please me understand how is_decoder changes the implementation of the bert to make it usable as a decoder?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.