The Keras library list only has 3 types of attentions - keras attention layers, which are :

  1. MultiHeadAttention layer
  2. Attention layer
  3. AdditiveAttention layer

However, in theory there are multiple types of attentions possible, e.g. (some of these may be synonyms):

  • Global
  • Local
  • Hard
  • Bahdanau Attention
  • Luong Attention
  • self
  • additive
  • Latent
  • what else?

Are other types just not practical or other types actually can be derived from existing implementation? Can someone please shed some light with examples?

  • $\begingroup$ This is a design choice from library authors... maybe ask them on github. But it is not clear we can provide a fact based answer. $\endgroup$ Commented Jan 11 at 10:48

1 Answer 1


You can always build your own custom attention layer using TF or Pytorch. Keras has three popular inbuilt Attention Layer now. Before we had to build our own custom layers even in Keras. The diffrence between Bahdanou and Loung is the way that the attention weights are calculated. One is additive and the other multiplicative(dot product) respectively. Both of these models the key and the value are same. For more advanced transformer techniques refer to BERT or DistilBERT. We also have various variations in BERT.

  • $\begingroup$ Of course we can build our code. But wondering why keras not provided the very much required code over years? Or have they actually provided all by just those 3 layers? $\endgroup$ Commented Jun 23, 2021 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.