1
$\begingroup$

The Keras library list only has 3 types of attentions - keras attention layers, which are :

  1. MultiHeadAttention layer
  2. Attention layer
  3. AdditiveAttention layer

However, in theory there are multiple types of attentions possible, e.g. (some of these may be synonyms):

  • Global
  • Local
  • Hard
  • Bahdanau Attention
  • Luong Attention
  • self
  • additive
  • Latent
  • what else?

Are other types just not practical or other types actually can be derived from existing implementation? Can someone please shed some light with examples?

$\endgroup$
0
$\begingroup$

You can always build your own custom attention layer using TF or Pytorch. Keras has three popular inbuilt Attention Layer now. Before we had to build our own custom layers even in Keras. The diffrence between Bahdanou and Loung is the way that the attention weights are calculated. One is additive and the other multiplicative(dot product) respectively. Both of these models the key and the value are same. For more advanced transformer techniques refer to BERT or DistilBERT. We also have various variations in BERT.

$\endgroup$
1
  • $\begingroup$ Of course we can build our code. But wondering why keras not provided the very much required code over years? Or have they actually provided all by just those 3 layers? $\endgroup$ Jun 23 '21 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.