How many definitions of attention are commonly employed for Deep Learning tasks?

That's what I've encountered up to now:

  • Self-attention
  • Bahdanau
  • Luong
  • Multi-Head (used in Transformers)

Could you provide formal explanations of each of these (and others in case the list is incomplete) and tips on when to prefer one to the other?

And what is the history of Attention models? How did they develop through time, and how did they improve previous formulations?

  • $\begingroup$ Your question is quite broad, can you specify the type of answer you are looking for? $\endgroup$ – Valentin Calomme Dec 4 '19 at 14:25

As far as I know, one of the first "successful" use of attention came in 2014 by Badhanau, Cho, and Bengio with their paper "Neural Machine Translation by Jointly Learning to Align and Translate"

Since then, attention has been refined and improved in multiple papers. Attention is All You Need and the ever-growing use of transformers has made it even more popular.

However, I think it's crucial to understand that even though it has shown incredible value, it is ultimately just a mechanism that weights inputs in an attempt to pool them together in a smarter way than max or averaging pooling.


Not the answer you're looking for? Browse other questions tagged or ask your own question.