How many definitions of attention are commonly employed for Deep Learning tasks?
That's what I've encountered up to now:
- Multi-Head (used in Transformers)
Could you provide formal explanations of each of these (and others in case the list is incomplete) and tips on when to prefer one to the other?
And what is the history of Attention models? How did they develop through time, and how did they improve previous formulations?