Having gone through articles.
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Attention in Neural Networks
- Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)
- How does attention mechanism learn?
- A Comprehensive Guide to Attention Mechanism in Deep Learning for Everyone
- Lecture 13: Attention
However, still not sure of the basic mechanism of attention. I see
hidden state and diagrams, and the matrix out of blue.
As a first step, please confirm if below is a correct understanding.
If asked, "Related with crime, cocaine, police, smuggle but not with honest, legitimate. Who is this?". Drug-dealer can pop up. The correlations from drug-dealer to (crime, cocaine, police, smuggle, ...) can be quantified as a vector. Some people have built such vectors by going through wikipedia article sentences, BBC news articles, etc. A vector possibly would be a weight in DNN.
When a NLP parser finds
smuggle in a sentence, those correlation vectors help to find which words are more strongly correlated. Is this the basic mechanism of attention, which is, the correlation vectors in a nutshell?
For image to caption, an object detection extracts (frisbee, girl, hair, cloths, ...) and those vectors tell that
girl is the word to pay attention to for
For translation, e.g. English to Spanish, I suppose there are two sets of vectors. One for English words and the other for Spanish, and association between a English word to a Spanish word is trained as a DNN model where training input is (english-vector, spanish-vector) and someone has prepared labels. Is this correct?
PS. A kind request not to list math formulas, or similar diagrams from the articles. The formulas should be the projections of solid intuitions that even a 6 grader can get. I appreciate such intuitions explained in layman's term as the answers.