I have been working on a Image Captioning model. And read many articles accomplishing it. Some used both attentions interchangably while some did not. And the formulaes differed too. So, I would like to know if Soft Attention and Bahdanau Attention are any different?
$\begingroup$
$\endgroup$
6
asked Sep 12, 2022 at 8:50
-
2$\begingroup$ Normally, "soft" attention is the opposite of "hard" attention. Soft means that the weights are in $[0, 1]$, while in hard attention, weights are either 0 or 1. Therefore, Bahdanau attention is just a type of soft attention. Can you add a reference of what you understand for "soft attention"? $\endgroup$– noeSep 12, 2022 at 11:05
-
$\begingroup$ thanks for the reply @noe, can you please add a reference that shows that Bahdanau is a type/variation of soft? $\endgroup$– Naveen Reddy MarthalaSep 12, 2022 at 13:19
-
$\begingroup$ articles that i had referenced for soft attention: 1) jhui.github.io/2017/03/15/Soft-and-hard-attention 2) medium.com/analytics-vidhya/… 3) medium.com/swlh/… $\endgroup$– Naveen Reddy MarthalaSep 12, 2022 at 13:21
-
$\begingroup$ Articles (1) and (2) use the term "soft attention" in the same sense I mentioned before. Article (3) does not mention soft attention anywhere. $\endgroup$– noeSep 12, 2022 at 15:07
-
$\begingroup$ Article (2) also shows how the Bahdanau attention is soft. $\endgroup$– noeSep 12, 2022 at 15:12
|
Show 1 more comment