I have been working on a Image Captioning model. And read many articles accomplishing it. Some used both attentions interchangably while some did not. And the formulaes differed too. So, I would like to know if Soft Attention and Bahdanau Attention are any different?

  • 2
    $\begingroup$ Normally, "soft" attention is the opposite of "hard" attention. Soft means that the weights are in $[0, 1]$, while in hard attention, weights are either 0 or 1. Therefore, Bahdanau attention is just a type of soft attention. Can you add a reference of what you understand for "soft attention"? $\endgroup$
    – noe
    Sep 12, 2022 at 11:05
  • $\begingroup$ thanks for the reply @noe, can you please add a reference that shows that Bahdanau is a type/variation of soft? $\endgroup$ Sep 12, 2022 at 13:19
  • $\begingroup$ articles that i had referenced for soft attention: 1) jhui.github.io/2017/03/15/Soft-and-hard-attention 2) medium.com/analytics-vidhya/… 3) medium.com/swlh/… $\endgroup$ Sep 12, 2022 at 13:21
  • $\begingroup$ Articles (1) and (2) use the term "soft attention" in the same sense I mentioned before. Article (3) does not mention soft attention anywhere. $\endgroup$
    – noe
    Sep 12, 2022 at 15:07
  • $\begingroup$ Article (2) also shows how the Bahdanau attention is soft. $\endgroup$
    – noe
    Sep 12, 2022 at 15:12


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.