How are attention mechanism used in different deep learning algorithms different from the cognitive function attention?


Are you reffering to biological "cognitive function attention"?

If yes than it could be argued its different.

TL;DR Fixed-length Context vector in the encoder decoder architecture could not remember long term dependencies. Attention could learn these long term dependencies, hence have a long-term memory. If we speak about human attention it is usually short spaned (relative).

On the other hand I believe it was Yoshua who first talked in the terms of attention and one could argue that we could implicitly focus (our attention) on some older parts of text and hence extract deeper meaning.

TL;DR TL;DR One can make a case its actually more natural to talk about long-term memory when discussing attention-ML-mechanism

  • $\begingroup$ Yes it is the biological cognitive attention but is it this attention which has been implemented in neural networks or the attention function is altogether different $\endgroup$
    – codeczar
    Dec 17 '19 at 12:41
  • 1
    $\begingroup$ As I said I like to think about it more in terms of cognitive "long-term-memory" rather than attention. But one can make case for both. Also I highly doubt that they were having biological models infront of them and then they tried to model them. Its rather solving one problem after another WHILE trying to take influence from biology. If they match they match. $\endgroup$
    – Noah Weber
    Dec 17 '19 at 12:46
  • $\begingroup$ What is the meaning of "TL;DR" to you? $\endgroup$ Dec 19 '19 at 8:56
  • $\begingroup$ too long; didn't read- summarization $\endgroup$
    – Noah Weber
    Dec 19 '19 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.