1
$\begingroup$

What is the difference between stacked lstm vs attention mechanism in LSTM? It seem to me that both produce the same context vector at the end.

EDIT: From suggestion by @shepan6,

  • the difference in architecture of stacked ltsm and attention mechanism
  • not too sure i am right here, it seem that both used the notion of being able to select important features to construct the final context vector.
  • if the answer to above is yes, am i right to say the goal is the same for both, just that the mechanism is different

Thanks

$\endgroup$
1
  • $\begingroup$ I could clarify the differences in network architecture, but I am not sure if that would answer your question. Any chance you edit your post to make explicit what differences you are referring to (architecture, output, etc.)? $\endgroup$
    – shepan6
    Jul 8, 2020 at 11:58

1 Answer 1

2
$\begingroup$

Stacked LSTM is a special version of hierachical recurrent neural networks, where hard-wired memory and gating units help long-term preservation of state information.

Hierarchy and recurrence have been explored in many works.

One early example is the Neural Abstraction Pyramid, which introduced recurrent computation to hierarchical convolutional neural networks (a.k.a. deep learning).

It incorporates partial interpretations from a larger-and-larger context via horizontal and vertical feedback loops in order to iteratively refine an interpretation. It was trained using Backpropagation Through Time to solve several computer vision tasks, such as image denoising, superresolution, and object detection. The recurrent computation is also well suited for maintaining hierarchical state information when processing image sequences.

http://www.ais.uni-bonn.de/books/LNCS2766.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.