I am new to NLP field and wanted to apply attention model in one of my projects. I have LSTM model to train, and concatenate some external data sources though attention mechanism.

The hidden state size of my LSTM : 128

The vectors I am taking are : 3 , for 120 row values, therefore (120 x 3)

What I am expecting from attention mechanism to take weighted sum of these 120 values at a given LSTM timesteps. The additive Bahdanau takes hidden states h1 and h2, and map them to matrices W1 and W2. However, if these hidden states are different size, would it not cause redundant zero values in these matrices? On the other hand, What I would expect from an encoder-decoder type of network to have different size of encoder and decoder hidden states.

I hope I could explain the problem clear enough. I think attention mechanism would be useful to integrate other sources during training phrase, however I did not see any practical application in the literature yet.

Thank you so much.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.