In the past days, I read up on the theory behind attention, when to apply it and what types there are. I think I have a decent first understanding of the concept, but now I would like to apply some of the insights I got to my own project and I find myself stuck with the implementation of attention in TF. (Quick Link to TF Attention)
The attention layer requires me to provide at least the queries and values. Correct me if I am wrong already, but this is my idea of what they are:
- Queries: These are the hidden states of my decoder
- Values: These are the hidden states of my encoder
So far so good. The thing I am struggling with is the fact that I have no idea where the hidden states of my decoder might come from. I would like to implement a self-attention mechanism. So my decoder hidden states are generated dynamically and I cannot know them before actually applying the attention layer. The example provided in the docs was not helpful for me, because it focused on a problem where I already have some query sequence.
Apart from whether the mentioned TF attention layer is applicable for self-attention, how do I interpret the different inputs?