The attention weights are formed through the last hidden state of the LSTM and the feature map from some kind of image encoder (in my case resnet so the features are in the form of 14x14x2048). They first go through the following calculations:
u_hs = self.U(features)
w_ah = self.W(hidden_state)
combined_states = torch.tanh(u_hs + w_ah.unsqueeze(1))
attention_scores = self.A(combined_states) #attention score per pixel in condensed filters (196, 1)
attention_scores = attention_scores.squeeze(2) #squeeze to get (196,) for softmax
alpha = F.softmax(attention_scores, dim=1)
The softmaxxed alphas represent a 14x14 filter that can be plotted to see which part of the image is important to the model.
My question is this: The context vector that is used for the LSTM is calculated using
attention_weights = features * alpha.unsqueeze(2) #deterministic soft attention
attention_weights = attention_weights.sum(dim=1) #sum across i = 1 to 196 to get the attention weights [batch_size, encoder_dim (2048)]
This context vector is of size 1x2048, but why can't the softmaxxed alphas (14x14) in the previous code block be directly passed to the LSTM since they have info about which part of the image is important? THe context vector is for some reason I don't understand the same size as the amount of feature map the encoder produces. Is it because multiplying the alphas with the features AGAIN retains some information about the filters? I tested the model by directly feeding the alphas into the LSTM and the results weren't that different compared to the normal procedure with the context vectors. Btw, I feed the context vector into the LSTM by simply concatenating the context vector and the word embeddings.
This is a question about a paper. I tested my theory and results weren't that different. I need help understanding the theory behind the code.
