1
$\begingroup$

I am looking at the Luong paper on Attention models and global attention. I understand how the alignment vector is computed from a dot product of the encoder hidden state and the decoder hidden state. So that all makes sense.

My question is, what should the dimensions of the alignment score tensor be? If I have my data in batches, I basically compute a single score for each timestep in the hidden state, right. So should the alignment vector be of dimension [sequence length, 1], or something like that? I would then softmax this alignment vector and multiply it by every element in the batch to compute the context vector, right.

Again, my key question is what the dimensions of the alignment score vector or tensor should be. Thanks.

$\endgroup$
1
$\begingroup$

You answered yourself [sequence length, 1] is correct assuming you work with a single sentence. (Or actually, the 1 dimension depends on implementation.)

In practice, the data is typically batched, so it will be [batch, sequence length 1]. This can be element-wise multiplied with the encoder states of dimension [batch, sequence length, hidden size] and summed in the middle dimension to get a context vector of dimension [batch, hidden_size].

Note also that if the sentences do not have the same length, you need to deal with padding, i.e., set the padded positions to -inf before softmax.

$\endgroup$
1
  • $\begingroup$ thanks so much for your answer. Glad to know I was on the right track ;). I was getting confused because I could not tell whether there was a different alignment vector by batch element, or if it was just a single alignment vector repeated batch number of times. But you confirmed the later. Thanks again. $\endgroup$
    – krishnab
    Dec 29 '20 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.