6
$\begingroup$

In the paper, "Hierarchical Attention Networks for Document Classification", the authors use attention to compose words to sentences and then sentences to a document representation. They make use of a context vector $u_w$ to compute the attention weights for the annotation of each word in the sentence.

The paper states in section 2.2, "The context vector $u_w$ can be seen as a high level representation of a fixed query “what is the informative word” over the words. The word context vector $u_w$ is randomly initialized and jointly learned during the training process."

This implies that the context vector is independent of the sentence input. It remains the same for every sentence and is learned as a parameter of the neural network. If this is the case, how will $u_w$ accurately provide attention weights for words in a random sentence, given that sentences are so diverse in meaning.

I do not understand the workings of $u_w$, since it is independent of the sentence input.

Can someone explain?

$\endgroup$
2
  • 1
    $\begingroup$ you're completely right, and this is a major limitation of the model. The context vector only represents the 'ideal word', on average, which is the same for all examples... $\endgroup$
    – Antoine
    Aug 31, 2018 at 10:20
  • $\begingroup$ I agree with the question too. The query vector (Word and Sentence Context Vector) would be fixed during the training. $\endgroup$ Dec 7, 2018 at 11:52

3 Answers 3

4
$\begingroup$

I can not completely agree with this previous answer.

The context vector $u_w$ is not computed by the Eq. 5 and 6. As stated in the paper, $u_w$ is randomly initialised and learned during the training. Instead, it is $u_{it}$ that is computed by Eq. 5 and 6, using the (again) random initialised $W_w$ and $b_w$.

$\endgroup$
1
  • 1
    $\begingroup$ Here you can find an implementation of the attention mechanism applied at the word level coming from the discussed paper. It is an implementation in Pytorch made by the Facebook group: github.com/facebookresearch/InferSent/blob/master/models.py Have a look at line 471, class "InnerAttentionNAACLEncoder". In this case too, $u_w$ is not the result of any projection. I opened an issue to get more clarifications. $\endgroup$
    – Gabrer
    Sep 25, 2018 at 14:54
2
$\begingroup$

Gabrer is right, the way the research paper has used the context vector is not generalizable and hence the limitation. Look at a similar approach from facebook (https://arxiv.org/pdf/1705.02364.pdf). They insist on using different context vectors to capture different topics/distribution of words. So, in a way, we are not using a single context but multiple context vectors and model would assign the right one based on the input.

$\endgroup$
0
$\begingroup$

It is the layer weights that are learnt. Layer weights will be the same for all sentences. So every sentence is 'transformed' using these hardcoded weights to give out a single word context which best represents the meaning of that entire sentence. If you think of it that way, then in every NN, the layer weights are hardcoded, nonetheless they do a good job of transforming different values of input

This does not mean that the output word representing the sentence is the same for all sentences.

Second point - the context vector here is slightly different from the context vector we associate with Attention mechanism (weighted sum of all states). The semantics are slightly different

Lastly - Yes it is possible that this way to determine 'context' is not the most optimum way. Nonetheless when the paper came out it did present a refreshing point of view. More importantly the paper was not much about attention in itself - but more about classification of large corpus of documents using Attention

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.