Attention using Context Vector: Hierarchical Attention Networks for Document Classification

Question

In the paper, "Hierarchical Attention Networks for Document Classification", the authors use attention to compose words to sentences and then sentences to a document representation. They make use of a context vector $u_w$ to compute the attention weights for the annotation of each word in the sentence.

The paper states in section 2.2, "The context vector $u_w$ can be seen as a high level representation of a fixed query “what is the informative word” over the words. The word context vector $u_w$ is randomly initialized and jointly learned during the training process."

This implies that the context vector is independent of the sentence input. It remains the same for every sentence and is learned as a parameter of the neural network. If this is the case, how will $u_w$ accurately provide attention weights for words in a random sentence, given that sentences are so diverse in meaning.

I do not understand the workings of $u_w$, since it is independent of the sentence input.

Can someone explain?

Answer 1

I can not completely agree with this previous answer.

The context vector $u_w$ is not computed by the Eq. 5 and 6. As stated in the paper, $u_w$ is randomly initialised and learned during the training. Instead, it is $u_{it}$ that is computed by Eq. 5 and 6, using the (again) random initialised $W_w$ and $b_w$.

Answer 2

Gabrer is right, the way the research paper has used the context vector is not generalizable and hence the limitation. Look at a similar approach from facebook (https://arxiv.org/pdf/1705.02364.pdf). They insist on using different context vectors to capture different topics/distribution of words. So, in a way, we are not using a single context but multiple context vectors and model would assign the right one based on the input.

Answer 3

It is the layer weights that are learnt. Layer weights will be the same for all sentences. So every sentence is 'transformed' using these hardcoded weights to give out a single word context which best represents the meaning of that entire sentence. If you think of it that way, then in every NN, the layer weights are hardcoded, nonetheless they do a good job of transforming different values of input

This does not mean that the output word representing the sentence is the same for all sentences.

Second point - the context vector here is slightly different from the context vector we associate with Attention mechanism (weighted sum of all states). The semantics are slightly different

Lastly - Yes it is possible that this way to determine 'context' is not the most optimum way. Nonetheless when the paper came out it did present a refreshing point of view. More importantly the paper was not much about attention in itself - but more about classification of large corpus of documents using Attention