0
$\begingroup$

Let's take an example sentence for translation:

I am going to my home and play with toy house.

For translating 'home', as per my understanding, Query will be 'house's embedding vector, Key will be each of the token's vectors i.e size 11 (word based token).

Then we take cosine to find the similarity.

Why? Why similarity? 'Home' and 'House' shall be most similar, but that doesn't play a part in the translation. Rather, probably 'toy' is more important here from translation perspective.

$\endgroup$

3 Answers 3

1
$\begingroup$

Remember that the embedding representations change as the input goes through the model.

For the very first input (after embedding tokens), you have an embedding sequence where each item represents a single token.

But then you do an attention step to update. In the attention step, each token is enriched with information from each other token (or each previous token for causal language models).

This means that as the input moves through the model, the representations for each item in the sequence are enriched with information from other items in the sequence, allowing the model to learn relationships and representations that take into account the full content of the sequence. This is also enhanced by the QKV projection layers.

By the end of the model, the relationships between items in the sequence will be very different from the relationships at the start. You might find that the output embedding for house is mainly attending to toy and play, while the output embedding for home is attending to my and going.

The point of having multiple layers is to extract context based representations that go beyond simple similarity to solve whatever the training task is.

$\endgroup$
2
  • $\begingroup$ Is that the reason we don't use any pretrained embeddings ? $\endgroup$
    – Pratham
    Commented May 7 at 18:55
  • $\begingroup$ You could initialize a model with pretrained embeddings but usually embeddings are trained in-line with the rest of the model $\endgroup$
    – Karl
    Commented May 7 at 19:07
1
$\begingroup$

the purpose of this similarity is not to find the most similar words in terms of their meaning, but rather to identify which words in the input sequence are most relevant for generating each output word.

Let's break this down using the example sentence you provided: "I am going to my home and play with toy house."

When translating the word "home," the query vector is indeed the embedding vector of the word "home." The key vectors, on the other hand, are the embedding vectors of each word in the input sentence. The attention mechanism computes the cosine similarity between the query vector and each key vector to determine how much each input word should contribute to the generation of the output word "home."

In this case, the words "toy" and "house" may have high cosine similarity with "home" due to their semantic relatedness. However, the attention mechanism is not simply looking for the most similar words. Instead, it is trying to determine which words in the input sentence provide the most relevant context for translating "home" correctly.

In the context of the entire sentence, the word "my" is likely to have a high attention score when generating "home," as it indicates possession and helps to disambiguate the meaning of "home" in this specific context. The word "house," although similar to "home," may receive a lower attention score because it is part of a separate phrase ("toy house") that is less directly relevant to the translation of "home."

$\endgroup$
1
  • $\begingroup$ I understand the intuition, that attention uses cosine to determine how much each value should contribute. But my question is how can cosine do that. $\endgroup$
    – Pratham
    Commented May 11 at 4:36
0
$\begingroup$

Totally agree, Focus on Meaning , context matters.

Solutions I consider which can overcome this :

1. Incorporate Part-of-Speech (POS) Tags - identifies the grammatical role of each word. In this case, recognizing "toy" as an adjective modifying "house" helps understand the context of "playing" in a toy house.
2. Use of Pre-trained models like ( LSTM or Transformers )
3.Instead of just considering the current word ("house"), look at a bigger chunk of the sentence, like a 3-word or 5-word window ("my home" or "going to my home") - Just use a larger context window.

Hope this helps!

$\endgroup$
2
  • $\begingroup$ Is the LLM generated answer, Or just formatting issue? $\endgroup$
    – Pratham
    Commented May 7 at 6:36
  • $\begingroup$ sorry, I couldn't understand your comment, can you elaborate $\endgroup$
    – priyanka
    Commented May 7 at 6:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.