Let's take an example sentence for translation:
I am going to my home and play with toy house.
For translating 'home', as per my understanding, Query will be 'house's embedding vector, Key will be each of the token's vectors i.e size 11 (word based token).
Then we take cosine to find the similarity.
Why? Why similarity? 'Home' and 'House' shall be most similar, but that doesn't play a part in the translation. Rather, probably 'toy' is more important here from translation perspective.
Remember that the embedding representations change as the input goes through the model.
For the very first input (after embedding tokens), you have an embedding sequence where each item represents a single token.
But then you do an attention step to update. In the attention step, each token is enriched with information from each other token (or each previous token for causal language models).
This means that as the input moves through the model, the representations for each item in the sequence are enriched with information from other items in the sequence, allowing the model to learn relationships and representations that take into account the full content of the sequence. This is also enhanced by the QKV projection layers.
By the end of the model, the relationships between items in the sequence will be very different from the relationships at the start. You might find that the output embedding for house is mainly attending to toy and play, while the output embedding for home is attending to my and going.
The point of having multiple layers is to extract context based representations that go beyond simple similarity to solve whatever the training task is.
the purpose of this similarity is not to find the most similar words in terms of their meaning, but rather to identify which words in the input sequence are most relevant for generating each output word.
Let's break this down using the example sentence you provided: "I am going to my home and play with toy house."
When translating the word "home," the query vector is indeed the embedding vector of the word "home." The key vectors, on the other hand, are the embedding vectors of each word in the input sentence. The attention mechanism computes the cosine similarity between the query vector and each key vector to determine how much each input word should contribute to the generation of the output word "home."
In this case, the words "toy" and "house" may have high cosine similarity with "home" due to their semantic relatedness. However, the attention mechanism is not simply looking for the most similar words. Instead, it is trying to determine which words in the input sentence provide the most relevant context for translating "home" correctly.
In the context of the entire sentence, the word "my" is likely to have a high attention score when generating "home," as it indicates possession and helps to disambiguate the meaning of "home" in this specific context. The word "house," although similar to "home," may receive a lower attention score because it is part of a separate phrase ("toy house") that is less directly relevant to the translation of "home."
The similarity between Query and Key is not about finding similar words in the traditional sense (like "home" and "house"). Instead, it's about finding which parts of the input are relevant for encoding each position.
The attention mechanism allows the model to focus on different parts of the input when encoding each word. This helps capture context and long-range dependencies.This is learned during training, so the model figures out what kind of attention patterns are useful for the task.
Your "home" and "toy" Example - when translating "home", the model might indeed pay attention to "toy" if it's relevant. The attention mechanism allows for this flexibility. It's not constrained to only looking at similar words.
Totally agree, Focus on Meaning , context matters.
Solutions I consider which can overcome this :
1. Incorporate Part-of-Speech (POS) Tags - identifies the grammatical role of each word. In this case, recognizing "toy" as an adjective modifying "house" helps understand the context of "playing" in a toy house.
2. Use of Pre-trained models like ( LSTM or Transformers )
3.Instead of just considering the current word ("house"), look at a bigger chunk of the sentence, like a 3-word or 5-word window ("my home" or "going to my home") - Just use a larger context window.
Hope this helps!
