I'm a newbie in NLP and I'm now stuck in GPT. The question I'm struggling with is related to a term 'context vector'enter image description here

It says in the following (sorry that the material provided is written in korean) that U represents a context vector. I searched for both terms 'context window' and 'context vector' Now I understand that a context window means the number of words I will accept as a context of the token. I also searched for 'context vector' and what I found out is that it is calculated by multiplying value parameters of each token to its softmax (attention score) (I hope I am right, feel free to tell me what I'm getting wrong) My question is that if 'context vector' can be calculated by using attention mechanism, then how can it(U) be calculated in advance in GPT since a transformer block including a masked attention layer gets 'UW(e) + W(p)' as its input ?

I know this question might seems really non-essential and out of topics to many great experts here, but I guess it's just making me get stuck in a logical step to understand NLP. enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.