0
$\begingroup$

For LLM decoder, how exactly is the K, Q, V for each decoding step?

Say my input prompt is "today is a" (good day).

At t= 0 (generation step 0): K, Q, V are the projections of the sequence ("today is a") Then say the next token generated is "good"

At t= 1(generation step 1): Which one is true:

  • K, Q, V are the projections of the sequence ("today is a good")

OR

  • K, Q, are the projections of the sequence ("today is a") , V is the projection of sequence ("good")?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

At t=1: K, Q, V are the projections of the sequence "today is a good".

However, given that the computations of the first tokens were already done in the previous step, usually, there is some sort of caching mechanism to save their repeated computation in the following steps.

Improve this answer
$\endgroup$
2
  • $\begingroup$ thanks for the comment, would you please explain (at a high level), how would the caching mechanism work? $\endgroup$
    – wrek
    Commented May 12, 2023 at 18:27
  • $\begingroup$ Keeping in memory the tensors with the hidden states of all layers from the previous tokens and using them in the computation of the attention heads instead of computing them. $\endgroup$
    – noe
    Commented May 12, 2023 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.