I am reading up on the decoder only architecture

Relevant excerpts:

We can use any model that maps token sequences into contextual embeddings (e.g., LSTMs, Transformers):

$$\phi : V^L \to R^{d \times L}$$

Recall that an autoregressive language model defines a conditional distribution:


We define it as follows:

  • Map $x_{1:i-1}$ to contextual embeddings $\phi(x_{1:i-1})$
  • Apply an embedding matrix $E \in R^{V×d}$to obtain scores for each token $E\phi(x_{1:i-1})_{i-1}$
  • Exponentiate and normalize it to produce the distribution over xi

Succinctly: $$p(x_{i+1} \mid x_{1:i}) = softmax(E \phi(x_{1:i})_i)$$


  1. What is the meaning of the second subscript on $\phi$ in $ E \phi (x_{1:i-1})_{i-1}$
  2. I think $softmax(E \phi(x_{1:i})_i)$ just takes the dot product of the context embedding for the word at the position $i$ with the embeddings $E$ of the entire vocabulary. This means that for the word at $i$, we are basically just trying to learn the context embeddings as something that would essentially be equal to the embedding of the next token (if the model learns perfectly) in $E$. Why is that the case? Should there not be a feed-forward between the final context embeddings and the embedding $E$ for the next token and then the similarity should be checked? This way, in the best case scenario, the contextual embeddings learned would just be the vector that appeared for the word at $i$ in $E$. Please help me understand how we are not just asking the contextual embedding to be equal to the embedding $E$ for the word at $i$?
  • $\begingroup$ has my answer clarified your doubts? $\endgroup$
    – noe
    Commented Oct 19, 2023 at 15:25
  • 1
    $\begingroup$ Yes. Most definitely. Thank you for your valuable time and knowledge :) $\endgroup$ Commented Oct 19, 2023 at 15:36

1 Answer 1

  1. The second subscript refers to the position within the sequence of the contextual embedding being computed. The inner subscript range refers to what input tokens are usted as input for the previously mentioned computation. That is, to generate each position, the transformer decoder can use the input tokens from the same position and before. Take into account that the input of the transformer decoder is shifted one position to the right by prepending a <bos> (beginning of sequence) token, which implies that, speaking about token positions within the original unshifted sequence, each prediction depends on the previous ones (and not on the current one or the following ones, which are not available at inference time).

  2. Note that Transformer decoders are meant for generating text. At each position, we are predicting the next token. The final multiplication by the embedding matrix simply computes a distance between the contextual embedding and each of the embedding vectors, and then the softmax normalizes such distances into probabilities. The closer the contextual embedding to one of the token embeddings, the more probable to be the next token.

And, surely enough, if you use a Transformer decoder specifically to get token representations, you should take the representation of the previous position. However, Transformer decoders are seldom used to compute token representations because their computation only depends only on the previous tokens. Transformer encoders are the norm when we want to compute token representations.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.