# A question about contextual embeddings in the decoder only transformer architecture (gpt)

I am reading up on the decoder only architecture

Relevant excerpts:

We can use any model that maps token sequences into contextual embeddings (e.g., LSTMs, Transformers):

$$\phi : V^L \to R^{d \times L}$$

Recall that an autoregressive language model defines a conditional distribution:

$$p(x_i∣x_{1:i−1})$$

We define it as follows:

• Map $$x_{1:i-1}$$ to contextual embeddings $$\phi(x_{1:i-1})$$
• Apply an embedding matrix $$E \in R^{V×d}$$to obtain scores for each token $$E\phi(x_{1:i-1})_{i-1}$$
• Exponentiate and normalize it to produce the distribution over xi

Succinctly: $$p(x_{i+1} \mid x_{1:i}) = softmax(E \phi(x_{1:i})_i)$$

Questions:

1. What is the meaning of the second subscript on $$\phi$$ in $$E \phi (x_{1:i-1})_{i-1}$$
2. I think $$softmax(E \phi(x_{1:i})_i)$$ just takes the dot product of the context embedding for the word at the position $$i$$ with the embeddings $$E$$ of the entire vocabulary. This means that for the word at $$i$$, we are basically just trying to learn the context embeddings as something that would essentially be equal to the embedding of the next token (if the model learns perfectly) in $$E$$. Why is that the case? Should there not be a feed-forward between the final context embeddings and the embedding $$E$$ for the next token and then the similarity should be checked? This way, in the best case scenario, the contextual embeddings learned would just be the vector that appeared for the word at $$i$$ in $$E$$. Please help me understand how we are not just asking the contextual embedding to be equal to the embedding $$E$$ for the word at $$i$$?
1. The second subscript refers to the position within the sequence of the contextual embedding being computed. The inner subscript range refers to what input tokens are usted as input for the previously mentioned computation. That is, to generate each position, the transformer decoder can use the input tokens from the same position and before. Take into account that the input of the transformer decoder is shifted one position to the right by prepending a <bos> (beginning of sequence) token, which implies that, speaking about token positions within the original unshifted sequence, each prediction depends on the previous ones (and not on the current one or the following ones, which are not available at inference time).