# What is the "Extract" token and how is the final Linear layer applied in GPT?

In the manuscript of GPT, the authors have given the following image:

Questions:

1. What is the final "Extract" (token?)? Is it the "END" token?
2. How is the final linear layer applied? We would get a tensor of shape (N,L,D) out of the transformer (tranformer's final layer would itself be a pointwise feedforward) where N is the number of samples, L is the sequence length and D is the output dimension for every token. Taking just one sample it would be (1,L,D). How is the linear layer applied on top of this? The output of the final linear layer would have to be a vector with dimensions equal to the number of decisions at hand (n in case of n-way-classifier, vocabulary size in case of cloze task etc)

All transformations include adding randomly initialized start and end tokens $$(\langle s \rangle, \langle e \rangle)$$.
2. The way they apply the final linear layer is described in section 3.2 of the original paper and consists of taking the output of the last Transformer layer $$l$$ at the last position ($$h^m_l$$ in the paper) and use it as input for the linear layer: $$P(y|x^1, ..., x^m) = \mathit{softmax}(h^m_l W_y)$$