# What exactly is the linear layer in the transformer model?

There are linear layers to modify the Query, key and value matrices and one linear layer after the multi head attention as they mention also from here:

Are these linear layers simply dense or fully connected layers? Let's consider the weight matrix Wi Q. Does this represent a dense layer with "Q" nodes? As they are using matrices as input rather than 1D vectors, I am getting a little confused.

As described in the paragraph you referred to in your question $$W^Q$$ is a matrix of dimensions $$d_{model} \times d_k$$, that is, it is a fully connected layer with $$d_k$$ units.
• Yes, for each attention head, the same $W^Q_i$ is applied to all query vectors in $Q$.