In addition, why do we need a FFN in each layer when we already have attention?
Here's a screenshot of the relevant section from Vaswani et al. (2017):
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
A position-wise feedforward layer is just a matrix multiplication plus the addition of a bias vector for each position along the time dimension. You can express this as a 1D convolution of kernel size 1. This was first suggested by Lin et al. (2014), who called it "network in network".
Given that the position-wise feedforward network is the concatenation of 2 feedforward layers, you can express it as two 1D convolutions of kernel size 1.
Convolution is actually applying a sliding window projection over a tensor. With the kernel size of 1, the convolution reduces a to a for loop over the vectors, i.e., each of the vectors gets multiplied by a filter matrix. But this is actually a definition of matrix multiplication which is how feed-forward layers are implemented.