1
$\begingroup$

In addition, why do we need a FFN in each layer when we already have attention?

Here's a screenshot of the relevant section from Vaswani et al. (2017):

eScreenshot of relevant section from Vaswani et al. (2017)

$\endgroup$
0
$\begingroup$

A position-wise feedforward layer is just a matrix multiplication plus the addition of a bias vector for each position along the time dimension. You can express this as a 1D convolution of kernel size 1. This was first suggested by Lin et al. (2014), who called it "network in network".

Given that the position-wise feedforward network is the concatenation of 2 feedforward layers, you can express it as two 1D convolutions of kernel size 1.

$\endgroup$
0
$\begingroup$

Convolution is actually applying a sliding window projection over a tensor. With the kernel size of 1, the convolution reduces a to a for loop over the vectors, i.e., each of the vectors gets multiplied by a filter matrix. But this is actually a definition of matrix multiplication which is how feed-forward layers are implemented.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.