1
$\begingroup$

So I’ve read in Attention is All You Need that Transformers remove the need for recurrence and convolutions entirely. However, I’ve seen some TNNs (such as SepFormer, DPTNet, and TSTNN) that still utilize convolutions. Is there any particular reason for this? Doesn’t that defeat the purpose of Transformers?

$\endgroup$
2
$\begingroup$

We find some justifications in the Conformer paper:

  • Convolutions are better than Transformers at detecting fine-grained patterns:

    While Transformers are good at modeling long-range global context, they are less capable to extract fine-grained local feature patterns. Convolution neural networks (CNNs), on the other hand, exploit local information and are used as the de-facto computational block in vision.

  • Together, they Transformers and convolutions work better than separately:

    Recent works have shown that combining convolution and self-attention improves over using them individually [14]. Together, they are able to learn both position-wise local features, and use content-based global interactions.

$\endgroup$
1
  • $\begingroup$ Thanks for your answer! I haven’t come across this paper. Will definitely read it. $\endgroup$ – J. Herrera Apr 15 at 1:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.