The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the encoder and decoder still be necessary?


Self-attention means X pays attention to X, as opposed to "normal" attention where X pays attention to Y.

Multi-head attention is as opposed to single-head attention. You can choose to use multi- or single-head attention equally for self-attention and for normal-attention.

Masking X and/or Y is a third independent aspect of a design.

In a Transformer encoder there is only self-attention and feed-forward networks (FFNs). Without the self-attention aspect the layers are reduced to just being FFNs. The FFNs operate only on one item, so without the self-attention they would never discover relationships between items. In a text application that means it would never discover the relationships between words. In a time series analysis it would never discover relationships across time.

The decoder uses both self-attention and normal attention. If there is no relationship across words/time in the output of the model, then maybe you could get rid of the self-attention part. But, if the transformer is a useful model for your data, that is unlikely to be the case.

So if you wanted a clear-cut yes/no answer (to if it removes the need for self-attention): in the case of the encoder, "no", and in the case of the decoder, "probably not".

UPDATE: Having now read the papers (SepFormer, and the two in comments, TSTNN, and DPTNet) I think the above still applies. Those models only use the encoder part of the transformer.

DPTNet and TSTNN change the first part of the FFN with a RNN or GRU respectively, so I suppose theoretically it can learn about other tokens from that; but the motivation for this is as an alternative to needing positional encodings, and I doubt it is worth the trade-off of no longer being able to process the data in parallel.

  • $\begingroup$ Interesting answer. What I found most confusing was why certain TNN architectures (like SepFormer, TSTNN, and DPTNet) only use multi-head attention in the middle layer between the encoder and decoder, and both the encoder and decoder use very few convolutions. Is there any reason behind this? $\endgroup$ – J. Herrera Apr 13 at 0:31
  • $\begingroup$ @J.Herrera I'm not too aware of transformers applied to speech analysis; you've motivated me to add those three papers to my to-read list. $\endgroup$ – Darren Cook Apr 13 at 6:50
  • $\begingroup$ It was just something that I’ve noticed upon reading those papers. Maybe I should also try implementing it myself to see if it has any effects on the performance. Thanks for your answer! $\endgroup$ – J. Herrera Apr 13 at 7:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.