Does multi-head attention remove the need for self-attention?

The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the encoder and decoder still be necessary?

Self-attention means X pays attention to X, as opposed to "normal" attention where X pays attention to Y.

Multi-head attention is as opposed to single-head attention. You can choose to use multi- or single-head attention equally for self-attention and for normal-attention.

Masking X and/or Y is a third independent aspect of a design.

In a Transformer encoder there is only self-attention and feed-forward networks (FFNs). Without the self-attention aspect the layers are reduced to just being FFNs. The FFNs operate only on one item, so without the self-attention they would never discover relationships between items. In a text application that means it would never discover the relationships between words. In a time series analysis it would never discover relationships across time.

The decoder uses both self-attention and normal attention. If there is no relationship across words/time in the output of the model, then maybe you could get rid of the self-attention part. But, if the transformer is a useful model for your data, that is unlikely to be the case.

So if you wanted a clear-cut yes/no answer (to if it removes the need for self-attention): in the case of the encoder, "no", and in the case of the decoder, "probably not".

UPDATE: Having now read the papers (SepFormer, and the two in comments, TSTNN, and DPTNet) I think the above still applies. Those models only use the encoder part of the transformer.

DPTNet and TSTNN change the first part of the FFN with a RNN or GRU respectively, so I suppose theoretically it can learn about other tokens from that; but the motivation for this is as an alternative to needing positional encodings, and I doubt it is worth the trade-off of no longer being able to process the data in parallel.

• Interesting answer. What I found most confusing was why certain TNN architectures (like SepFormer, TSTNN, and DPTNet) only use multi-head attention in the middle layer between the encoder and decoder, and both the encoder and decoder use very few convolutions. Is there any reason behind this? – J. Herrera Apr 13 at 0:31
• @J.Herrera I'm not too aware of transformers applied to speech analysis; you've motivated me to add those three papers to my to-read list. – Darren Cook Apr 13 at 6:50
• It was just something that I’ve noticed upon reading those papers. Maybe I should also try implementing it myself to see if it has any effects on the performance. Thanks for your answer! – J. Herrera Apr 13 at 7:58