Does multi-head attention remove the need for self-attention?

Question

The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the encoder and decoder still be necessary?

Answer 1

Self-attention means X pays attention to X, as opposed to "normal" attention where X pays attention to Y.

Multi-head attention is as opposed to single-head attention. You can choose to use multi- or single-head attention equally for self-attention and for normal-attention.

Masking X and/or Y is a third independent aspect of a design.

In a Transformer encoder there is only self-attention and feed-forward networks (FFNs). Without the self-attention aspect the layers are reduced to just being FFNs. The FFNs operate only on one item, so without the self-attention they would never discover relationships between items. In a text application that means it would never discover the relationships between words. In a time series analysis it would never discover relationships across time.

The decoder uses both self-attention and normal attention. If there is no relationship across words/time in the output of the model, then maybe you could get rid of the self-attention part. But, if the transformer is a useful model for your data, that is unlikely to be the case.

So if you wanted a clear-cut yes/no answer (to if it removes the need for self-attention): in the case of the encoder, "no", and in the case of the decoder, "probably not".

UPDATE: Having now read the papers (SepFormer, and the two in comments, TSTNN, and DPTNet) I think the above still applies. Those models only use the encoder part of the transformer.

DPTNet and TSTNN change the first part of the FFN with a RNN or GRU respectively, so I suppose theoretically it can learn about other tokens from that; but the motivation for this is as an alternative to needing positional encodings, and I doubt it is worth the trade-off of no longer being able to process the data in parallel.