enter image description here

Which parts of a transformer share weights, like, do all the encoders share the same weight or all the decoders share the same weights?

  • $\begingroup$ What do you mean by encoders and decoders? In a standard Transformer architecture, there is one encoder and encoder. Do you mean if encoder/decoder layers share parameters? They do not share parameters. Usually, embeddings are shared between the encoder and the decoder. $\endgroup$
    – Jindřich
    Nov 4, 2020 at 8:44
  • $\begingroup$ @Jindřich I was referring to the transformer model which has multiple encoders and decoders, that is why I said 'encoders' and not 'encoder'... My question is that are there any weight sharing between the encoders themselves... and what are the other different parts of the transformer that share their weights? $\endgroup$
    – user105282
    Nov 4, 2020 at 11:01
  • 1
    $\begingroup$ Then I guess it depends on the task. If the encoders encode the same type of data (e.g., sentences in one language), then they should share the weights, if they encode conceptually different data (e.g., text and representations of image regions), then there is no reason to share anything. $\endgroup$
    – Jindřich
    Nov 4, 2020 at 13:17

1 Answer 1


Updated answer

The Transformer model has 2 parts: encoder and decoder.

Both encoder and decoder are comprised of a sequence of attention layers. Each layer is comprised of a combination of multi-head attention blocks, positional feedforward layers, normalization, and residual connections.

The attention layers from the encoder and decoder are slightly different: the encoder only has self-attention blocks while the decoder alternates self-attention with encoder attention blocks. Also, the self-attention blocks are masked to ensure causal predictions (i.e. the prediction of token N only depends on the previous N - 1 tokens, and not on the future ones).

In the blocks in the attention layers no parameters are shared.

Apart from that, there are other trainable elements that we have not mentioned: the source and target embeddings and the linear projection in the decoder before the final softmax.

The source and target embeddings can be shared or not. This is a design decision. They are normally shared if the token vocabulary is shared, and this normally happens when you have languages with the same script (i.e. the Latin alphabet). If your source and target languages are e.g. English and Chinese, which have different writing systems, your token vocabularies would probably not be shared, and then the embeddings wouldn't be shared either.

Then, the linear projection before the softmax can be shared with the target embedding matrix. This is also a design decision. It is frequent to share them.

Finally, the positional embeddings (which can be either trainable or pre-computed) are shared for the source and target languages.

Old Answer (mistakenly addressing multi-encoder/decoder transformers)

In general, parameters are not shared in multi-encoder/multi-decoder transformer architectures, in the sense that each encoder/decoder has its own parameters. This is because sharing parameters defeats the very purpose of having multiple encoders/decoders: if you have two encoders that share parameters, they are effectively the same encoder.

This, of course, is not a rule, and there may be cases where it makes sense to share some or all encoder/decoder parameters. An example could be a scenario where two sentences in a certain language (or two very similar languages, like dialects) are received as input. It may be beneficial to share parameters between both encoders if the amount of data in one of them is not enough.

Background: multi-encoder and/or multi-decoder Transformers are generally not used for "simple" sequence to sequence tasks like machine translation. There are certain special cases where this kind of architectures have been used with different purposes, e.g.:

  • Automatic Post-Editing (APE) consists of having a primary machine translation system which output is refined with a secondary translation system that corrects the errors from the primary system. The secondary system normally receives as inputs both the original source sentence and the translation from the primary system and generates the fixed translation as output. An option for this kind of scenario is having a dual-encoder single-decoder transformer, where the output of the encoders are both fed to the decoder, either concatenating them (example) or injecting them to different attention blocks (example). In this scenario, normally there is no parameter sharing at all.

  • Multimodal translation: in this case, we receive different data modalities (e.g. speech and text), so we need modality-specific architectures, so normally there is no parameter sharing.

Note that here I understand that the multiplicity in encoders and/or decoders happens both at training and inference time. There are other cases where such a multiplicity happens only at training time (as in multi-task learning), but at inference time only one encoder and decoder are selected for each case, e.g.:

  • Multilingual translation: while many multilingual MT systems have a single model for all supported languages (normally supplying special tokens that identify the source/target language), it is possible to have language-specific encoders or decoders. You can see examples of this kind of architecture in works like this or this. In this scenario, each encoder/decoder has its own parameters and there is no sharing.
  • $\begingroup$ But I think that transformers can be used for machine translation.... That is what they did in their 'attention is all you need' paper. $\endgroup$
    – user105282
    Dec 7, 2020 at 15:03
  • $\begingroup$ Yes, but in normal machine translation, there is one encoder and one decoder, not multiple ones, which is what you explicitly asked for, as clarified in your comment: I was referring to the transformer model which has multiple encoders and decoders, [...]. $\endgroup$
    – noe
    Dec 7, 2020 at 15:12
  • $\begingroup$ I understand that there has been a misunderstanding here. I think you are not referring to multi-encoder/decoder transformers, but normal transformers and that, when you say "has multiple encoders and decoders", you actually mean that in the Transformer encoder and decoder there are multiple layers. Is that right? $\endgroup$
    – noe
    Dec 7, 2020 at 15:18
  • 1
    $\begingroup$ Just wanted to say thank you for the brilliant answer $\endgroup$
    – user105282
    Dec 7, 2020 at 16:00
  • 1
    $\begingroup$ Text in neural networks is represented as sequences of discrete elements called "tokens". Tokens can be defined either at word-level, character-level or subword-level (e.g. byte-pair encoding). The list of all possible tokens is called the "vocabulary". If you decide to use a word-level vocabulary, then the vocabulary will be the list of all supported words by your model. You can have separate source and target language vocabularies, or share them in a single combined vocabulary. $\endgroup$
    – noe
    Dec 7, 2020 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.