0
$\begingroup$

The forward function of PyTorch's Transformer implementation torch.nn.Transformer have a number of masking arguments that are all optional :

forward(src, tgt, src_mask=None, tgt_mask=None, memory_mask=None, src_key_padding_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None)

Parameters:

  • src (Tensor) – the sequence to the encoder (required).
  • tgt (Tensor) – the sequence to the decoder (required).
  • src_mask (Optional[Tensor]) – the additive mask for the src sequence (optional).
  • tgt_mask (Optional[Tensor]) – the additive mask for the tgt sequence (optional).
  • memory_mask (Optional[Tensor]) – the additive mask for the encoder output (optional).
  • src_key_padding_mask (Optional[Tensor]) – the Tensor mask for src keys per batch (optional).
  • tgt_key_padding_mask (Optional[Tensor]) – the Tensor mask for tgt keys per batch (optional).
  • memory_key_padding_mask (Optional[Tensor]) – the Tensor mask for memory keys per batch (optional).

If I just want to use a vanilla Transformer, what masking arguments should I provide? What are the effects if I do not provide them?

This is confusing to me because I found people selectively provide masking arguments.

For example, this one provided everything except memory_mask in TransformerDecoder layer. How can I know whether I should provide it or not?

Another example, this one uses a look-ahead mask (upper triangular) in the encoder part.

And in this example, the author didn't input any mask.

Improve this question
$\endgroup$

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.