0
$\begingroup$

fairseq includes an implementation of a non autoregressive transformer - which (as much as I understand) means that the whole output sequence is generated in a single forward run (in contrast to autoregresive models where each forward run predicts the next token from the input and the previous predicted tokens)

However, from the code it appears that the models still expects the previous tokens as input:

def forward(self, src_tokens, src_lengths, prev_output_tokens, tgt_tokens, **kwargs)

https://github.com/pytorch/fairseq/blob/master/fairseq/models/nat/nonautoregressive_transformer.py#L78

$\endgroup$
1
$\begingroup$

It is there to maintain consistency with the signature of the forward method of the base class TransformerModel and therefore allow to use it in place of any other autoregressive transformer, but it is actually not used. The same happens in the model decoder.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.