I am working on sequence-to-sequence tasks where the input is an n-length sequence of discrete values from a finite set S (say {x | x is a non-negative integer less than 10}). An example input sequence of length 5 is: 1 8 3 5 2.

The output is supposed to be some length preserving transformation of the input sequence (say reverse, shift, etc.). To be explicit, the tokens of the output sequence also come from the same set as the input sequence. For example, for the input sequence above, the reverse transformation produces the output sequence: 2 5 3 8 1.

I want the model to predict the output tokens exactly, so the task is closer to classification than regression. However, since the output is a sequence, we need to predict multiple classes (as many as the input length) for each input sequence.

I searched for references but could not find a similar setting. Please link some suitable references you are aware of that may be helpful. I have the following questions for my use case:

  1. What changes are needed such that the model works on discrete sequences as defined above?
  2. What loss function would be appropriate for my use case?

For 1), one might change the input sequence such that each token is replaced by an embedding vector (learned or fixed) and input that to the model. For the prediction, I was thinking of ensuring that the model produces a n x k length output (n = sequence length; k = |S| or the vocab size) and then using each of these n vectors to make a class prediction (from k classes).

For 2), the loss could be a sum of n cross-entropy losses corresponding to the n classifications.

Please help me with better answers to these two questions.

Thank you.

Edit: My setup is encoder-only (non-autoregressive prediction). Please account for this while answering the questions by suggesting approaches that are in line with the setup, if possible.

  • $\begingroup$ so you want to predict a permutation of the original sequence? $\endgroup$
    – alexmolas
    Feb 13, 2023 at 9:18
  • $\begingroup$ @alexmolas Not always a permutation. Only length preservation is guaranteed. For instance, the input sequence tokens could be elements of a finite group and the k-th output token is defined as the product of k input tokens. In this case, the O/P is not a permutation of the I/P. $\endgroup$ Feb 13, 2023 at 9:24

1 Answer 1


This is exactly the setting of neural machine translation.

The typical architecture used nowadays for that is the Transformer model. It receives a discrete series of tokens, converts them to continuous vectors with an embedding layer, uses an encode-decoder structure and generates probabilities over the token space. The loss used for discrete outputs is categorical cross-entropy.

You may also look into different decoding (i.e. token generation) strategies, like greedy decoding and beam search.

You can find implementations, tutorials, and a vibrant community in the HuggingFace Transformers library.

  • $\begingroup$ Thank you! Yes, it is similar to NMT. An important difference though: tokens in the input and output sequence have different vocab (corresponding to two languages) in NMT whereas the vocabs are same in my case. Also, as you noted, encoder-decoder setup suits this problem. But I am testing non-transformer models (single stack, equivalent to decoder-only setup). These worked fine for the continuous setting (where sequence tokens are real-valued) as that was a regression problem; discrete case perplexes me. Do you have any insights on using decoder-only models for this use case (or NMT)? $\endgroup$ Feb 13, 2023 at 8:51
  • $\begingroup$ In NMT is very usual to share the same source and target vocabularies (and hence share also embedings). $\endgroup$
    – noe
    Feb 13, 2023 at 8:53
  • $\begingroup$ Is it decoder-only or encoder-only? (i.e. do you generate autoregressive or non-autoregressively?) $\endgroup$
    – noe
    Feb 13, 2023 at 8:54
  • $\begingroup$ non-autoregressively, predicting all output tokens at once. $\endgroup$ Feb 13, 2023 at 8:55
  • $\begingroup$ In my experience non-autoregressively sequence prediction is difficult to get right and achieving that usually involves knowledge distillation from an autoregressive teacher model. Therefore, I would not advise using it. $\endgroup$
    – noe
    Feb 13, 2023 at 8:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.