When I convert my multilingual transformer model to a single lingual transformer model (got my languages embedding from the multilingual transformer and deleted other embeddings, decreased dimensions of embedding layers), the loss is much less. But I didn't understand why. What can be the reason for that?


1 Answer 1


New Answer

The loss of a text generation task like question generation is normally the average categorical cross-entropy of the output at every time step.

Drastically reducing the number of tokens means that the number of classes of the output probability distribution is greatly reduced.

The value of cross-entropy depends on the number of classes. Having more classes means that the output distribution must cover more options and it is more difficult to assign more probability to the ground truth class (i.e. the correct token).

Therefore, it is to be expected that, if you drastically reduce the number of tokens, the value of the loss is lower.

Old answer

From your description, I understand that:

  • What you had was a Transformer trained on multilingual data with word-level tokens (because if you had subword-level tokens like BPE or unigram then you would not be able to filter by language from the token list so easily).
  • What you did was:
    • Removing the entries associated with words in other languages from the token list.
    • Reduce the embedding size.
    • Retrain your model on the data of a single language pair.

With those assumptions:

When you "converted your model from multilingual to single lingual", you simplified the task enormously. It seems that the gain in the simplicity of the task surpassed the loss of capacity of the model caused by the reduction of the embedding size.

  • 1
    $\begingroup$ I see the misunderstanding. "Embedding size" is normally used to refer to the dimensionality of the embedding, not the number of elements in the embedding table. What they are changing is the number of elements in the embedding table. We were speaking about different things. $\endgroup$
    – noe
    Commented Jul 23, 2022 at 16:04
  • 1
    $\begingroup$ It is not clear to me what loss you are referring to. They are not retraining the model. Are you? On what data? And what are you comparing the loss with? $\endgroup$
    – noe
    Commented Jul 23, 2022 at 16:11
  • 1
    $\begingroup$ I edited the answer with this new information. $\endgroup$
    – noe
    Commented Jul 23, 2022 at 16:47
  • 2
    $\begingroup$ Next time, please include all this information in the original question. This will save you and the people answering your question a lot of wasted time due to attempting to answer without enough context. $\endgroup$
    – noe
    Commented Jul 23, 2022 at 16:48
  • 1
    $\begingroup$ You are right, my bad. Thank you so much. $\endgroup$
    – canP
    Commented Jul 23, 2022 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.