Smaller embedding size causes lower loss

Question

When I convert my multilingual transformer model to a single lingual transformer model (got my languages embedding from the multilingual transformer and deleted other embeddings, decreased dimensions of embedding layers), the loss is much less. But I didn't understand why. What can be the reason for that?

Answer 1

New Answer

The loss of a text generation task like question generation is normally the average categorical cross-entropy of the output at every time step.

Drastically reducing the number of tokens means that the number of classes of the output probability distribution is greatly reduced.

The value of cross-entropy depends on the number of classes. Having more classes means that the output distribution must cover more options and it is more difficult to assign more probability to the ground truth class (i.e. the correct token).

Therefore, it is to be expected that, if you drastically reduce the number of tokens, the value of the loss is lower.

Old answer

From your description, I understand that:

• What you had was a Transformer trained on multilingual data with word-level tokens (because if you had subword-level tokens like BPE or unigram then you would not be able to filter by language from the token list so easily).
• What you did was:
  • Removing the entries associated with words in other languages from the token list.
  • Reduce the embedding size.
  • Retrain your model on the data of a single language pair.

With those assumptions:

When you "converted your model from multilingual to single lingual", you simplified the task enormously. It seems that the gain in the simplicity of the task surpassed the loss of capacity of the model caused by the reduction of the embedding size.