Do I need training data in multiple languages for a multilingual transformer?

Question

I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.

Do I need to have labelled training data in both English and Arabic to fine tune a pretrained transformer, such as BLOOM, or can I fine tune the model using only English samples, and then the model should also work well on Arabic samples for my classificaiton task, since the fine tuning only trained the classification head?

My thoughts are that the pretraining of this model should allow it to transform the same input texts in English and Arabic to the same (or similar) embedding, which the classification head would have learned to then predict these embeddings accurately through the fine tuning.

Answer 1

As far as I know, few multilingual models study their representation space to see if the representations of different languages occupy overlapping regions. The ones that do, usually find that the representations of different languages are disjoint, e.g. this:

Therefore, I would not be very confident doing what you propose.

Of course, it can happen that it works, but I would certainly suggest having training data for all the languages you need at inference time. If you train just with English, at least ensure that the validation and test data contain all languages, to understand the actual performance.