I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.

Do I need to have labelled training data in both English and Arabic to fine tune a pretrained transformer, such as BLOOM, or can I fine tune the model using only English samples, and then the model should also work well on Arabic samples for my classificaiton task, since the fine tuning only trained the classification head?

My thoughts are that the pretraining of this model should allow it to transform the same input texts in English and Arabic to the same (or similar) embedding, which the classification head would have learned to then predict these embeddings accurately through the fine tuning.


1 Answer 1


As far as I know, few multilingual models study their representation space to see if the representations of different languages occupy overlapping regions. The ones that do, usually find that the representations of different languages are disjoint, e.g. this:

enter image description here

Therefore, I would not be very confident doing what you propose.

Of course, it can happen that it works, but I would certainly suggest having training data for all the languages you need at inference time. If you train just with English, at least ensure that the validation and test data contain all languages, to understand the actual performance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.