Consider the following training corpora:

  • dataset1: composed of French instances

  • dataset2: dataset1 + Arabic instances

  • test_dataset (for both scenarios): composed of French instances

(the same annotation guidelines were used for both languages).

After analyzing the results of our preliminary experimental setup, we chose BERT as our baseline system.

Considering the different languages involved, we experimented with different models capable of handling them: FlauBERT and CamemBERT (for French), AraBERT (for Arabic) as well as BERT multilingual. Generally, for both languages, the results obtained by BERT multilingual are lower than those obtained by the language specific models.

Is it theoretically possible to merge multiple models together into one model, effectively combining all the data learnt so far? For example, combining CamemBERT trained only on the French part of dataset2 and AraBERT trained only on the Arabic part?

  • $\begingroup$ theoretically it is possible to merge any models into one merged model and methods (and performance) vary based on the kind of models and scope. One simple method is taking the average output of all models as combined model's output $\endgroup$
    – Nikos M.
    Oct 19 '21 at 17:06

An engineering solution would be: Create a language detector, feed the input to the detector, based on the language type classification, send the input to the appropriate model, that is if the language is french, feed the input directly to the CamemBERT. The output will be as accurate as the CamemBERT multiplied by the accuracy of the language detector.

But if you are asking if the model weights can be manipulated so that we can get a new fully unified model, it is still in a research phase.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.