Consider the following training corpora:
dataset1: composed of French instances
dataset2: dataset1 + Arabic instances
test_dataset (for both scenarios): composed of French instances
(the same annotation guidelines were used for both languages).
After analyzing the results of our preliminary experimental setup, we chose BERT as our baseline system.
Considering the different languages involved, we experimented with different models capable of handling them: FlauBERT and CamemBERT (for French), AraBERT (for Arabic) as well as BERT multilingual. Generally, for both languages, the results obtained by BERT multilingual are lower than those obtained by the language specific models.
Is it theoretically possible to merge multiple models together into one model, effectively combining all the data learnt so far? For example, combining CamemBERT trained only on the French part of dataset2 and AraBERT trained only on the Arabic part?