How should I treat these non-English documents in the NLP task?

Question

So I have a small corpus of about 30k documents and about 50 documents in this corpus are in other languages (Persian, Chinese, Arabic, German, Spanish etc). I will be using this corpus for training a machine learning model.

Now the question is: How should these non-English documents be treated?

1. Should I exclude them from the final corpus and from training the model?
2. or should I manually translate them (Requesting natives from each language to translate it for me) and include them in the final corpus?
3. or should I use Google translate/DeepL to translate these non-English documents into English and then include them in the final corpus?

Each document in the corpus under question is not larger than 500 words each.

Any help or hint will be appreciated.

Answer 1

You can use these tips :

Should I exclude them for the corpus and from training the model?

You can do this if you don't have a lack of data. But I think excluding 500 docs from 30K docs won't make a big difference in training. The model's generalisation power won't be compromised.

should I manually translate them (Requesting natives from each language to translate it for me) and include them in the final corpus?

You should do this only when you need the 500 docs as they hold significant information. I would not recommend this method personally.

should I use Google translate/DeepL to translate these non-English documents into English and then include them in the final corpus?

That is something you can do. Using Google Translate could be a help if the structure of the sentences in the docs is simple and sober. You can get good translations without the need of any natives of other countries.

I think you should opt for this method.

Conclusion:

1. Translate the docs which hold importance using Google Translate.
2. Else, omit them if you have an extra of 500 docs. They willl not affect the model's performance significantly.

Tip:

I don't think that you can paste the 500 docs in the Google Translate console. This will be time consuming if the docs belong to different languages. Hence, try to omit these documents. Otherwise, you will require a mini application which could use some translation API to transform the docs.