How to approach Multilingual Text Classification?

Question

The company I'm working for runs social network sites, and we're classifying messages sent one-to-one as spam or not spam. The issue is that it's been trained exclusively on German training data, since most of the networks we operate are local to Germany. Soon we will need to support multiple languages for this, since the one trained on only German will often think messages in other languages are spam as well.

I'd prefer not to have to maintain different training data for different languages if possible, so my question is:

What's a reasonable approach to doing text classification for multiple languages?

A spam message in our cases usually contains obfuscated links and subtle references to other websites. Example of a message that should be classified as spam:

hi funnyguy kennst du mich noch ? konnte hier kein bilder hoch laden komm mal bitte zu ( somethingelse.com) mein nik ist ( 19theusername91 ) mal sehen ob du mich noch kennst :)

Somegirl

The current approach is the normal stemming, TF-IDF and LSA for pre-processing, then a two-level classifier: an ensemble of normal classifiers that's used as input for a linear classifier that will make the final decision.

Answer 1

In a similar situation, -after trying some alternatives- I had to build a language classifier in front of all learning and classification steps.

That is, for learning:

1. Detect the language of the input (say, an enumeration like "DE", "EN", etc.)
2. Apply language specific stemming to the words of the input.
3. Prepend words in the learning phase with the language identifier (i.e. "de_du", "en_you")
4. Use these words in a single training model.

In classification phase, use the same steps.

Answer 2

I suggest to acquire some multilingual training and test data and do experiments whether it is better to train on mixed languages or doing language detection first and use monolingual models. The fact that you use some NLT techniques (like the stemming mentioned) suggests the latter path.

Answer 3

Some discriminative features like presence of URLs, frequency of proper punctuation and spelling mistakes translate easily. For semantic features you can use multilingual word embeddings, so your content can be treated by the same classifier, regardless of language. My educated guess is that you should be able to detect most spam without going this far.

Answer 4

Text2Text can produce neural embeddings and TFIDF embeddings as input features for a classifier. Would not be necessary to do stop words or stemming as described in this paper.