What would be the best approach for classifying non-English (Sinhala / Tamil) text? Currently I use Fasttext. Are there any better options?

I want to classify user questions into chatbot intents. Therefore, there may be many target classes.

  • 1
    $\begingroup$ Could you please describe your problem more? By many target classes you mean multi-class or multi-label as well? $\endgroup$ Dec 17 '19 at 10:19
  • $\begingroup$ Multi classes. There are more than 80 classes to predict. Need to classify a question into only one intent. $\endgroup$
    – Kabilesh
    Dec 17 '19 at 10:21
  • $\begingroup$ What's your data set balance? $\endgroup$ Dec 17 '19 at 10:53
  • $\begingroup$ Total of nearly 20, 000 sample questions belonging to 80 classes. $\endgroup$
    – Kabilesh
    Dec 17 '19 at 11:21

As far as I know, the best way would be to use pretrained embedder. Embedder encodes your text into language-agnostic latent space. You input your text and you get fixed-length numerical vector as an output. You can use latent space encodings as a feature vector, which you can use to train discriminative models. They also suit well for resampling like SMOTE or ADASYN.

Some time ago Facebook released a model called LASER. You can read about it here. It supports Sinhala and Tamil as well. Here is a github repository. There is also unofficial distribution on pypi. It substitutes internal tools for tokenization and BPE encoding. For sake of convenience I've been working mostly with this distribution and I can confirm it works just fine. Here is a repository.

I'd also suggest to consider embedder abuse. It covers a lot of languages, which means you can train your models on e.g. English and prediction will work for Tamil out of the box!

Natural language is hyper-dimensional space and most seminal models are using encoders. To my knowledge it is the go-to approach for any language.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.