What would be the best approach for classifying non-English (Sinhala / Tamil) text? Currently I use Fasttext. Are there any better options?
I want to classify user questions into chatbot intents. Therefore, there may be many target classes.
4
-
1$\begingroup$ Could you please describe your problem more? By many target classes you mean multi-class or multi-label as well? $\endgroup$– Piotr RarusCommented Dec 17, 2019 at 10:19
-
$\begingroup$ Multi classes. There are more than 80 classes to predict. Need to classify a question into only one intent. $\endgroup$– KabileshCommented Dec 17, 2019 at 10:21
-
$\begingroup$ What's your data set balance? $\endgroup$– Piotr RarusCommented Dec 17, 2019 at 10:53
-
$\begingroup$ Total of nearly 20, 000 sample questions belonging to 80 classes. $\endgroup$– KabileshCommented Dec 17, 2019 at 11:21
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
As far as I know, the best way would be to use pretrained embedder. Embedder encodes your text into language-agnostic latent space. You input your text and you get fixed-length numerical vector as an output. You can use latent space encodings as a feature vector, which you can use to train discriminative models. They also suit well for resampling like SMOTE or ADASYN.
Some time ago Facebook released a model called LASER. You can read about it here. It supports Sinhala and Tamil as well. Here is a github repository. There is also unofficial distribution on pypi. It substitutes internal tools for tokenization and BPE encoding. For sake of convenience I've been working mostly with this distribution and I can confirm it works just fine. Here is a repository.
I'd also suggest to consider embedder abuse. It covers a lot of languages, which means you can train your models on e.g. English and prediction will work for Tamil out of the box!
Natural language is hyper-dimensional space and most seminal models are using encoders. To my knowledge it is the go-to approach for any language.
