0
$\begingroup$

I have a Logistic Regression model that works fine when used with an English Dataset. But, when I pass a same format data set in a different language, (Language is URDU which is right to left and UTF-8 encoded ) the model does not predict.

I understand that the preprocessing differs for URDU, Arabic, and Hindi but even if it has been performed separately, still the model is unable to predict at all.

I would like to know why the same model can not perform on different languages directly. Good or bad results are another topic. How can I make a model which is scalable enough to perform on both English and Arabic? What are the constraints in doing so?

$\endgroup$

2 Answers 2

1
$\begingroup$

We would need to know more about how your model actually works. Logistic regression means that the model transforms an input in numbers to output in numbers using a sigmoid function. But your input is not numbers, it's text.

That means that you're doing something already to transform the text to numbers before you're doing Logistic Regression. That's why your question is a little confusing.

Also, are you training a model on English text, and evaluating on a separate English test set, and then training a second model on Arabic text and evaluating that on Arabic test data? If you try to train a model on one language and evaluate it on another it won't work.

What library are you using? NLTK? Sklearn? All of these have simple text classifiers.

It's possible you're using a bag of words model, where each word is given an ID and transformed to a sparse vector, and the entire document is transformed to a long vector whose size is the length of your vocabulary. You might be using a tf*idf stage which weights words like "the" with a lower weight than content words.

A model like this doesn't need to break necessarily if you change language. However there are a few important points you need to know:

  1. there will be a tokenisation stage. Arabic tokenisation in particular is different from English because some prefixes and suffixes are attached to the word without spaces (e.g. article al- ال, preposition "to" li- لِ, etc.) You need to have replaced the tokenisation stage correctly.

  2. If there is a stopword removal stage you should replace the stopwords with the list of the new language.

  3. If there is a lemmatisation stage you need to find an Arabic/Hindi/Urdu lemmatiser or stemmer. Otherwise you will have a problem with sparsity, e.g. an inflection of a word will occur in the test text and your model won't recognise it because the word appeared in a different inflection in the training text.

  4. Some languages don't use spaces between words like English, the obvious example is Chinese, however I don't think this is an issue for the languages you mentioned.

Whether the language is RTL or LTR makes no difference for the classifier since that only affects how the text is displayed on screen by a browser or word processor. The bytes are in the same order in an Arabic text file as an English text file.

However any model architecture would work fine with any language provided the preprocessing is adapted correctly to the language. If you used a character based model with neural network you could even make a model that works for any language without worrying about adapting the preprocessing, for example BERT has a multilingual pretrained model which can be run directly on almost any language. However these models tend to be bigger and require a powerful computer and even a GPU.

If you wanted a model that would work on text in any of the four languages you have mentioned, you could train four separate models and have a language identification component which first identifies the language and passes the input to the correct model.

$\endgroup$
0
$\begingroup$

I do not completely understand the background of the task that you are doing with logistic regression for some language. But I will give you a general picture.

Anything that is related to language has to consider two main things - Syntactics and Semantics.

These two things vary completely with different languages. For example, if you try to do a word to word translation of English to say, French, it may look syntactically fine but semantic wise (grammatical meaning) it may look completely meaningless.

Language is a "sequence" and the ordering in the sequence matters a lot. So whatever the model is trying to learn in the multidimensional dataspace with one language will not map properly to the other language even if the preprocessing is correct.

$\endgroup$
2
  • $\begingroup$ let's say i am trying to classify real news from fake news using a BOW approach all i have is a headlines and News body on which i make my classification based on the frequency of words. I appreciate your response. $\endgroup$
    – Door Lasst
    Commented May 8, 2019 at 10:07
  • $\begingroup$ Is your BOW same for all languages? I guess not. $\endgroup$ Commented May 8, 2019 at 10:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.