I want to build a simple classifier that classifies if the text is a question or just a simple message. I understand logistic regression and can work to create a simple neural network.

I have the labeled input data in English, Japanese, Korean, Thai. How could I transform this data before I feed it into the classifier?

  • $\begingroup$ Take a look at Tf-Idf scheme scikit-learn.org/stable/modules/generated/… $\endgroup$
    – Ankit Seth
    Jun 12, 2018 at 6:07
  • $\begingroup$ @AnkitSeth Could you please elaborate more on this. $\endgroup$ Jun 12, 2018 at 6:31
  • $\begingroup$ It is basically a scheme to convert words to numeric form. For each document, it will take the frequency of a particular word in that document, number of documents which contain that word, and find a numerical equivalent of that word. You can see the working of Tf-Idf in detail on this tfidf.com $\endgroup$
    – Ankit Seth
    Jun 12, 2018 at 6:53
  • $\begingroup$ @AnkitSeth Okay. Does it use some kind of pre-trained model? Also, after I get the output as {u'boy': '1.6931471805599454', u'good': '1.6931471805599454', u'this': '1.2876820724517808', u'is': '1.0', u'very': '1.2876820724517808', u'strange': '1.6931471805599454', u'suhail': '1.6931471805599454', u'nice': '1.6931471805599454'}, should I use these values as in input to classifier? $\endgroup$ Jun 12, 2018 at 7:48
  • $\begingroup$ No, it does not use a pre-trained model. Now, your features are these words - "boy", "good", "this", "is" etc and the values are the numbers you got. Yes, you can use these values as input to classifier. Create a dataframe of this and pass that frame in your model. The columns of the frame should be these words and the number of rows should be number of documents/texts you have. $\endgroup$
    – Ankit Seth
    Jun 12, 2018 at 9:41

1 Answer 1


An approach would be to sort out all the words in your data according to how often they appear, i.e. their "frequency". After that, pick the "X" most frequent words in your dataset to use them for the classification of your dataset.

Assuming that you are working with Python and Keras, you should use the Embedding layer. For more details about how to use that layer, check this.

Shortly, what this layer does is that it maps the input to a high dimensional vector domain. A word is converted to a real-valued vector and word similarity is evaluated by the "closeness" of two word-vectors in the high-dimensional vector space.

Also make sure that your dataset consists of texts of fixed-length, by truncating long sequences or zero-padding short ones.

After all this is done, you can train a recurrent neural network with LSTM neurons as a text classifier. LSTMs have been proven very successful in text processing due to their inherent memory.

A hands-on Python/Keras tutorial that demonstrates all the above can be found here, I am sure it will be of high help :)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.