# Transforming words in sentences into vector form to prepare a model

I want to build a simple classifier that classifies if the text is a question or just a simple message. I understand logistic regression and can work to create a simple neural network.

I have the labeled input data in English, Japanese, Korean, Thai. How could I transform this data before I feed it into the classifier?

• Take a look at Tf-Idf scheme scikit-learn.org/stable/modules/generated/… – Ankit Seth Jun 12 '18 at 6:07
• @AnkitSeth Could you please elaborate more on this. – Suhail Gupta Jun 12 '18 at 6:31
• It is basically a scheme to convert words to numeric form. For each document, it will take the frequency of a particular word in that document, number of documents which contain that word, and find a numerical equivalent of that word. You can see the working of Tf-Idf in detail on this tfidf.com – Ankit Seth Jun 12 '18 at 6:53
• @AnkitSeth Okay. Does it use some kind of pre-trained model? Also, after I get the output as {u'boy': '1.6931471805599454', u'good': '1.6931471805599454', u'this': '1.2876820724517808', u'is': '1.0', u'very': '1.2876820724517808', u'strange': '1.6931471805599454', u'suhail': '1.6931471805599454', u'nice': '1.6931471805599454'}, should I use these values as in input to classifier? – Suhail Gupta Jun 12 '18 at 7:48
• No, it does not use a pre-trained model. Now, your features are these words - "boy", "good", "this", "is" etc and the values are the numbers you got. Yes, you can use these values as input to classifier. Create a dataframe of this and pass that frame in your model. The columns of the frame should be these words and the number of rows should be number of documents/texts you have. – Ankit Seth Jun 12 '18 at 9:41