1
$\begingroup$

I have one data set of customer review, but the text data is given is tokenized text number. I am unable to proceed thinking about how to proceed?

As I am encountering such data set the first time, so just need guide how to proceed.

Blockquote

As you can see text field is given in number, so how to proceed please guide?. it will predict the 0/1 +ve or -ve category.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Using numbers for words is a very common way of representing words in a corpus. The thing that is missing here is how you have arrived at these numbers. Generally, if you have a corpus, you get these numbers by getting the index of these words in the vocabulary. For example, if you have two sentences : 1. John went to London 2. John went to London with Mary

Based on the order in which these numbers come, you can assign the following representation :

John : 1 , went : 2, to : 3, London : 4, with : 5, Mary : 6 Further, your sentence will be 1. 1 2 3 4 2. 1 2 3 4 5 6

Improve this answer
$\endgroup$
5
  • $\begingroup$ In the given dataset it was in number already. So from here how should I proceed? Means I have to apply TFIDF or can directly apply any classification algorithm $\endgroup$
    – Taylor
    Sep 22, 2019 at 18:54
  • $\begingroup$ Think of it this way, instead of words you have the numbers now. But you need to do everything that you would have done if you had words here. You can start with using a count vectoriser and using that with a supervised model $\endgroup$
    – Gyan Ranjan
    Sep 22, 2019 at 18:58
  • $\begingroup$ Thanks for the reply, but my doubt is lemmatization/stemming/stopword removal all these will not applicable for number as it is for words, right? So if that is the case then how the approach will the same as it is for words? $\endgroup$
    – Taylor
    Sep 22, 2019 at 19:04
  • $\begingroup$ Yeah, since you don’t have the words, you can do any preprocessing on those. You can just hope to be able to learn the pattern using the information of the presence or absence of a particular word( number in your case) $\endgroup$
    – Gyan Ranjan
    Sep 22, 2019 at 19:05
  • $\begingroup$ But when I do count vectorizer or TFIDF vectorizer then it returns all-zero array, which to confusing. Then in that case what we can have done? $\endgroup$
    – Taylor
    Sep 22, 2019 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.