0
$\begingroup$

Im doing preprocessing on a text dataset. I have certain numerics in it like:

  • date(1st July)
  • year(2019)
  • tentative values (3-5 years/ 10+ advantages).
  • unique values (room no 31/ user rank 45)
  • percentage(100%)

Is it recommended to discard this numerics before creating a vectorizer(bow/tf-idf) for any model(classification/regression) development?

Any quick help on this is much appreciated. Thank you

$\endgroup$
0
$\begingroup$

Is it recommended to discard this numerics before creating a vectorizer(bow/tf-idf) for any model(classification/regression) development?

It depends on the problem statement for example year could be significant if you want to find the trend and year has many unique value but if it's constant then you can remove it.

To add to that if you are doing sentiment analysis then numeric variables don't make much sense.

$\endgroup$
2
  • $\begingroup$ Thanks for the inputs Prashant. Talking about my problem, it is a sentiment classification problem. It is the 2019 product review dataset so year I can neglect. But in case of numerics like the percentage values 100% accuracy and 2nd most useful product, here the numerics has some contribution to positive sentiment. How can I make use of these useful contributor numerics in preprocessing? $\endgroup$ – emily Sep 1 '20 at 14:34
  • $\begingroup$ Generally sentiment analysis is used for finding sentiment from the word corpus but if you want to add numeric you could calculate the normalized frequency of percentage value in text and then assign whether it's positive or negative. Similar is the case of useful product. Hope this answers your question. $\endgroup$ – prashant0598 Sep 1 '20 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.