0
$\begingroup$

Im doing preprocessing on a text dataset. I have certain numerics in it like:

  • date(1st July)
  • year(2019)
  • tentative values (3-5 years/ 10+ advantages).
  • unique values (room no 31/ user rank 45)
  • percentage(100%)

Is it recommended to discard this numerics before creating a vectorizer(bow/tf-idf) for any model(classification/regression) development?

Any quick help on this is much appreciated. Thank you

$\endgroup$

2 Answers 2

2
$\begingroup$

To build on Prashant's answer, it will depend on your problem. If you think those values are important to your task, you might try to extract them and tack them onto the end of your data (I'm thinking like [this question asked here] which uses multiple different kinds of data in a regression problem).

An easy thing to do (and probably the right call the majority of the time) would be to just remove all those numbers, but another strategy I've seen elsewhere is to use rules to convert the different numbers into their "type". Meaning 2019 used as a year would be replaced with a token like #YEAR, 100% replaced by #PERCENT, etc.

$\endgroup$
1
$\begingroup$

Is it recommended to discard this numerics before creating a vectorizer(bow/tf-idf) for any model(classification/regression) development?

It depends on the problem statement for example year could be significant if you want to find the trend and year has many unique value but if it's constant then you can remove it.

To add to that if you are doing sentiment analysis then numeric variables don't make much sense.

$\endgroup$
2
  • $\begingroup$ Thanks for the inputs Prashant. Talking about my problem, it is a sentiment classification problem. It is the 2019 product review dataset so year I can neglect. But in case of numerics like the percentage values 100% accuracy and 2nd most useful product, here the numerics has some contribution to positive sentiment. How can I make use of these useful contributor numerics in preprocessing? $\endgroup$
    – emily
    Commented Sep 1, 2020 at 14:34
  • $\begingroup$ Generally sentiment analysis is used for finding sentiment from the word corpus but if you want to add numeric you could calculate the normalized frequency of percentage value in text and then assign whether it's positive or negative. Similar is the case of useful product. Hope this answers your question. $\endgroup$ Commented Sep 1, 2020 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.