I understand that my question is very broad and that the correct answer may depend on various things. I want to get an idea in general what we may expect if we have repetitive data in our dataset. Lets say we are trying to do a sentiment analysis
and that there would be a class associated to each text (pos, neg, neu). I intentionally chose samples that the label
associated to them may vary for example Are you there.
and are you there??
with label pos
and neg
respectively.
This is the example:
text. sentiment
Are you there. pos
anyone there. pos
are you there. pos
Is anybody here? pos
are you there?? neg
Hello. neu
Hello????? neg
agent. neu
agent please. pos
get me an agent neg
human. neg
agenttttt. neg
agent. neg
Is it common to get rid of duplicates in our dataset? if so what would be the reason? How about the samples that conceptually are the same but do not follow the same word/order (for example agent
and agent please
)
I appreciate it if you can share your thought on this.
Hello
isneu
butHello????
isneg
. $\endgroup$Hello
andHello???
would be considered as the same observation. As seen in most tutorials and blogs, the texts are also turned into lowercase for tokenization which creates adict
containing word-id pairs. $\endgroup$???
or any sign that might carry some information for me. $\endgroup$