# What effect repetitive data will have on the performance of the model

I understand that my question is very broad and that the correct answer may depend on various things. I want to get an idea in general what we may expect if we have repetitive data in our dataset. Lets say we are trying to do a sentiment analysis and that there would be a class associated to each text (pos, neg, neu). I intentionally chose samples that the label associated to them may vary for example Are you there. and are you there?? with label pos and neg respectively.

This is the example:

text.             sentiment
Are you there.    pos
anyone there.     pos
are you there.    pos
Is anybody here?  pos
are you there??   neg
Hello.            neu
Hello?????        neg
agent.            neu
get me an agent   neg
human.            neg
agenttttt.        neg
agent.            neg


Is it common to get rid of duplicates in our dataset? if so what would be the reason? How about the samples that conceptually are the same but do not follow the same word/order (for example agent and agent please)

I appreciate it if you can share your thought on this.

• Here a short answer I found in CrossValidated. Oct 14 at 4:44
• @ShubhamPanchal thanks for sharing the link with me. Im gonna read the thread that was suggested on that link. Just one thing is that the classes assign to my repetitive data may change as explained in the question. For example Hello is neu but Hello???? is neg. Oct 14 at 15:40
• If you're processing the texts i.e. removing all the punctuation, symbols, emojis etc. then Hello and Hello??? would be considered as the same observation. As seen in most tutorials and blogs, the texts are also turned into lowercase for tokenization which creates a dict containing word-id pairs. Oct 15 at 2:52
• Right but knowing the business behind my data I won't get rid of ??? or any sign that might carry some information for me. Oct 15 at 15:28
• You are correct, if the label change then the data is not duplicated. A different task may consider punctuation irrelevant, but for you nor punctuation nor capitalization should be discarded Oct 16 at 19:10