I understand that my question is very broad and that the correct answer may depend on various things. I want to get an idea in general what we may expect if we have repetitive data in our dataset. Lets say we are trying to do a sentiment analysis and that there would be a class associated to each text (pos, neg, neu). I intentionally chose samples that the label associated to them may vary for example Are you there. and are you there?? with label pos and neg respectively.

This is the example:

text.             sentiment
Are you there.    pos
anyone there.     pos
are you there.    pos
Is anybody here?  pos
are you there??   neg
Hello.            neu
Hello?????        neg
agent.            neu
agent please.     pos
get me an agent   neg
human.            neg
agenttttt.        neg
agent.            neg

Is it common to get rid of duplicates in our dataset? if so what would be the reason? How about the samples that conceptually are the same but do not follow the same word/order (for example agent and agent please)

I appreciate it if you can share your thought on this.

  • 1
    $\begingroup$ Here a short answer I found in CrossValidated. $\endgroup$ Oct 14, 2021 at 4:44
  • $\begingroup$ @ShubhamPanchal thanks for sharing the link with me. Im gonna read the thread that was suggested on that link. Just one thing is that the classes assign to my repetitive data may change as explained in the question. For example Hello is neu but Hello???? is neg. $\endgroup$
    – Maria
    Oct 14, 2021 at 15:40
  • $\begingroup$ If you're processing the texts i.e. removing all the punctuation, symbols, emojis etc. then Hello and Hello??? would be considered as the same observation. As seen in most tutorials and blogs, the texts are also turned into lowercase for tokenization which creates a dict containing word-id pairs. $\endgroup$ Oct 15, 2021 at 2:52
  • $\begingroup$ Right but knowing the business behind my data I won't get rid of ??? or any sign that might carry some information for me. $\endgroup$
    – Maria
    Oct 15, 2021 at 15:28
  • $\begingroup$ You are correct, if the label change then the data is not duplicated. A different task may consider punctuation irrelevant, but for you nor punctuation nor capitalization should be discarded $\endgroup$
    – Khanis Rok
    Oct 16, 2021 at 19:10

2 Answers 2


In general it's not recommended to get rid of duplicates because it modifies the distribution of the data and this could bias the model. In other words, if the final application (or any test data) is expected to contain cases like these in similar proportions then it is preferable to train the model with these cases.

So the duplicates by themselves are not an issue, however I would say that the inconsistencies in the labels are bit more annoying. In my opinion all of the cases shown in the example should be labelled as neutral, they don't really show any particular sentiment. The fact that they are annotated with different labels without any clear reason why will probably cause inconsistencies in the model.


First of all, in problems like you brought, you typically start with preprocessing. Specifically, in your case, you need to normalize it. That means, using some processing, you have to change both Are you there and are you there??? to just are you there. By doing that, you are removing duplicate examples there. Now, unlike Erwin, I suggest that in general. The reason is your model will only learn on important examples. It will also generalize well with small data. That is because, your preprocessing step removes any kind of impurities that will create headache for your model. Specially in deep learning case, you have to have good examples. Not removing them will only work if you have very sufficient amount of data that allows your model to accurately capture the relationships between words and word derivatives. For example, your model should learn that there and there??? are the same given Are you and are you are present. If you normalize it, you are doing the model a huge favor there.

Secondly, as Erwin pointed out, the labeling in the problem is concerning. Yes, sometimes, same sentences spoke with different tones create different sentiments. In that case, you need to augment more information on the inputs to signal that. That could be a few seconds of the actual voice. Without such treatment, your model will get confused and never converge.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.