What effect repetitive data will have on the performance of the model

Question

I understand that my question is very broad and that the correct answer may depend on various things. I want to get an idea in general what we may expect if we have repetitive data in our dataset. Lets say we are trying to do a sentiment analysis and that there would be a class associated to each text (pos, neg, neu). I intentionally chose samples that the label associated to them may vary for example Are you there. and are you there?? with label pos and neg respectively.

This is the example:

text.             sentiment
Are you there.    pos
anyone there.     pos
are you there.    pos
Is anybody here?  pos
are you there??   neg
Hello.            neu
Hello?????        neg
agent.            neu
agent please.     pos
get me an agent   neg
human.            neg
agenttttt.        neg
agent.            neg

Is it common to get rid of duplicates in our dataset? if so what would be the reason? How about the samples that conceptually are the same but do not follow the same word/order (for example agent and agent please)

I appreciate it if you can share your thought on this.

Answer 1

In general it's not recommended to get rid of duplicates because it modifies the distribution of the data and this could bias the model. In other words, if the final application (or any test data) is expected to contain cases like these in similar proportions then it is preferable to train the model with these cases.

So the duplicates by themselves are not an issue, however I would say that the inconsistencies in the labels are bit more annoying. In my opinion all of the cases shown in the example should be labelled as neutral, they don't really show any particular sentiment. The fact that they are annotated with different labels without any clear reason why will probably cause inconsistencies in the model.

Answer 2

First of all, in problems like you brought, you typically start with preprocessing. Specifically, in your case, you need to normalize it. That means, using some processing, you have to change both Are you there and are you there??? to just are you there. By doing that, you are removing duplicate examples there. Now, unlike Erwin, I suggest that in general. The reason is your model will only learn on important examples. It will also generalize well with small data. That is because, your preprocessing step removes any kind of impurities that will create headache for your model. Specially in deep learning case, you have to have good examples. Not removing them will only work if you have very sufficient amount of data that allows your model to accurately capture the relationships between words and word derivatives. For example, your model should learn that there and there??? are the same given Are you and are you are present. If you normalize it, you are doing the model a huge favor there.

Secondly, as Erwin pointed out, the labeling in the problem is concerning. Yes, sometimes, same sentences spoke with different tones create different sentiments. In that case, you need to augment more information on the inputs to signal that. That could be a few seconds of the actual voice. Without such treatment, your model will get confused and never converge.