Accuracy decreases after adding more samples

Question

I'm working on a multiclass text classification task (5 classes). I've 2 types of datasets:

1. regular (~22000 samples)
2. dataset of duplicates (~19000 samples)

I've written a logic that labels them all.

I've noticed that after adding an additional set of data (in which being labeled using a different logic code path), the val_accuracy doesn't reach more than 67%, while using only the regular data set I can easily each 74%.

A few questions:

1. Is working only with ~22000 samples is sufficient to this kind of classification problem?

2. How come adding more samples damages the val_accuracy (I was under the impression it should increase it).

Some more info

I feel like my use case wasn't elaborated enough:

My goal is to classify bugs to the relevant owner group (there're 5 of them).

A duplicate bug is not nearly identical (text wise) to his "dupped" one and so I though adding it can improve the model accuracy.

Again, as mentioned, my logic takes care of labeling the duplicated bugs correctly (by finding the owner group of the original one).

Once doing so, I'm adding the duplicates to the dataset, shuffle it and only then splitting it to train and test.

Another point to mention is that indeed my dataset is imbalance and I use class weights to handle that (also tried augmentation but it took a lot of time and didn't change much)

Answer 1

When you add data, do you change also the validation dataset?

Are the duplicates identical to the original sentences?

Normally, you shouldn't add identical duplicates, otherwise, you would reinforce some weights and have a kind of overfitting that could alter weights from data that are not duplicates.

In addition, if by chance some duplicates are not part of the validation dataset and have a wrong result, it would double the error value.

If duplicates are similar but not identical, you should ensure that the validation dataset takes into account random data the regular and the duplicate data set.

Answer 2

• The answer to the first question depends on your problem's complexity and what you want exactly.

• Related to increasing your accuracy, I think adding random data does not solve your problem. Instead, check those:

  • Maybe your dataset includes outlier samples
  • Maybe your dataset imbalance
  • Maybe your samples are not enough (I prefer using the over/under-sampling algorithms instead of random samples)
  • Maybe some features are less important than others (Use Feature Selection algorithms)
  • Use different normalization algorithms

Generally speaking,One of those steps solve your problem