I'm working on a multiclass text classification task (5 classes). I've 2 types of datasets:
- regular (~22000 samples)
- dataset of duplicates (~19000 samples)
I've written a logic that labels them all.
I've noticed that after adding an additional set of data (in which being labeled using a different logic code path), the
val_accuracy doesn't reach more than 67%, while using only the regular data set I can easily each 74%.
A few questions:
Is working only with ~22000 samples is sufficient to this kind of classification problem?
How come adding more samples damages the val_accuracy (I was under the impression it should increase it).
Some more info
I feel like my use case wasn't elaborated enough:
My goal is to classify bugs to the relevant owner group (there're 5 of them).
A duplicate bug is not nearly identical (text wise) to his "dupped" one and so I though adding it can improve the model accuracy.
Again, as mentioned, my logic takes care of labeling the duplicated bugs correctly (by finding the owner group of the original one).
Once doing so, I'm adding the duplicates to the dataset, shuffle it and only then splitting it to train and test.
Another point to mention is that indeed my dataset is imbalance and I use class weights to handle that (also tried augmentation but it took a lot of time and didn't change much)