I have a dataset of multiple classes (About 50). The dataset doesn't have the same number of pictures per class, some have 300, some have 1000, and some more, and I've seen that this ruined my accuracy on my model
First of all, the data is too big for me to store in the RAM, so I have to split it into parts (I take 300 pictures of each class at once), train the network on that data then repeat the process until I have no more photos left.
The question I have is, is it a problem if I do this (I guess it is from my results)? I have for example from 100 classes of pictures, only 10 left and I train the model only on those 10,then on only 5,then on 3,etc.? Because I did that and after I went over the smallest classes, the overall accuracy went up, but those smaller classes accuracy went down, and in the end from 80% in the beginning for each class I got 0-2% on 48 classes, and 99% on 2 classes.
How can I solve this 'unevenness' (I don't know the word) then so I won't have this problem anymore?