I have a question regarding
BATCH_SIZE on multi-class classification task with imbalanced data. I have 5 classes and a small dataset of around
5000 examples. I have watched G. Hinton's lectures on Deep Learning and he states that each mini batch should ideally be balanced (meaning each batch should contain approximately same number of data points for each class). This can be approximated by shuffling data and then drawing random batch from it.
But, in my mind this will only work if we have some what large and BALANCED dataset. In my case, I think that setting
BATCH_SIZE to be
>=16 it might have a bad impact on learning, and network will not be able to generalize. Is it better to maybe use
SGD and update weights after each sample has been processed (i.e. online training)? P.S. have in in mind that I am using label smoothing (i.e. class weighted loss)