If you are interested in convergence, then most probably it shouldn't converge. I used "should" because although we have an intuition of how the convergence work but no one is 200% sure of everything.
Let's understand a few of the aspects of the whole learning process.
- Every data point has a distinct Loss space
- Overall Loss space will be the average of all individual Loss surface
- We can safely assume, that the overall Loss space of the data points in one class will be closer to each other but very different when compared to the other class.
- Lastly, the Gradient is calculated at the end of a Batch
When we have well-shuffled batch -
In this case, the Gradient after each batch will be directed towards the overall Gradient because the Gradient will be averaged. It means a smooth Loss reduction with each batch
When we have each batch of one Class -
In this case, every alternate batch will move on two different Loss space(One of each Class).
The issue will be that we will calculate Loss for one space, but after the weight update at the end of a batch, the next batch will continue in its own space but using the weight from the previous batch. This will make learning very unpredictable.
Also, many logic applied in optimizers will not help because Gradient will randomly change with each batch.
I tried this on MNIST digit (just using 0,1 digit) and got this Loss per Batch. I have not out enough effort on LR optimization, LR decay, etc. So, can't conclude that result will be always similar.