Deep Learning: Does starting the training on a smaller subset of the data make sense?

Question

I trained a deep neural network with a small subset of my data, which allowed me to go through many epochs in a short amount of time and allowed the model to perform reasonably, then I gave it the entire data set (10X more data) and it improved further. When I give it just the entire data set the model seems to perform terribly, but also it was difficult to reach many epochs due to the extra time needed for training.

My question is: does it make sense to "warm up" the training a large deep neural network with a smaller subset of the data, or should one always just provide the full data?

To add a bit more detail: I had a heavy class imbalance. The model always predicts 0 when I give it the full data set, but when I overfit on a smaller training set, it learns at least to not always predict 0. I notice this guide in step 5, suggests something similar to what I did, but I am not sure if it is theoretically sound or just a nice way of checking the model "can work given enough epochs".

Answer 1

It makes the most sense to train a deep learning model on the entire dataset. If you train on a subset of training, you are more likely to end up a local minimum.

One option is to adjust the size of the batch for stochastic gradient descent (SGD). The batch size is the number of data points given to the model before performing a learning update.