I have a very large dataset and due to computational constraints, I have to divide the data into 20 parts (each part is around 1.5GB). I constructed a deep CNN model using Keras for this dataset. The original scenario (assuming I have a very big RAM that can contain the whole ~30GB dataset to be processed) is that I'm going to train this for 10 epochs with the first 6 epochs to have a learning rate of 0.001. After that, every subsequent epoch, the learning rate will be reduced by half. At the moment, what I am doing is to train each part for 10 epochs following the scheduled learning rate stated previously. After some thinking, I then realized that there may be a better training scenario that resembles better with the original scenario. Which is to loop all the 20 parts using the first 6 epochs (learning rate= 0.001), and then going to the next loop will be 1 epoch with learning rate of 0.0005 (half of the previous epoch), and looping again by halving the learning rate until it reaches the original 10 epochs that I want to achieve. Here is the summary of these 2 scenarios.

Scenario 1: Part 1 data are trained with 10 epochs with scheduled learning rates stated above. After finish training part 1, the model is saved and then loaded again for further training using part 2 data. This part 2 data will be used for training the previously trained model for 10 epochs with the same scheduled learning rate. This loop is continued until all 20 parts are gone through.

Scenario 2: Part 1 trained with 6 epochs with a learning rate of 0.001. After that, the model is saved and further continue the training using part 2 data with another 6 epochs using the same learning rate (0.001). This continues until all 20 parts have been feed into the model. After that, we go back to part 1, but now we trained the model with only 1 epoch using a 0.0005 learning rate (0.5 * 0.001). Continue the training until all 20 parts are gone through. Re-loop again with 0.00025 learning rate (0.5*0.0005). We will stop re-looping until the learning rate becomes 0.0000625.

I think scenario 2 is the one that matches the original scenario but I'm not sure why scenario 1 can't achieve the same thing as scenario 2. Thank you in advance, and sorry if there is any wrong grammatical usage.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.