I have not GPU support so it often happens that my model takes hours to train. Can I train my model in batches , for example if I want to have 100 epochs for my model,but due to power cut my training stops(at 50th epoch) but when I retrain my model I want to train it from where it was left (from 50th epoch).

It would be much appreciated if anyone can explain it by some example.

  • $\begingroup$ This is possible with most (all?) mainstream deep learning frameworks by simply storing the model every N training iterations and checking for the last stored model before starting the training. Which framework are you using? $\endgroup$ – ncasas Oct 16 '17 at 17:56
  • $\begingroup$ I am using tensorflow $\endgroup$ – Berry Oct 16 '17 at 18:27
  • $\begingroup$ @ncasas could you please give how to do the same using keras ? $\endgroup$ – user2351509 Apr 7 '19 at 17:23
  • $\begingroup$ A simple google search leads you to the appropriate information. $\endgroup$ – ncasas Apr 8 '19 at 11:15

With tensorflow, currently the most straightforward and easy way to get persistence for your model is to use a tf.train.MonitoredTrainingSession. You just need to use it instead the normal tf.Session() that is frequently used. This an illustrative Python snippet:

with tf.train.MonitoredTrainingSession(checkpoint_dir='/tmp/mymodel',
                                       save_summaries_secs=600) as sess:
   _ = sess.run(train_op, feed_dict={...})

With this, your model is automagically saved every 600 secs in /tmp/mymodel and restored the next time you restart the program.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.