0
$\begingroup$

I have a question on batch learning of neural network.
A neural network learns in batches and modifies weights in every iteration. Question: If I save checkpoints after a batch, and then load the weights at a later time and train with new batch, will it be different from training both batches in 1 go?

Example: If I have a batch size of 100 and training data of 1000 points. So would it be different, in outputted checkpoint file, if I train with 9 batches (900 data points) in one go -> save checkpoint -> load checkpoint next day -> train with last batch -> save checkpoint ... Vs give all 1000 datapoints (i.e., 10 batches) -> train -> save checkpoint file ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

There will not be any difference between :

  1. Training all batches + Epoch in one go
  2. Saving a checkpoint and resuming the training later

Saving a checkpoint is a very good practice. It is not uncommon for training to fail due to environmental issues (E.g.: Spot instance on AWS gets kicked out , PC crash).

With Libraries like Keras, it is very straight-forward to checkpoint models.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.