1
$\begingroup$

I need to train a model but the data after preprocessing is too large to fit in the RAM. Is it possible to preprocess the data in batches and train a model for the small data set and then save the weights and then train it again for the next batches with the saved weights?

the workflow would be this way:

preprocess -> train -> save weights -> preprocess next batch -> train with saved weights ...

If yes, how can I do it?

$\endgroup$
0
$\begingroup$

As you said:

Is it possible to preprocess the data in batches and train a model for the small data set

Yes! In practice, training of Neural Networks always happens with batches. You never fit the whole dataset at once in the model, whatever medium sized dataset could crash any machine.

This is how it works: You extract a slice of your dataframe (the number is usually in the range of 32-256 observations, but it's very task-specific), you train the model on that, and you move to the next until all dataset is completed. This is what an epoch is, in the Deep Learnig jargon.

... and then save the weights and then train it again for the next batches with the saved weights?

There's no need to save and reload the model at each iteration, just save it once it was trained for the number of epochs of your choice. If the following day you want to keep training, just reload it and keep iterating one batch at a time.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Yes, you can pre-process a single batch, one at a time. This is what is commonly done, for example, when you do image augmentation for CNN training. You keep your original images saved in a directory, you load a batch, you augment them, and feed them to the model. This image augmentation is just the first example that came up to my mind, but this "batch processing" can work for any kind of model. $\endgroup$ – Leevo May 21 at 14:29
  • $\begingroup$ How do we reload the saved model, I am very new to this, I am using google colab. $\endgroup$ – inquisitive May 21 at 14:46
  • $\begingroup$ What library are you using? tensorflow? Check this: tensorflow.org/guide/keras/save_and_serialize $\endgroup$ – Leevo May 21 at 14:58
  • $\begingroup$ I am using Keras Tensorflow $\endgroup$ – inquisitive May 21 at 15:01
  • $\begingroup$ Thank you so much! $\endgroup$ – inquisitive May 21 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.