So I'm working with a rather large dataset (perhaps not really by ML standards - but too big to fit into my computer's RAM at any rate). And so, I train the model by successively loading a subsample of the dataset and calling model.fit using that data. This means I call model.fit quite regularly and it seems to take significantly more time to initialize model.fit than it does to perform 10 epochs of training (on the data subset).

I'm wondering if there any solutions or tricks to improve initialization time? Should I input the data in a certain way (say tensorflow dataset object instead of numpy arrays) or employ model.train_on_batch in some fashion instead?

  • 1
    $\begingroup$ I think in your use-case it would make more sense to use a create a custom data generator (or an existing one, such as ImageDataGenerator, depending on your data) which will load data into memory as needed. $\endgroup$
    – Oxbowerce
    Apr 9 at 16:21
  • $\begingroup$ See for instance this question $\endgroup$ Apr 9 at 18:47


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.