I'm using the MLPRegressor in sklearn to train a network with approx 1000 inputs and a continuous output variable. Essentially, the issue is one of image classification (1000 pixels) with the distribution of the pixels related to a continuous variable output.

I can generate an arbitrarily large training set, but run into memory problems running this on a single machine when loading arrays into numpy. I've selected hyper parameters that give a reasonable result with a limited (5k) training set, but am trying to work out the best way to train the network with two orders of magnitude more data. Is there a way to pass mini-batches to sklearn myself so that I can manage the memory and IO? Is training and retraining the network helpful if I can setup a means to ensure I read batches for each training step in a randomised fashion?

I'm guessing this is a common problem, but I'm struggling to find a sensible solution.


1 Answer 1


Scikit-learn MLPRegressor has a .partial_fit method for training on batches which will overcome this memory issue.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.