I'm using the MLPRegressor in sklearn to train a network with approx 1000 inputs and a continuous output variable. Essentially, the issue is one of image classification (1000 pixels) with the distribution of the pixels related to a continuous variable output.
I can generate an arbitrarily large training set, but run into memory problems running this on a single machine when loading arrays into numpy. I've selected hyper parameters that give a reasonable result with a limited (5k) training set, but am trying to work out the best way to train the network with two orders of magnitude more data. Is there a way to pass mini-batches to sklearn myself so that I can manage the memory and IO? Is training and retraining the network helpful if I can setup a means to ensure I read batches for each training step in a randomised fashion?
I'm guessing this is a common problem, but I'm struggling to find a sensible solution.
