Deep learning with Tensorflow: training with big data sets

Question

Goal
I am trying to build a neural network that recognizes multiple label within a given image. I started with a database composed of 1800 images (each image is an array of shape (204,204,3). I trained my model and concluded that data used wasn't enough in order to build a good model ( with respect to chosen metric). So i decided to apply data augmentation technique in order to get more images. I managed to get 25396 images ( all of them are of shape (204,204,3)). I stored all of them in arrays . I obtained (X,Y) where X are the training examples (is an array of shape (25396,204,204,3)) and Y are the labels ( an array of shape (25396,39) : the number 39 refers to the possible labels in a given image).
Issues
My data (X,Y) weights approximately arround 26 giga bytes. I successfully managed to use them . However, when i try to do manipulation (like permutations) I encounter memory Error in python. Exemple
1. I started jupyter and successfully imported my data (X,Y)

x=np.load('x.npy')
y=np.load('y.npy')

---

output: x is an np.array of shape (25396,204,204,3) and y is an np.array of shape (25396,39).

2. I divide my dataSet in train and test by using sklearn built in function train_test_split

X_train, X_valid, Y_train, Y_valid= `train_test_split(x_train,y_train_augmented,test_size=0.3, random_state=42)`

---

output
-------------testing size of different elements et toplogie: 
-------------x size: (25396, 204, 204, 3)
-------------y size: (25396, 39) 
-------------X_train size:  (17777, 204, 204, 3)
-------------X_valid size:  (7619, 204, 204, 3)
-------------Y_train size:  (17777, 39)
-------------Y_valid size:  (7619, 39)

3. I am creating a list composed of random batches extracted from (X,Y) and then iterate over the batches in order to complete the learning process for a given epoch :'this opperation is done in each epoch of the training part. Here is the function used in order to create the list of random batches:

def random_mini_batches(X, Y, mini_batch_size = 64, seed = 0):
    """
    Creates a list of random minibatches from (X, Y)

    Arguments:
    X -- input data, of shape (input size, number of examples)
    Y -- true "label" vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)
    mini_batch_size -- size of the mini-batches, integer

    Returns:
    mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)
    """


    np.random.seed(seed)            
    m = X.shape[0]                  
    mini_batches = []
    # Step 1: Shuffle (X, Y)
    permutation = list(np.random.permutation(m))



    shuffled_X = X[permutation,:]
    shuffled_Y = Y[permutation,:]


    # Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.
    num_complete_minibatches = floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning
    for k in range(0, num_complete_minibatches):


        mini_batch_X = shuffled_X[k * mini_batch_size : (k + 1) * mini_batch_size, :]
        mini_batch_Y = shuffled_Y[k * mini_batch_size : (k + 1) * mini_batch_size, :]
        mini_batch = (mini_batch_X, mini_batch_Y)
        mini_batches.append(mini_batch)
        '''
        mini_batches.append((X[permutation,:][k * mini_batch_size : (k + 1) * mini_batch_size, :], Y[permutation,:][k * mini_batch_size : (k + 1) * mini_batch_size, :]))
        '''
    # Handling the end case (last mini-batch < mini_batch_size)
    if m % mini_batch_size != 0:
        ### START CODE HERE ### (approx. 2 lines)
        mini_batch_X = shuffled_X[ num_complete_minibatches * mini_batch_size:, :]
        mini_batch_Y = shuffled_Y[ num_complete_minibatches * mini_batch_size:, :]
        ### END CODE HERE ###
        mini_batch = (mini_batch_X, mini_batch_Y)
        mini_batches.append(mini_batch)
        '''
        mini_batches.append((X[permutation,:][ num_complete_minibatches * mini_batch_size:, :], Y[permutation,:][ num_complete_minibatches * mini_batch_size:, :]))
        '''
    shuffled_X=None
    shuffled_Y=None
    return mini_batches

4. I am creating a loop (of 4 iterations) and i am testing the random_mini_batch function in each iteration. At the end of each iteration I am assigning None values to the list of mini_batches in order to liberate memory and redo the random_mini_batch_function in the next iteration .So these line of codes works fine and I ve got no memory issues:

minibatch_size=32
seed=2
for i in range(4):
    seed=seed+1
    minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)

    minibatches=None 
    minibatches_valid=create_mini_batches(X_valid, Y_valid, minibatch_size)
    print(i)
minibatches_valid=None

5. If I add iteration over the different batches! then I am getting a memory issue. In other words, if a run this code i get an error:

minibatch_size=32
seed=2
for i in range(4):
    seed=seed+1
    minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)
    #added code: iteration over mini_batches
    for minibatch in minibatches:
                print('batch training number ')
    #end of added code        
    minibatches=None 
    minibatches_valid=create_mini_batches(X_valid, Y_valid, minibatch_size)
    print(i)
    minibatches_valid=None

---

MemoryError                               Traceback (most recent call last)
<ipython-input-13-9c1942cdf0bc> in <module>()
      3 for i in range(4):
      4     seed=seed+1
----> 5     minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)
      6     
      7     for minibatch in minibatches:

<ipython-input-3-2056fee14def> in random_mini_batches(X, Y, mini_batch_size, seed)

     23 
---> 24     shuffled_X = X[permutation,:]
     25     shuffled_Y = Y[permutation,:]
     26 

MemoryError: 

Does any one knows what's the issue with np.arrays ? And why does the simple fact of adding an loop (iterating over the list of batches) result in a memory error.

Questions

1.Is it a good idea to load the whole dataset and then proceed to training? ( I need to create random batches in each epoch, so I don't see how to do so if the data is not preloaded ? You take random mini-batches from preloaded data, right?) 2. Are there any possible solutions guys?

Answer 1

Does any one knows what's the issue with np.arrays ?

shuffled_X = X[permutation,:] makes copies, so it will allocate new array each time you do a permutation and blow up your memory.

If you don't have problems with storing whole dataset in memory you should be fine if you create batches just by using random indices, not shuffling the entire data matrix (np.random.choice is your friend).

Is it a good idea to load the whole dataset and then proceed to training?

If your data fits into memory, then yes. You might want to try to learn what to do when that's not the case though - I personally find Keras - stuff from keras.preprocessing.image useful for that (at least for loading images).