Goal
I am trying to build a neural network that recognizes multiple label within a given image. I started with a database composed of 1800 images (each image is an array of shape (204,204,3). I trained my model and concluded that data used wasn't enough in order to build a good model ( with respect to chosen metric). So i decided to apply data augmentation technique in order to get more images. I managed to get 25396 images ( all of them are of shape (204,204,3)).
I stored all of them in arrays . I obtained (X,Y) where X are the training examples (is an array of shape (25396,204,204,3)) and Y are the labels ( an array of shape (25396,39) : the number 39 refers to the possible labels in a given image).
Issues
My data (X,Y) weights approximately arround 26 giga bytes. I successfully managed to use them . However, when i try to do manipulation (like permutations) I encounter memory Error in python.
Exemple
1. I started jupyter and successfully imported my data (X,Y)
x=np.load('x.npy')
y=np.load('y.npy')
output: x is an np.array of shape (25396,204,204,3) and y is an np.array of shape (25396,39).
2. I divide my dataSet in train and test by using sklearn built in function train_test_split
X_train, X_valid, Y_train, Y_valid= `train_test_split(x_train,y_train_augmented,test_size=0.3, random_state=42)`
output
-------------testing size of different elements et toplogie:
-------------x size: (25396, 204, 204, 3)
-------------y size: (25396, 39)
-------------X_train size: (17777, 204, 204, 3)
-------------X_valid size: (7619, 204, 204, 3)
-------------Y_train size: (17777, 39)
-------------Y_valid size: (7619, 39)
3. I am creating a list composed of random batches extracted from (X,Y) and then iterate over the batches in order to complete the learning process for a given epoch :'this opperation is done in each epoch of the training part. Here is the function used in order to create the list of random batches:
def random_mini_batches(X, Y, mini_batch_size = 64, seed = 0):
"""
Creates a list of random minibatches from (X, Y)
Arguments:
X -- input data, of shape (input size, number of examples)
Y -- true "label" vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)
mini_batch_size -- size of the mini-batches, integer
Returns:
mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)
"""
np.random.seed(seed)
m = X.shape[0]
mini_batches = []
# Step 1: Shuffle (X, Y)
permutation = list(np.random.permutation(m))
shuffled_X = X[permutation,:]
shuffled_Y = Y[permutation,:]
# Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.
num_complete_minibatches = floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning
for k in range(0, num_complete_minibatches):
mini_batch_X = shuffled_X[k * mini_batch_size : (k + 1) * mini_batch_size, :]
mini_batch_Y = shuffled_Y[k * mini_batch_size : (k + 1) * mini_batch_size, :]
mini_batch = (mini_batch_X, mini_batch_Y)
mini_batches.append(mini_batch)
'''
mini_batches.append((X[permutation,:][k * mini_batch_size : (k + 1) * mini_batch_size, :], Y[permutation,:][k * mini_batch_size : (k + 1) * mini_batch_size, :]))
'''
# Handling the end case (last mini-batch < mini_batch_size)
if m % mini_batch_size != 0:
### START CODE HERE ### (approx. 2 lines)
mini_batch_X = shuffled_X[ num_complete_minibatches * mini_batch_size:, :]
mini_batch_Y = shuffled_Y[ num_complete_minibatches * mini_batch_size:, :]
### END CODE HERE ###
mini_batch = (mini_batch_X, mini_batch_Y)
mini_batches.append(mini_batch)
'''
mini_batches.append((X[permutation,:][ num_complete_minibatches * mini_batch_size:, :], Y[permutation,:][ num_complete_minibatches * mini_batch_size:, :]))
'''
shuffled_X=None
shuffled_Y=None
return mini_batches
4. I am creating a loop (of 4 iterations) and i am testing the random_mini_batch function in each iteration. At the end of each iteration I am assigning None values to the list of mini_batches in order to liberate memory and redo the random_mini_batch_function in the next iteration .So these line of codes works fine and I ve got no memory issues:
minibatch_size=32
seed=2
for i in range(4):
seed=seed+1
minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)
minibatches=None
minibatches_valid=create_mini_batches(X_valid, Y_valid, minibatch_size)
print(i)
minibatches_valid=None
5. If I add iteration over the different batches! then I am getting a memory issue. In other words, if a run this code i get an error:
minibatch_size=32
seed=2
for i in range(4):
seed=seed+1
minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)
#added code: iteration over mini_batches
for minibatch in minibatches:
print('batch training number ')
#end of added code
minibatches=None
minibatches_valid=create_mini_batches(X_valid, Y_valid, minibatch_size)
print(i)
minibatches_valid=None
MemoryError Traceback (most recent call last)
<ipython-input-13-9c1942cdf0bc> in <module>()
3 for i in range(4):
4 seed=seed+1
----> 5 minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)
6
7 for minibatch in minibatches:
<ipython-input-3-2056fee14def> in random_mini_batches(X, Y, mini_batch_size, seed)
23
---> 24 shuffled_X = X[permutation,:]
25 shuffled_Y = Y[permutation,:]
26
MemoryError:
Does any one knows what's the issue with np.arrays ? And why does the simple fact of adding an loop (iterating over the list of batches) result in a memory error.
Questions
1.Is it a good idea to load the whole dataset and then proceed to training? ( I need to create random batches in each epoch, so I don't see how to do so if the data is not preloaded ? You take random mini-batches from preloaded data, right?) 2. Are there any possible solutions guys?
