3
$\begingroup$

I have the data in the following format:

1: DATA NUMPY ARRAY (trainX)

A numpy array of a set of numpy array of 3d np arrays. To be more articulate the format is: [[3d data], [3d data], [3d data], [3d data], ...]

2: TARGET NUMPY ARRAY (trainY)

This consists of a numpy array of the corresponding target values for the above array.

The format is [target1, target2, target3]

The numpy array gets quite large, and considering that I'll be using a deep neural network, there will be many parameters that would need fitting into the memory as well.

How can I push the numpy arrays in batches for trainX and trainY

$\endgroup$
5
$\begingroup$

You should implement a generator and feed it to model.fit_generator().

Your generator may look like this:

def batch_generator(X, Y, batch_size = BATCH_SIZE):
    indices = np.arange(len(X)) 
    batch=[]
    while True:
            # it might be a good idea to shuffle your data before each epoch
            np.random.shuffle(indices) 
            for i in indices:
                batch.append(i)
                if len(batch)==batch_size:
                    yield X[batch], Y[batch]
                    batch=[]

And then, somewhere in your code:

train_generator = batch_generator(trainX, trainY, batch_size = 64)
model.fit_generator(train_generator , ....)

UPD.: I order to avoid placing all your data into memory beforehand, you can modify the generator to consume only the identifiers of your data-set and then load your data on-demand:

def batch_generator(ids, batch_size = BATCH_SIZE):
    batch=[]
    while True:
            np.random.shuffle(ids) 
            for i in ids:
                batch.append(i)
                if len(batch)==batch_size:
                    yield load_data(batch)
                    batch=[]

Your loader function may look like this:

def load_data(ids):
   X = []
   Y = []

   for i in ids:
     # read one or more samples from your storage, do pre-processing, etc.
     # for example:
     x = imread(f'image_{i}.jpg')
     ...
     y = targets[i]

     X.append(x)
     Y.append(y)

   return np.array(X), np.array(Y)
$\endgroup$
2
  • $\begingroup$ Thanks so much for answering. Wouldn't batch_generator require X and Y as a numpy array already loaded in the memory which would still take up half the space. Is there a solution for that? $\endgroup$
    – thegravity
    Mar 20 '19 at 1:07
  • $\begingroup$ yes, there is a solution for that. please, see the updated comment. $\endgroup$
    – M0nZDeRR
    Mar 20 '19 at 1:36
0
$\begingroup$

Another approach using Keras Sequence class:

class DataGenerator(keras.utils.Sequence):
  def __init__(self, x_data, y_data, batch_size):
    self.x, self.y = x_data, y_data
    self.batch_size = batch_size
    self.num_batches = np.ceil(len(x_data) / batch_size)
    self.batch_idx = np.array_split(range(len(x_data)), self.num_batches)

  def __len__(self):
    return len(self.batch_idx)

  def __getitem__(self, idx):
    batch_x = self.x[self.batch_idx[idx]]
    batch_y = self.y[self.batch_idx[idx]]
    return batch_x, batch_y

train_generator = DataGenerator(x_train, y_train, batch_size = 128)
model.fit(train_generator,...)

```
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.