# How feed a numpy array in batches in Keras

I have the data in the following format:

1: DATA NUMPY ARRAY (trainX)

A numpy array of a set of numpy array of 3d np arrays. To be more articulate the format is: [[3d data], [3d data], [3d data], [3d data], ...]

2: TARGET NUMPY ARRAY (trainY)

This consists of a numpy array of the corresponding target values for the above array.

The format is [target1, target2, target3]

The numpy array gets quite large, and considering that I'll be using a deep neural network, there will be many parameters that would need fitting into the memory as well.

How can I push the numpy arrays in batches for trainX and trainY

You should implement a generator and feed it to model.fit_generator().

Your generator may look like this:

def batch_generator(X, Y, batch_size = BATCH_SIZE):
indices = np.arange(len(X))
batch=[]
while True:
# it might be a good idea to shuffle your data before each epoch
np.random.shuffle(indices)
for i in indices:
batch.append(i)
if len(batch)==batch_size:
yield X[batch], Y[batch]
batch=[]



And then, somewhere in your code:

train_generator = batch_generator(trainX, trainY, batch_size = 64)
model.fit_generator(train_generator , ....)


UPD.: I order to avoid placing all your data into memory beforehand, you can modify the generator to consume only the identifiers of your data-set and then load your data on-demand:

def batch_generator(ids, batch_size = BATCH_SIZE):
batch=[]
while True:
np.random.shuffle(ids)
for i in ids:
batch.append(i)
if len(batch)==batch_size:
batch=[]



def load_data(ids):
X = []
Y = []

for i in ids:
# read one or more samples from your storage, do pre-processing, etc.
# for example:
...
y = targets[i]

X.append(x)
Y.append(y)

return np.array(X), np.array(Y)

• Thanks so much for answering. Wouldn't batch_generator require X and Y as a numpy array already loaded in the memory which would still take up half the space. Is there a solution for that? Mar 20 '19 at 1:07
• yes, there is a solution for that. please, see the updated comment. Mar 20 '19 at 1:36

Another approach using Keras Sequence class:

class DataGenerator(keras.utils.Sequence):
def __init__(self, x_data, y_data, batch_size):
self.x, self.y = x_data, y_data
self.batch_size = batch_size
self.num_batches = np.ceil(len(x_data) / batch_size)
self.batch_idx = np.array_split(range(len(x_data)), self.num_batches)

def __len__(self):
return len(self.batch_idx)

def __getitem__(self, idx):
batch_x = self.x[self.batch_idx[idx]]
batch_y = self.y[self.batch_idx[idx]]
return batch_x, batch_y

train_generator = DataGenerator(x_train, y_train, batch_size = 128)
model.fit(train_generator,...)

$$$$
`