# Keras fit_generator() for long signals

I want to make an LSTM network and I have quite a long signal that I want to use as my training data

• My X_train is a CSV-file which contains 12 signals with a length of 54 837 488
• My y_train is an array that contains a One Hot encoded signal (9 categories) with a length of 54 837 488

If I try to upload the CSV-file as a data frame or as an array in Python I exceed the limit of RAM, so my thought is to use fit_generator(). I have used this before when making image models but then I just used some prebuild generators, but I can't really find any prebuild generators for signals so I decided to try to make one myself.

def generate_data_to_model(y_train):
while True:
with open("/mypath/myData.csv") as f:
for line in f:
x= line.rstrip('\n').split(",")
x= np.asarray(x)
x=x[1:]
x= x.reshape(1,1,12)
yield (x, y_train[line])

model = Sequential()

model.fit_generator(generate_data_to_model(y_train),
steps_per_epoch=1, epochs=2, verbose=1)


When I start training I get this error:

<ipython-input-29-33b2195fea44> in generate_data_to_model(y_train)
8                 x= x.reshape(1,1,12)
---> 9                 yield (x, y_train[line])


only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid indices

I found a solution to this problem with some good help from this article. What I found really convenient is to make two generators. One for features, or X, and one for labels, or y.

def generate_X():
while True:
with open("/mypath/myData.csv") as f:
for line in f:
x= line.rstrip('\n').split(",")
x= np.asarray(x)
x=x[1:]
x= x.reshape(1,1,12)
yield x


def generate_y():
while True:
for i in range(len(y_train)):
y= y_train[i]
yield y


And then I go ahead and use these two generators as input to my third generator which will make batches of a desired size to the fit_generator()

def batch_generator(batch_size, gen_x,gen_y):

batch_features = np.zeros((batch_size,1, 12))
batch_labels = np.zeros((batch_size,9))

while True:
for i in range(batch_size):
batch_features[i] = next(gen_x)
batch_labels[i] = next(gen_y)
yield batch_features, batch_labels


And last, but not least I am now able to use the batch_generator when training my model

model.fit_generator(batch_generator(128, generate_X(), generate_y()),
steps_per_epoch=(len(y_train)/128), epochs=2, verbose=1)


The error Is pretty clear, you're trying to access an element in y_train using 'line' as an index, but line is the line you read from the file, not a number. I don't get what y_train should be and what rule it has in the function, since you don't use it and only return it, but from the rest of the code I suppose that you want to return a tuple containing the features and the target label, and since you used x=x[1:] to extract the features from the line I suppose that the first element might be the label so you could try rewriting your function like this:

def generate_data_to_model():
while True:
with open("/mypath/myData.csv") as f:
for line in f:
x= line.rstrip('\n').split(",")
x= np.asarray(x)
y = x[0]
x=x[1:]
x= x.reshape(1,1,12)
yield (x, y)

• Thanks! Yes, your rigth, I did a mistake by writing: y_train[line] – bjornsing Mar 20 '20 at 9:52
• The purpose of x=x[1:] was to remove the row index from the input signal to the model – bjornsing Mar 20 '20 at 10:32