0
$\begingroup$

I want to make an LSTM network and I have quite a long signal that I want to use as my training data

  • My X_train is a CSV-file which contains 12 signals with a length of 54 837 488
  • My y_train is an array that contains a One Hot encoded signal (9 categories) with a length of 54 837 488

If I try to upload the CSV-file as a data frame or as an array in Python I exceed the limit of RAM, so my thought is to use fit_generator(). I have used this before when making image models but then I just used some prebuild generators, but I can't really find any prebuild generators for signals so I decided to try to make one myself.

def generate_data_to_model(y_train):
    while True:
        with open("/mypath/myData.csv") as f:
            for line in f:
                x= line.rstrip('\n').split(",")
                x= np.asarray(x)
                x=x[1:]
                x= x.reshape(1,1,12)
                yield (x, y_train[line])


model = Sequential()
model.add(LSTM(32, input_shape=(1, 12)))
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(9, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer="Adam", metrics=['acc'])

model.fit_generator(generate_data_to_model(y_train),
                    steps_per_epoch=1, epochs=2, verbose=1)

When I start training I get this error:

<ipython-input-29-33b2195fea44> in generate_data_to_model(y_train)
      8                 x= x.reshape(1,1,12)
 ---> 9                 yield (x, y_train[line])

only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid indices

$\endgroup$
0
$\begingroup$

The error Is pretty clear, you're trying to access an element in y_train using 'line' as an index, but line is the line you read from the file, not a number. I don't get what y_train should be and what rule it has in the function, since you don't use it and only return it, but from the rest of the code I suppose that you want to return a tuple containing the features and the target label, and since you used x=x[1:] to extract the features from the line I suppose that the first element might be the label so you could try rewriting your function like this:

def generate_data_to_model():
while True:
    with open("/mypath/myData.csv") as f:
        for line in f:
            x= line.rstrip('\n').split(",")
            x= np.asarray(x)
            y = x[0]
            x=x[1:]
            x= x.reshape(1,1,12)
            yield (x, y)
| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks! Yes, your rigth, I did a mistake by writing: y_train[line] $\endgroup$ – bjornsing Mar 20 at 9:52
  • $\begingroup$ The purpose of x=x[1:] was to remove the row index from the input signal to the model $\endgroup$ – bjornsing Mar 20 at 10:32
0
$\begingroup$

I found a solution to this problem with some good help from this article. What I found really convenient is to make two generators. One for features, or X, and one for labels, or y.

def generate_X():
    while True:
        with open("/mypath/myData.csv") as f:
            for line in f:
                x= line.rstrip('\n').split(",")
                x= np.asarray(x)
                x=x[1:]
                x= x.reshape(1,1,12)
                yield x

def generate_y():
    while True:
        for i in range(len(y_train)):
            y= y_train[i]
            yield y

And then I go ahead and use these two generators as input to my third generator which will make batches of a desired size to the fit_generator()

def batch_generator(batch_size, gen_x,gen_y): 

    batch_features = np.zeros((batch_size,1, 12))
    batch_labels = np.zeros((batch_size,9))

    while True:
        for i in range(batch_size):
            batch_features[i] = next(gen_x)
            batch_labels[i] = next(gen_y)
        yield batch_features, batch_labels

And last, but not least I am now able to use the batch_generator when training my model

model.fit_generator(batch_generator(128, generate_X(), generate_y()),
                    steps_per_epoch=(len(y_train)/128), epochs=2, verbose=1)
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.