1
$\begingroup$

I am trying to classify 6 classes time-frequency domain signal (STFT spectrogram) with a size of 3601x217 pixels. Assume that for each classes have 70 training samples, 20 validation samples, and 10 test samples. Each sample is a PNG image of size 3601x217x1 (grayscale), while in the future I am thinking to expand this into 3601x217x4 (3 color channels + alpha). But right now, I am focusing on grayscale first.

After trying pure LSTM, I found out that the model overfits quickly (training accuracy > 90%, but val_acc stucks at 20%). ow I'd like to try out Time Distributed CNN+LSTM.

Following is the code:

img_width, img_height = 3601,217
train_data_dir = 'sensor1/training'
validation_data_dir = 'sensor1/validation'

num_classes = 6
nb_train_samples = 70
nb_validation_samples = 20
epochs = 50
batch_size = 5

if K.image_data_format() == 'channels_first':
    input_shape = (3, img_width, img_height)
else:
    input_shape = (img_width, img_height, 3)

model = Sequential()
model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu', input_shape = input_shape)))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Conv2D(32, (3,3), padding='same', strides=(2,2), activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Conv2D(64, (3,3), padding='same', strides=(2,2), activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(units=128, return_sequences=False))
model.add(LSTM(units=64, return_sequences=False))
model.add(Dense(32))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

train_datagen = ImageDataGenerator(rescale = 1. / 255)
test_datagen = ImageDataGenerator(rescale = 1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir, target_size=(img_width, img_height),
    batch_size=batch_size, color_mode='grayscale')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir, target_size=(img_width, img_height),
    batch_size=batch_size, color_mode='grayscale')

model.fit_generator(
    train_generator, steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs, callbacks=[plot_losses],
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

The code gives following error:

File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py", line 496, in _make_train_function raise RuntimeError('You must compile your model before using it.')

RuntimeError: You must compile your model before using it.

What's wrong with the code? I actually compiled the model before calling model.fit_generator:

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
$\endgroup$
-1
$\begingroup$

You are specifying the input_shape inside the Conv2D layer, meaning: model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu', input_shape = input_shape)))

Instead you should write: model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu'), input_shape = input_shape))

or more clearly: model.add(TimeDistributed(Conv2D(...), input_shape=input_shape))

Furthermore I should note that you are using a TimeDistributed with a Conv2D. This implies that your input_shape should be like this (timesteps, dim1_size, dim2_size, n_channels).

You are using: input_shape=(img_width, img_height, 3)

If you want to take the img_width as timesteps you should use TimeDistributed with Conv1D.

To summarize, always consider that a TimeDistibuted layer adds an extra dimension to the input_shape of its argument-layer.

Lastly, your first LSTM layer with return_sequences=False will raise an error. You must give it a True value.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.