How to use TimeDistributed fo CNN+LSTM?

Question

I am trying to classify 6 classes time-frequency domain signal (STFT spectrogram) with a size of 3601x217 pixels. Assume that for each classes have 70 training samples, 20 validation samples, and 10 test samples. Each sample is a PNG image of size 3601x217x1 (grayscale), while in the future I am thinking to expand this into 3601x217x4 (3 color channels + alpha). But right now, I am focusing on grayscale first.

After trying pure LSTM, I found out that the model overfits quickly (training accuracy > 90%, but val_acc stucks at 20%). ow I'd like to try out Time Distributed CNN+LSTM.

Following is the code:

img_width, img_height = 3601,217
train_data_dir = 'sensor1/training'
validation_data_dir = 'sensor1/validation'

num_classes = 6
nb_train_samples = 70
nb_validation_samples = 20
epochs = 50
batch_size = 5

if K.image_data_format() == 'channels_first':
    input_shape = (3, img_width, img_height)
else:
    input_shape = (img_width, img_height, 3)

model = Sequential()
model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu', input_shape = input_shape)))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Conv2D(32, (3,3), padding='same', strides=(2,2), activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Conv2D(64, (3,3), padding='same', strides=(2,2), activation='relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(Dropout(0.5))
model.add(TimeDistributed(Flatten()))
model.add(LSTM(units=128, return_sequences=False))
model.add(LSTM(units=64, return_sequences=False))
model.add(Dense(32))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

train_datagen = ImageDataGenerator(rescale = 1. / 255)
test_datagen = ImageDataGenerator(rescale = 1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir, target_size=(img_width, img_height),
    batch_size=batch_size, color_mode='grayscale')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir, target_size=(img_width, img_height),
    batch_size=batch_size, color_mode='grayscale')

model.fit_generator(
    train_generator, steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs, callbacks=[plot_losses],
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

The code gives following error:

File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py", line 496, in _make_train_function raise RuntimeError('You must compile your model before using it.')

RuntimeError: You must compile your model before using it.

What's wrong with the code? I actually compiled the model before calling model.fit_generator:

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Answer 1

You are specifying the input_shape inside the Conv2D layer, meaning: model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu', input_shape = input_shape)))

Instead you should write: model.add(TimeDistributed(Conv2D(16, (3,3), padding='same', strides=(2,2), activation='relu'), input_shape = input_shape))

or more clearly: model.add(TimeDistributed(Conv2D(...), input_shape=input_shape))

Furthermore I should note that you are using a TimeDistributed with a Conv2D. This implies that your input_shape should be like this (timesteps, dim1_size, dim2_size, n_channels).

You are using: input_shape=(img_width, img_height, 3)

If you want to take the img_width as timesteps you should use TimeDistributed with Conv1D.

To summarize, always consider that a TimeDistibuted layer adds an extra dimension to the input_shape of its argument-layer.

Lastly, your first LSTM layer with return_sequences=False will raise an error. You must give it a True value.