2D-Input to LSTM in Keras

Question

I have following problem: I would like to feed LSTM with

train_datagen.flow_from_directory

The input is basically a spectrogram images converted from time-series into time-frequency-domain in PNG format that has a dimension of: timestep x frequency spectrum. 1 sample = 1 PNG image in uint8. In my example: 3601 timesteps with 217 frequency spectrum (=features) / timestep.

The spectrogram itself is just 1D, but I think "flow from directory" function was hard-coded to only prepare 3D image matrix and thus the input shape was becoming , which is totally pity because there are some people who are only working with purely greyscale uint8 image, and some who are working with multispectral and hyperspectral images.

My codes are following:

import keras
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Activation, Dropout, Flatten, Dense
from keras.layers import LSTM
from keras import optimizers
from keras import backend as K
import tensorflow as tf

img_width, img_height = 3601,217
train_data_dir = 'sensor1/training'
validation_data_dir = 'sensor1/validation'

num_classes = 10
nb_train_samples = num_classes*70
nb_validation_samples = num_classes*20
epochs = 20
batch_size = 10
input_shape = (img_width, img_height)

model.add(LSTM(units=256, input_shape= input_shape, return_sequences=True))
model.add(LSTM(units=128, return_sequences=True))
model.add(LSTM(units=64))
model.add(Dense(128))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

train_datagen = ImageDataGenerator(rescale = 1. / 255)
test_datagen = ImageDataGenerator(rescale = 1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size)

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size)

model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    callbacks=[plot_losses],
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

And then as soon as I run the program, of course it gives an error message::

**ValueError: Error when checking input: expected lstm_50_input to have 3 dimensions, but got array with shape (10, 3601, 217, 3)**

The message:

expected lstm_50_input to have 3 dimensions, but got array with shape (10, 3601, 217, 3)

clearly suggests it does not agree with my definition of input shape of: (3601, 217)

Any idea to easy fix the problem? Thanks in advance.

Answer 1

Why do you define the last dimension of input_shape as $3$? Just put your desired input dimensions accordingly and it should be fine:

input_shape = (img_width, img_height)

Update with the full code:

The best way would be to use TimeseriesGenerator instead of ImageDataGenerator but there seems there not flow_from_directory method meeting your needs. So, I think the best solution is to squeeze the last dimension of the generator output. Also, you have a color_mode option that allows to generate a 1-channel only tensor for grayscale images. Full code of concerned parts:

model = Sequential()
model.add(Lambda(lambda x: x[:,:,:,0], input_shape=(*input_shape, 1)))
model.add(LSTM(units=256, return_sequences=True))
model.add(LSTM(units=128, return_sequences=True))
model.add(LSTM(units=64))
model.add(Dense(128))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])


train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    color_mode='grayscale')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    color_mode='grayscale')