# 2D-Input to LSTM in Keras

I have following problem: I would like to feed LSTM with

train_datagen.flow_from_directory

The input is basically a spectrogram images converted from time-series into time-frequency-domain in PNG format that has a dimension of: timestep x frequency spectrum. 1 sample = 1 PNG image in uint8. In my example: 3601 timesteps with 217 frequency spectrum (=features) / timestep.

The spectrogram itself is just 1D, but I think "flow from directory" function was hard-coded to only prepare 3D image matrix and thus the input shape was becoming , which is totally pity because there are some people who are only working with purely greyscale uint8 image, and some who are working with multispectral and hyperspectral images.

My codes are following:

import keras
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Activation, Dropout, Flatten, Dense
from keras.layers import LSTM
from keras import optimizers
from keras import backend as K
import tensorflow as tf

img_width, img_height = 3601,217
train_data_dir = 'sensor1/training'
validation_data_dir = 'sensor1/validation'

num_classes = 10
nb_train_samples = num_classes*70
nb_validation_samples = num_classes*20
epochs = 20
batch_size = 10
input_shape = (img_width, img_height)

model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

train_datagen = ImageDataGenerator(rescale = 1. / 255)
test_datagen = ImageDataGenerator(rescale = 1. / 255)

train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size)

validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size)

model.fit_generator(
train_generator,
steps_per_epoch=nb_train_samples // batch_size,
epochs=epochs,
callbacks=[plot_losses],
validation_data=validation_generator,
validation_steps=nb_validation_samples // batch_size)


And then as soon as I run the program, of course it gives an error message::

**ValueError: Error when checking input: expected lstm_50_input to have 3 dimensions, but got array with shape (10, 3601, 217, 3)**


The message:

expected lstm_50_input to have 3 dimensions, but got array with shape (10, 3601, 217, 3)

clearly suggests it does not agree with my definition of input shape of: (3601, 217)

Any idea to easy fix the problem? Thanks in advance.

Why do you define the last dimension of input_shape as $$3$$? Just put your desired input dimensions accordingly and it should be fine:

input_shape = (img_width, img_height)


Update with the full code:

The best way would be to use TimeseriesGenerator instead of ImageDataGenerator but there seems there not flow_from_directory method meeting your needs. So, I think the best solution is to squeeze the last dimension of the generator output. Also, you have a color_mode option that allows to generate a 1-channel only tensor for grayscale images. Full code of concerned parts:

model = Sequential()
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

train_generator = train_datagen.flow_from_directory(
train_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
color_mode='grayscale')

validation_generator = test_datagen.flow_from_directory(
validation_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
color_mode='grayscale')

• Yes, I tried that too. Please read the last 5 sentences in my post. The new error when I define input_shape = (img_width, img_height) was "expected lstm_50_input to have 3 dimensions, but got array with shape (10, 3601, 217, 3)". – user2754279 Sep 5 '19 at 10:46
• (10, 3601, 217, 3) means 10 batches, 3601 timesteps, 217 frequency spectrums, and 3 (RGB)-layers. Currently I don't need 3-RGB layers as the PNG files are saved as uint8 (greyscale). So I believe the "flow from directory" function must have converted it into 3 RGB layers. – user2754279 Sep 5 '19 at 10:50
• I have read your 5 sentences. The code with input_shape = (img_width, img_height) runs perfectly fine on my side. – Elliot Sep 5 '19 at 11:05
• Hi, I edited my post to show the full code. Can you run my code in your Python? – user2754279 Sep 5 '19 at 13:14
• Oh ok I see now with the generator. Edited the answer with code. – Elliot Sep 5 '19 at 14:00