0
$\begingroup$

I am building a model to predict if a video images are describing a sleepy person or awake person.

I have trained a CNN custom model to classify blink eyes or not.

Now it's time to join these Conv2D layers with a LSTM layers using TimeDistributed layers.

I would like that LSTM sees 4 frames before the current.

This is my ImageGenerator code

from tensorflow.keras.preprocessing.image import ImageDataGenerator

TRAINING_SET = 'train_set/'
TEST_SET = 'test_set/'

batch_size = 32
image_size = 64

train_datagen = ImageDataGenerator(rescale=1./255,
                                   rotation_range=10,
                                   width_shift_range=0.2,
                                   height_shift_range=0.2,
                                   shear_range=0.2)

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(TRAINING_SET,
                                                    target_size=(image_size, image_size),
                                                    batch_size=1,
                                                    class_mode='binary',
                                                    shuffle=False,
                                                    color_mode = "grayscale")

validation_generator = test_datagen.flow_from_directory(TEST_SET,
                                                        target_size=(image_size, image_size),
                                                        batch_size=1,
                                                        class_mode='binary',
                                                        shuffle=False,
                                                        color_mode = "grayscale")
# Found 10385 images belonging to 2 classes.
# Found 2002 images belonging to 2 classes.

and the model definition

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout, 
BatchNormalization, TimeDistributed, LSTM
from tensorflow.keras.callbacks import ModelCheckpoint, ReduceLROnPlateau

classifier = Sequential()

# CNN
classifier.add(TimeDistributed(Conv2D(filters=32, kernel_size = (3, 3), activation = "relu"), 
              input_shape=(None, 4, image_size, image_size, 1)))


classifier.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2)))) # esto he quitado para añadir batch
classifier.add(TimeDistributed(Conv2D(filters=64, kernel_size = (3, 3),
                       activation = "relu")))
classifier.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
classifier.add(TimeDistributed(Conv2D(filters=128, kernel_size = (3, 3),
                       activation = "relu")))
classifier.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
classifier.add(TimeDistributed((Flatten())))

# LSTM
classifier.add(LSTM(units=50, return_sequences=True, dropout=0.2))
classifier.add(LSTM(units=50, dropout=0.2, return_sequences=True))
classifier.add(LSTM(units=50, dropout=0.2, return_sequences=True))
classifier.add(LSTM(units=50))
classifier.add(Dense(units=1, activation="sigmoid"))

from tensorflow.keras.optimizers import Adam

opt = Adam(learning_rate=0.0001)
classifier.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])

history = classifier.fit(train_generator,
                     steps_per_epoch=int(10385  /batch_size),
                     epochs=1,
                     validation_data=validation_generator,
                     validation_steps=int(2002 /batch_size))

And when the fit starts it raises next error

expected time_distributed_62_input to have 5 dimensions, but got array with shape (None, None, None, None)

If I change my TimeDistributed input shape to something like (None, 1, 64, 64, 1) I cannot compile the model and the error throws input tensor must have rank 4

I know that TimeDistributed needs a 5D input like (samples, timesteps, width, height, channel) but what is the reason that I received (None, None, None, None) error?

If I execute next instruction

X, y = train_generator.next()

X shape is (1, 64, 64, 1)

Any idea that what I am doing wrong?

Thank you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.