# Q: Training a CNN-LSTM on video inputs

Hello everyone!

I implemented the following model, for action classification from videos, where each frame is 224x224x3, a video consists of 30 frames and I have 6 classes:

inputs = Input(shape=(30, 224, 224, 3))
model = TimeDistributed(Conv2D(filters=32, kernel_size=(3, 3), strides=2, padding="same", activation="relu"))(inputs)
model = TimeDistributed(MaxPooling2D(pool_size=(2, 2), strides=2, padding="same"))(model)
model = TimeDistributed(Dropout(0.25))(model)
model = TimeDistributed(Conv2D(filters=64, kernel_size=(3, 3), strides=2, padding="same", activation="relu"))(model)
model = TimeDistributed(MaxPooling2D(pool_size=(2, 2), strides=2, padding="same"))(model)
model = TimeDistributed(Dropout(0.25))(model)
model = TimeDistributed(Conv2D(filters=128, kernel_size=(3, 3), strides=2, padding="same", activation="relu"))(model)
model = TimeDistributed(MaxPooling2D(pool_size=(2, 2), strides=2, padding="same"))(model)
model = TimeDistributed(Dropout(0.25))(model)
model = TimeDistributed(Flatten())(model)
model = LSTM(128, return_sequences=False)(model)
model = Dense(128)(model)
model = Dropout(0.25)(model)
outputs = Dense(6, activation="softmax")(model)
model = Model(inputs=inputs, outputs=outputs)


My problem is that I don't know if I am training this model PROPERLY.

I tried the following:

1. Generate numpy arrays from extracted frames of both the train and test videos, these numpy arrays are called: train_data with the shape being: (1260, 224, 224, 3), test_data with the shape being: (360, 224, 224, 3)
2. Generate numpy arrays that contain the one-hot encoded data from classes corresponding to the videos, these numpy arrays are called train_labels with the shape being: (1260, 6), test_labels with the shape being: (360, 6)
3. Use train_data with train_labels and test_data with test_labels as parameters for Keras's built-in timeseries_dataset_from_array function, where the sequence_length=30, sequence_stride=30, sampling_rate=1, batch_size=8. This grants me 2 tf.data.Dataset, these are called: train_dataset and test_dataset
4. Use the train_dataset with Keras's built-in model.fit() function in the following format: model.fit(train_dataset, epochs=60)

During the training I observed that the model finishes learning in 2 minutes. I find this odd, since training a pure CNN took more time to train.

Also when I use model.predict(test_dataset) it doesn't give accurate predictions, even though the model's accuracy is: 98%

Am I completely in the wrong if I want to generate the train and test data with Keras's built-in timeseries_dataset_from_array function, or did I just overlook something?