How to arrange the dataset/images for CNN+LSTM

Question

I am working on an image classification problem using Transfer Learning with Resnet50 as base model (in Keras) (For example Class A and Class B).

There is a time factor involved in this classification. For example, I need sufficient evidence to make transition from one class to another.
So one of the thought came to my mind to make smooth transition is adding LSTM layer to the CNN layer (CNN+LSTM).

Need your help in understanding below queries.

1. Is there any way that I can add LSTM layer to the transfer learning process (assuming the CNN layer weights are not trainable)

2. How do I need to prepare the dataset (image frames). For example I have 10 videos each for class A and Class B. Do I need to keep the images in sequential order as it is in video. (As of now for the normal image classification, I have shuffled the image frames)

3. Any thought on building my own CNN + LSTM model.

A reference link would be great for 2 and 3.

Whether the image frames need to be updated as it is in the video ?

Any updates on this please.

Thank you, Deep Guy

Answer 1

I'll expand my answer if there is any interest. I propose building a simple test model using Vgg and two custom fully connected layers that end with a single sigmoid per the document below. If you prefer early fusion I recommend using stacked black and white images.
https://www.cs.cmu.edu/~rahuls/pub/cvpr2014-deepvideo-rahuls.pdf

if you then want to go the route of LSTM, then this graphic from Karpathy will help. Use the many to one architecture. The inputs to the RNNS or LSTMS vary by their implementation. You would most likely want to pass the final convolutional layer though a fully connect layer and then into the RNN/LSTM. https://karpathy.github.io/2015/05/21/rnn-effectiveness/