I am working on an image classification problem using Transfer Learning with Resnet50 as base model (in Keras) (For example Class A and Class B).

There is a time factor involved in this classification. For example, I need sufficient evidence to make transition from one class to another.
So one of the thought came to my mind to make smooth transition is adding LSTM layer to the CNN layer (CNN+LSTM).

Need your help in understanding below queries.

  1. Is there any way that I can add LSTM layer to the transfer learning process (assuming the CNN layer weights are not trainable)

  2. How do I need to prepare the dataset (image frames). For example I have 10 videos each for class A and Class B. Do I need to keep the images in sequential order as it is in video. (As of now for the normal image classification, I have shuffled the image frames)

  3. Any thought on building my own CNN + LSTM model.

A reference link would be great for 2 and 3.

Whether the image frames need to be updated as it is in the video ?

Any updates on this please.

Thank you, Deep Guy


1 Answer 1


I'll expand my answer if there is any interest. I propose building a simple test model using Vgg and two custom fully connected layers that end with a single sigmoid per the document below. If you prefer early fusion I recommend using stacked black and white images.

various options for setting up video classification

if you then want to go the route of LSTM, then this graphic from Karpathy will help. Use the many to one architecture. The inputs to the RNNS or LSTMS vary by their implementation. You would most likely want to pass the final convolutional layer though a fully connect layer and then into the RNN/LSTM. https://karpathy.github.io/2015/05/21/rnn-effectiveness/

The blue is the output, red the CNN, Green RNN or LSTM


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.