Using RNN (LSTM) for Gesture Recognition System

Question

I'm trying to build a gesture recognition system for classifying ASL (American Sign Language) Gestures, so my input is supposed to be a sequence of frames either from a camera or a video file then it detects the sequence and maps it to it's corresponding class (sleep, help, eat, run, etc.)

The thing is I've already built a similar system but for static images (no motion included), it was useful for translating alphabets only in which building a CNN was a straight forward task, as the hand doesn't move so much and the data set structure was also manageable as I was using keras and maybe still intending to do so (every folder contained a set of images for a particular sign and the name of the folder is the class name of this sign ex: A, B, C,..)

My question here, how can I organize my data set to be able to input it into a RNN in keras and what certain functions should I use to effectively train my model and any necessary parameters, some people suggested using TimeDistributed class but I don't have a clear idea on how to use it for my favor, and take in account the input shape of every layer in the network.

also considering that my data set would consist of images, I'll probably need a convolutional layer, how would it be feasible to combine the conv layer into the LSTM one (I mean in terms of code).

For example I imagine my data set to be something like this

Folder named 'Run' contains 3 folders 1, 2 and 3, each folder corresponds to it's frame in the sequence

So Run_1 will contain some set of images for the first frame, Run_2 for the second frame and Run_3 for the third, my model's objective is to be trained with this sequence to output the word Run.

Answer 1

Based on my understanding, this seems to boil down to a multi-class classification problem (each set/sequence of hand gestures eventually corresponding to a word/symbol). An RNN/LSTM/GRU architecture is a great place to start, but will be computationally expensive and rather slow to train. However, whether you absolutely need one or more of these depends on how your data is structured, and whether you can get away with treating your images as a set, rather than a sequence.

Here are some potential ideas:

1. Start by setting up a CNN layer for each "RUN_*" folder, along with dropout and/or flattening between each layer.

2. Start by setting up a single long vector representing all stages of the sequence, then add a couple of layers of CNNs, also with dropout and/or flattening.

This will take some exploration and tuning on your end to see what works the best, and you will need to choose with evaluation metrics you want to optimize for (e.g. precision/recall)

The reason I'm suggesting a CNN is because, in certain cases, CNNs can provide similar performance to a recurrent model, but at a fraction of the cost. Unfortunately, it's tough to tell with little to no information about you data.