# Using RNN (LSTM) for Gesture Recognition System

I'm trying to build a gesture recognition system for classifying ASL (American Sign Language) Gestures, so my input is supposed to be a sequence of frames either from a camera or a video file then it detects the sequence and maps it to it's corresponding class (sleep, help, eat, run, etc.)

The thing is I've already built a similar system but for static images (no motion included), it was useful for translating alphabets only in which building a CNN was a straight forward task, as the hand doesn't move so much and the data set structure was also manageable as I was using keras and maybe still intending to do so (every folder contained a set of images for a particular sign and the name of the folder is the class name of this sign ex: A, B, C,..)

My question here, how can I organize my data set to be able to input it into a RNN in keras and what certain functions should I use to effectively train my model and any necessary parameters, some people suggested using TimeDistributed class but I don't have a clear idea on how to use it for my favor, and take in account the input shape of every layer in the network.

also considering that my data set would consist of images, I'll probably need a convolutional layer, how would it be feasible to combine the conv layer into the LSTM one (I mean in terms of code).

For example I imagine my data set to be something like this

Folder named 'Run' contains 3 folders 1, 2 and 3, each folder corresponds to it's frame in the sequence

So Run_1 will contain some set of images for the first frame, Run_2 for the second frame and Run_3 for the third, my model's objective is to be trained with this sequence to output the word Run.

• Which dataset are you using for the ASL? – Sam Johnson Jul 24 '18 at 19:30
• Sorry but we recorded our own dataset and didn't manage to release it publicly, it wasn't that much tho and didn't give the expected accuracy. – Anasovich Jul 30 '18 at 15:00
• It's certainly doable and doesn't look like too much extra coding from the article mentioned below, but I'm not speaking from experience as it's not an architecture I have used before. Hopefully this article will point you in the right direction: machinelearningmastery.com/cnn-long-short-term-memory-networks – Ollie Graham Aug 9 '18 at 12:24

1. Start by setting up a CNN layer for each "RUN_*" folder, along with dropout and/or flattening between each layer.
2. Start by setting up a single long vector representing all stages of the sequence, then add a couple of layers of CNNs, also with dropout and/or flattening.