How would someone go about using deep learning to classify sign language gestures? For example, suppose I had video files of many different gestures. For any given gesture, I might have many videos of it, and each video would be comprised of many frames. When trying to classify MNIST digits in images the dimensions of the input are comparatively simple: height, width, and RGB channels. How would gestures (multiple frames over time) be accounted for? Would time be a fourth dimension? What should the neural net's architecture look like so as not to overfit? Should I use something instead of a convoluted neural network?
I know that there are probably clever ways to hand-code predictors for sign language, but I'm more interested in how to architect the neural net and take advantage of the time component of the data (where there is value in the transition of video frames over time). Classifying gestures is a simplification of the actual problem I'm trying to solve, so I'm looking for an approach that would be generalizable to other types of problems where it might be necessary to look at many frames of the video to predict the target variable.