I have just started learning image processing and this is my first time working on video classification. I am trying to develop a model that recognizes hand gestures using the EgoGesture dataset(more info: http://www.nlpr.ia.ac.cn/iva/yfzhang/datasets/egogesture.html). This dataset consists of videos taken by 50 people in 6 different environments containing 83 gesture categories. Also, some of these gestures are dynamic. But I am using data from only 18 people because my memory is not enough. Since this dataset consists of videos, I take 10 equally distributed frames from each gesture video Like this: Gesture Frames
I resize each frame to be 128x128 pixels and normalize it. In total, I get 79320 frames from 7932 videos. I set the shape of X to (7932, 128, 128, 10, 3). Then I split this 80-20.
Here is my 3D CNN model code:
model = Sequential() model.add(Conv3D(32, (3, 3, 3), activation='relu', input_shape=(128, 128, 10, 3))) model.add(MaxPooling3D((2, 2, 2))) model.add(Conv3D(64, (3, 3, 3), activation='relu')) model.add(MaxPooling3D((2, 2, 2))) model.add(Conv3D(64, (3, 3, 3), activation='relu', padding='same')) model.add(Flatten()) model.add(Dense(128, activation='relu')) model.add(Dropout(0.5)) model.add(Dense(num_classes, activation='softmax')) optimizer = Adam(learning_rate=0.000001, beta_1=0.9, beta_2=0.999) epochs = 30 batch_size = 10 model.compile(optimizer = optimizer , loss = "categorical_crossentropy", metrics=["accuracy"]) model.summary()
When I train this model, the training accuracy rate is around 0.04 and the validation accuracy rate is around 0.02.
I tried a more complex model by adding a few layers but there was no difference. I played with the hypermeters and again I could not achieve the desired success. I tried another model with CNN+LTSM and again I could not achieve a successful result.
Then I tried to use Depth videos (black and white segmented hands) instead of RGB videos in the dataset and both accuracy values increased to around 0.15 but again not enough.
It's obvious that I'm doing something wrong but I can't figure out what it is. Is my model wrong or maybe the data is not enough? I would be very grateful if you can help me. I will add more information if more information is needed.