Do I Need Pretrained Weights For Keras VGG16?

Question

As context, I am relatively new to the world of machine learning and I am attempting a project with a goal of classifying plays in an NBA game. My inputs are a sequence of 40 frames from each play in an NBA game and my labels are 11 all-encompassing classifications for a given play.

The plan is to take each sequence of frames and pass each frame into a CNN to extract a set of features. Then each sequence of features from a given video would be passed onto an RNN.

I am currently using Keras for most of my implementation and I chose to use a VGG16 model for my CNN. Here is some of the relevant code below:

video = keras.Input(shape = (None, 255, 255, 3), name = 'video')
cnn = keras.applications.VGG16(include_top=False, weights = None, input_shape=
(255,255,3), pooling = 'avg', classes=11)
cnn.trainable = True

My question is - would it still be beneficial for me to initialize the weights of the VGG16 ConvNet to 'imagenet' if my goal is to classify video clips of NBA games? If so, why? If not, how can I train the VGG16 ConvNet to get my own set of weights and then how can I insert them into this function? I have had little luck finding any tutorials where someone included their own set of weights when using the VGG16 model.

I apologize if my questions seem naive but I would really appreciate any help in clearing this up.

Answer 1

It would be beneficial to use pretrained weights trained on ImageNet because ImageNet is a huge dataset, and is fairly general. So you're more likely to get good image features from it. You might have to play around with from which layer you want to extract the features from.

If you want to create your own embeddings, you need to use the same code and instead of loading the weights, train it instead on the images you have. I do not recommend this for the following reasons

1. You most likely have too small a dataset for the model to converge
2. VGG16 does classification, what will be the labels for the individual frames is a design choice and being a novice, it might be frustrating for you to get it right
3. It takes a lot of time and resources, not an ideal situation

P.S, if you still feel that ImageNet does not sufficiently cover your use case, you can try the pretrained Inception V3 model trained on the Open Images dataset

Answer 2

I suggest you using the pre-trained model and freezing all the convolution layers. You should just train the weights of your dense layers in this situation. So use transfer learning and freeze the convolution layers and replace the dense layer with your desired one and also the last layer which is softmax. The reason is that the image-net already can find the features of a basketball play very well. You just need to classify each game. If you want to see an explanation, read this paper.