Train-test split and augmentation strategy for small dataset for video classification problem

Question

I have a small data set of videos of approximately 100 videos for each class for a binary classification problem. This results in a total of 200 videos. I am applying two types of augmentations on the videos.

1. Transformative augmentations which rotate and flip the videos. These are of 5 types so the resultant data set increases 5 times in size.
2. Increasing and decreasing brightness, saturation, contrast, blur, shake, noise etc.

All the videos have captured the same phenomenon of drop of some liquids spreading on paper. I have attached two frame of a video below, one near the start and another when the drop has expanded. On a 3D CNN i am getting a decent accuracy of around 85%.

My concern is:

1. As i have a small data set is it fine to apply transformative augmentations first and then do a train-test split so that my data set expands to 1000 videos and the split results in 800:200 videos? Or should i do split before augmentation which results in 80x5=400 train videos and 20 test videos?
2. Also, i have my CNN with three pretty standard convolutional layers and two batch normalization layers. On every run where i get a good accuracy, the test accuracy is 1-2% more than train accuracy. Should it be concerning?
3. Also, when i include second type of augmentation of brightness etc. i see the difference in train and test accuracy of 10-15%. I think this can be caused by training set becoming very hard in comparison to test set. Am i thinking right?

Answer 1

Your three questions are tightly related:

1. You should not augment the data before splitting. This leads to data leakage, as there is an overlap between the training and the test data, because you are testing your model on some images that have been already seen (although in a transformed version) during training. Therefore, you should first split, then augment.

2. It is possible that the training accuracy is lower than the test accuracy when you have train-only regularization elements like dropout in your network. From your description of the network architecture, this is not your case. Your performance difference is most probably caused by the data leakage derived from your augmentation.

3. Same as 2.