I currently want to train a CNN but I have two small datasets that are slightly different because of the camera setup that captured the images. I'm interested in ultimately tuning the neural network to only one of the camera setups and using the network for that one setup only from here on out.

Since I'm not going to be able to expand my dataset during this time, I realize I'll need to use both training datasets to get enough of variability. So I'm wondering what approach to this might be better -- should I combine the two datasets and train the CNN, or should I train on one dataset and finetune it to the second dataset? Or is there another method I should be using?

  • $\begingroup$ What are the sizes of the two datasets? $\endgroup$ – Leevo May 4 '20 at 21:10
  • $\begingroup$ I have one dataset of 200 sets of images (image + groundtruth) and the other dataset has 145. I am using the pretrained VGG-16 neural network at the moment but am not sure if I should train the 200-dataset, then fine-tune with the smaller dataset. OR, if I should train all of them together. $\endgroup$ – nmtp May 5 '20 at 1:17
  • $\begingroup$ And how many classes are you trying to predict? $\endgroup$ – Leevo May 5 '20 at 6:46
  • $\begingroup$ I have an encoder-decoder architecture, output is another image $\endgroup$ – nmtp May 5 '20 at 18:30

I suggest you to use as much data as possible. If the images are coming from two different cameras, it could be an original way to fight overfitting. I would use both datasets, and operate a train-validation-test split that is transversal to both.

Additionally, since the dataset is small, I strongly enourage you to use lots and lots of data augmentation, which is a very powerful technique to artificially increase the size of your dataset and fight overfitting.


So here is my thought. I suggest you do some sort of clustering of images. It is okay if it is not a robust one.

use any pre-trained model architecture of Inception which has been trained on huge corpus.

Freeze the layers and take the feature encodings on the images by passing them through the inception model (Output of FC layer)

apply pca and form cluster (n=number of classes)

For both dataset A and B do the clustering and from each cluster select 80% for training and keep 20% for testing.

In this way your model which you will train using this data will have all variations from both set and your test set will also contain the same variations as of your train data. Train and Test set will be an ideal data with all variations.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.