Transfer Learning or Custom Network?

Question

I am learning Computer Vision and I was wondering if it's usually worth it to build a custom convolutional network from scratch (through trials and errors) or if using transfer learning with a popular CNN structure (ResNet50, VGG16, etc.) is good enough?

Answer 1

First of all if you are starting out building a solution for a problem statement such as image classification, it will be worthwhile if you start on pretrained models like Resnet50 or VGG16 as they are trained on Imagenet dataset which is the benchmark and you can train the last layers on your own data. You can set the results you got as baseline on which you can improve upon. Though the results can be underwhelming if you have less data and hyperparameters are not tuned optimally.

You may get better results in training from scratch but it comes at increased cost of computational and time resources. Though if you have domain experience on the problem you are working then that knowledge can help you better plan the model architecture even with less data and give better results.

To better plan this, you can experiment with small dataset with same distribution as original data on both the cases and come to a conclusion.

So,it all comes down to the problem you are solving,resource constraints and the experience you have in working on these problems. In my experience having a baseline with pretrained model never hurts and iterating on it helps build and ship faster!!