How to tinker with CNN architectures?

Question

I was thinking of creating a CNN. Now it is known CNN takes long times to train so it is advisable to stick to known architectures and hyper-parameters.

My question is: I want to tinker with the CNN architecture (since it is a specialised task). One approach would be to create a CNN and check on small data-sets, but then I would have no way of knowing whether the Fully Connected layer at the end is over-fitting the data while the convolutional layers do nothing (since large FC layers can easily over-fit data). Cross Validation is a good way to check it, but it might not be satisfactory (since my opinion is that a CNN can be replaced with a Fully Connected NN if the data-set is small enough and there is little variation in the future data-sets).

So what are some ways to actually tinker with CNN and get a good estimate for future datasets in a reasonable training time? Am I wrong in my previous assumptions? A detailed answer would be nice!

Answer 1

CNN Architectures: These are proven configurations of the combination of CNN/FC/POOL etc layers, that seem to work in commonly defined problems.

Now from your question what I understand is you want to "Tinker" with the architecture and not to come up with an entirely new configuration from scratch. One thing that can help you in this scenario is transfer learning.

Transfer Learning: It is a process of using pre-learnt weights from a similar task on altogether different data, to your current problem and data.

In transfer learning what we do is we take some layers usually 80-90 % of the layers keep the weights the for these layers and delete the weights for remaining layers, and initialise these layers with some kind of initialisation batch norm or something like that. And train this newly created model with the data you have.

This have proven to be effective in various problems.

You can come up with innovative ways to make changes to arbitrary layers of this network train them and check for error metrics. If you follow this approach whatever you are claiming in your question can be achieved.