How to improve a CNN without changing the architecture?

Question

I'm currently using an autoencoder CNN that's built upon the VGG-16 architecture that was designed by someone else. I want to replicate their results using their dataset first but I'm finding that:

-Validation losses diverge from training losses fairly early on (I get to around 10 epochs and it already looks like it's overfitting) -At its best, the validation losses aren't even close to being as low as training losses -In general, the accuracy is still worse than reported in their paper.

I'm new to machine learning and want to know if there are hyperparameters I should try to change or what I can do to maybe tinker with it without changing its architecture?

Answer 1

Well there are a number of ways you can improve a CNN network without changing the architecture.

So I will try to explain each of them as much as I can also since your Validation Loss is diverging from your Training Loss quiet early on which indicates Overfitting but you have figured that out already.

1. How To Replicate The Result From The Paper: Use the code as is without changing anything. Also use the same dataset that they have used for training. This way you should be able to replicate their training results. Then produce the output on the same data as they have to check if you are getting similar results. Of-course keep in mind both the training and prediction results that you get and the paper have might be different but they should be within some acceptable range.

Now if you want to train the model on your own data and that is causing the issue you can try to look into the following:

2. Check Your Data: Since generally the model given in the paper is suppose to work readily for some other kind of data as well without changing anything the first thing we should consider is that something might be wrong with the data that we are providing the model with. It can be a number of things 1) Very few data points for training hence the reason for Overfitting 2) Even though you have large number of data points make sure that you are providing the model different data points in each epoch and not the same one which can cause Overfitting. 3) Data is just too simple, try some data augmentation techniques to increase the variance in the training set. There can be more but I would suggest you look at these first.

If you are satisfied that nothing is wrong with the training data then lets move on the next things that we can try:

1. Change Optimizer: If your model is overfitting it means either you are getting stuck in some local minima and or you have too high learning rate. But let's talk about optimizer in this step. Try changing default optimizer to one that is much less aggressive, in my knowledge the optimizers from less to most aggressive goes something like this Gradient Descent--> Stochastic Gradient Descent---> Stochastic Gradient Descent with Momentum---> Adagrad ----> RMSProp ----> Adam. So maybe you can start with this list in sequence and if something improves.
2. Change Learning Rate: Since the model is overfitting we can try to reduce the learning rate and see if training loss matches with the validation loss and comes down together. General Good Values for learning rate can be 0.0001 ---> 0.003 ----> 0.001 ----> 0.003 and so on...
3. Batch Size: Of Course if your batch size if too low your model will fit perfectly for the training set in that epoch but for the same epoch the validation loss will be too high. So try to keep maximum batch size that your system can handle is all I can say...

There are many more advance things that we can try but I am guessing these should fix the problem you are facing. Hope this will be of use and would love to hear your feedback if any of these things helped you out or not :)

Answer 2

Are you in fact using the same architecture as they are? If not that could potentially be the problem.

Otherwise, are you using the same trainings protocol as they, i.e. optimizer, learning rate, learning rate schedule, batch size, preprocessing, weight initialization, number of training epochs? Depending on the size of your model and the amount of training data, 10 epochs might not be enough to judge about your models performance.

Can you link the paper?