I have a CNN architecture for CIFAR-10 dataset which is as follows:

Convolutions: 64, 64, pool

Fully Connected Layers: 256, 256, 10

Batch size: 60

Optimizer: Adam(2e-4)

Loss: Categorical Cross-Entropy

When I train this model, training and testing accuracy along with loss has a very jittery behavior and does not converge properly.

Is the defined architecture correct? Should I have a maxpooling layer after every convolution layer?

The Conv-2 architecture is an attempted implementation of the following paper.

The following table is taken from The Lottery Ticket Hypothesis By: Jonathan Frankle, Michael Carbin

Figure - 2: Lottery Ticket Hypothesis



Usually, the setup of having two conv. layer followed by one pooling layer works well for CIFAR. Since your batch size is relatively high and the learning rate relatively low these are probably not the source for the lack of convergence. For further analysis a plot of your train and validation loss (not accuracy) would be helpful.

Therefore, I suggest the following steps:

1. Plot train and valid loss curves to understand where you're standing in terms of convergence and overfitting/underfitting, e.g. see this article on how to read loss curves.

2. If your current model capacity is insufficient train for more epochs or add another stack of layers with the same structure (features should be increasing throughout the network by for example first having two conv. layers with depth 32 and the next two conv. layers with depth 64)

3. If overfitting becomes a problem add dropout

Moreover, I do not think you need multiple fully connected layers at the end but they might not do harm either. Adam might work well the standard learning rate here but if you have manually optimized yours that is fine too (the learning rate is, in fact, the most influential hyperparameter so it almost always makes sense to optimize it).

In deep learning it usually makes a lot of sense to check what networks have worked on the task you're working on (or similar tasks). For CIFAR you find a simple but well working network here.

| improve this answer | |
  • $\begingroup$ edited my question to add more information. Since, I am implementing the Conv-2 architecture for CIFAR-10 dataset as mentioned in the paper paper, increasing number of conv layers is not my aim. All the parameters I use are taken from the paper (as mentioned) $\endgroup$ – Arun Feb 17 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.