Procedure for Designing CNNs

Question

Are there any standard procedure for designing a CNN?

I wrote some Python code for classifying speech signals using the 1D convolutional model in the Keras environment, but I can't meet the accuracy that I expect from it. In the training process the model reaches 98% UAR, but in the test process it overfits and remains at 58%. I used regularizer and dropout and early stopping tricks to overcome overfitting, but unfortunately, I couldn't do it. For more information, I can't augment the database or change the feature vector.

Answer 1

I don't know if this is what you are looking for, but Andrej Karpathy has a good blog article about his method for training networks in general: A Recipe for Training Neural Networks

I will put the bullet points of his recipe here, but there is way more practical advice in the actual blog.

The recipe

1. Become one with the data
2. Set up the end-to-end training/evaluation skeleton + get dumb baselines
3. Overfit
4. Regularize
5. Tune
6. Squeeze out the juice

I think the advice that best fit your situation is from step 2:

At this stage it is best to pick some simple model that you couldn’t possibly have screwed up somehow - e.g. a linear classifier, or a very tiny ConvNet. We’ll want to train it, visualize the losses, any other metrics (e.g. accuracy), model predictions, and perform a series of ablation experiments with explicit hypotheses along the way.

You usually want to start with something simple that will not allow you to have issues with overfitting like you have. Then you will progressively work your way into a more complex model that can overfit in step 3.

Answer 2

Are there any standard procedure for designing a CNN?

Not really.

If you want a more detailed answer, have a look at my masters thesis:

Thoma, Martin. "Analysis and optimization of convolutional neural network architectures." arXiv preprint arXiv:1707.09725 (2017).

Especially chapter 3 (Topology Learning) and 2.5 (Analysis Techniques) might be of interest to you.