Convolutional Neural Network overfitting

Question

I built a CNN to learn to classify EEG data (only about 4000 training examples, 2 classes, 50-50 class balance). Each training example is 64x512, with 5 channels each

Ive tried to keep the network as simple/small as possible for testing:

1. ConvLayer (4 filters)
2. MaxPool
3. Dropout 50%
4. Fully connected (50 neurons)
5. Dropout 50%
6. Softmax

Im also using weight decay (L2 reg, lambda = 0.001)

The problem is no matter how I play with the filter parameters (size, stride, number) my network keeps overfitting. It fits the training data 100%, but no matter what I do I can't get the test accuracy over 65%.

Why is such a small network overfitting? I thought it was a sample size issue, but I've read a number of research papers on EEG and BCI and they occasionally have even smaller sample sizes than I do

What else can be done to regularize a CNN?

Answer 1

This is because you have very little amount of data. If not enough data is provided to CNNs, they are very likely to overfit. You can do the following things in order to overcome this problem:

Data Augmentation : It is a technique to create new examples from the training examples by doing some preprocessing on them e.g. Rotation, Scaling, etc.

Rigorous Dropout : Dropout is a very powerful technique to control overfittting. You can try using more rigorous dropout in your architecture

Answer 2

Have you tried early stopping? Hold out 20% of the data and stop training when the error on the hold-out set starts to increase. If you can't get high enough accuracy, increase the size of your model until you do.

Answer 3

One possible solution when you do not have enough data is to use Transfer learning.

This helps you to improve the performance of your model on the test data set. So, you can easily use one of the available pre-trained models in technical literature and update its weights based on your data. Take a look at this video. It is very helpful and you get a lot of ideas how to deal with your problem.