0
$\begingroup$

I am building a CNN using keras for a classification task. I started with a simple model as a starting point and as almost all ML problems go, especially if the dataset is not very big, I faced an almost immediate major overfitting as you can see in graphenter image description here

In order to try to overcome this overfitting I started with reducing the network size to half of its size (half the number of units per layer) and adding L2 weight regulaizers which improved the overfitting a little as you can see here enter image description here

then I added droput layers which also helped especially with the noise in the validation curveenter image description here

After this, in order to increase the training data I used data augmentation with the ImageDataGenerator class from keras which works as expected (decreasing the train metrics while improving the validation metrics) and seems to help a lot with the overfittingenter image description here

As the validation curves start to follow the training curves, it seems like the task now is how to improve the training metrics and here comes my question, the training metrics are almost stable after 10 epochs and they don't improve no matter how many epochs are increased and the only thing I could think of trying is trying to reduce the learning rate when the training accuracy stops improving, so I added a ReduceLROnPlateau callback which reduces the learning to 20% of the previous one if the training accuracy doesn't increase for 3 epochs but this doesn't do the trick, the training accuracy is still stuck around 90-92%enter image description here

So, I am wondering if there is anything else that could help increasing the training accuracy or is it just a lack of data problem and the only possible solution is to increase the dataset.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

At some point, it all comes down to getting more data. And you seem to have tried a lot at this point but sure, you can try a few more things.

Even though you generate additional data by augmenting, I am guessing you have a rather small dataset. So, you might try transfer learning. But that probably is not a good fit for you as I guess you want to build a model from scratch. And also accuracy you have is already up to %90, so I am not sure if you'll get any improvement with that.

And I also wonder if you are using a pure CNN - without any fully connected layers. If that is the case, instead of regular dropout, you can try spatial drouput. You can also add a few fully connected layers at the expense of reducing the convolutional layers' size. In such a setting, you can also use Batch Normalization - maybe together with or instead of dropouts on the fully connected layers.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the reply, I already thought about using transfer learning, I haven't done it before but maybe it is a good chance to learn. regarding your question I am already using some fully connected layers, is it still worth looking into the spatial dropout? I will also look into the batch normalization as I honestly don't know how this works $\endgroup$
    – Luka
    Sep 21, 2021 at 12:04
  • $\begingroup$ @Luka Check this answer for a brief discussion and a brief argument against using regular dropout on convolutional layers. I suggest either use spatial dropout on convolutional layers or no dropout on them at all - save dropouts for fully connected layers. But I definitely suggest you looking into batch normalization. $\endgroup$
    – serali
    Sep 21, 2021 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.