0
$\begingroup$

I am implementing a CNN to do image classification of 4 classes representing different weathers : Haze, Sunny, Rainy, Snowy. I have as training set 3200 images, and as test set 3038 images.

The problem is that the test and training set are different set of pictures(so to be clear, I didn't split the dataset, but I have been given a training set and a test set in different folders) and they have so a different distribution one with respect to the other.

After training and evaluating, I have that my accuracy on the test set is 0.475313 and the loss always on the test set is 3.060471.

Also if I plot the hihstory of accuracy and of the loss I have the following:

enter image description here

what I don't understand is if I have the accuracy so low because I am overfitting, or because the distribution is different. And also, how do I interpret the plots I have posted about the histories of loss and accuracy? Is it normal that the gap between test and train accuracy is so large?

Thanks in advance.

[EDIT] By searching online, I have found that to cope with this kind of problems, it is possible to use domain adaptation, but honestly I have no idea how to implement it because I have just read about it.

Improve this question
$\endgroup$
6
  • $\begingroup$ Can't you merge those sets and do stratified splits yourself? Could you please upload bar char of class balance in your sets? What architecture are you using? Are you training net from scratch? $\endgroup$
    – Piotr Rarus
    Commented Dec 10, 2019 at 9:53
  • $\begingroup$ Thanks for answering. I am training a CNN from scratch and also I have a balanced dataset. The problem is that the distribution between the test and train is different, so I am not ableto go above the accuracy I wrote in the question. $\endgroup$
    – J.D.
    Commented Dec 10, 2019 at 11:24
  • $\begingroup$ Which architecture are you using? What is your augmentation procedure? $\endgroup$
    – Piotr Rarus
    Commented Dec 10, 2019 at 11:26
  • $\begingroup$ I am using a network with 8 convolutional layers and 3 dense layers. As augmentation, what I am doing is shifting and flipping the images. I am also rescaling them because they are not all of the same size. $\endgroup$
    – J.D.
    Commented Dec 10, 2019 at 11:42
  • $\begingroup$ What is the shape of input image? $\endgroup$
    – Piotr Rarus
    Commented Dec 10, 2019 at 12:18

3 Answers 3

Reset to default
0
$\begingroup$

How different is the test set from train set, like better/poorer quality? Can you do something to make them similar? And, you might be overfitting the train data. As suggested by piotr rarus, use data augmentation and transfer learning. Also, use early stopping.

Improve this answer
$\endgroup$
0
0
$\begingroup$

I'm afraid you have too little data. That's why optimizer can't converge to any reasonable place. I'd suggest you try doing transfer learning first. Few links for you to check out:

Consider also heavier augmentation. Albumentations is a nice repo, used in many kaggle competitions.

Improve this answer
$\endgroup$
0
$\begingroup$

If the distribution of the test and training sets are different, the metrics will be fairly different. And in cases of imbalanced classes, accuracy is not a great measure. Consider using precision, recall, or f-score. See if they improve over epochs.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.