1
$\begingroup$

I have to do image classificaion with a CNN, and for doing this I have been given a training set with 4 classes and a test set with 3 classes. I am really confused because I don't know if this is going to influence my prediction. It never happened to me.

How can I deal with this? Thanks in advance.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Why not training your model using 3/4 labels? Or splitting your training set and generating a new test set? Or evaluate your model with 4 outputs of the model and check if those labels cover 3 expected results? $\endgroup$
    – aminrd
    Commented Dec 6, 2019 at 21:56
  • $\begingroup$ Thanks for answering. I was thinking that I could train my model on 4 classes and test it on 3 classes, and create an empry folder for the fouth class. Do you think it could work? Thanks. $\endgroup$
    – J.D.
    Commented Dec 7, 2019 at 7:37

1 Answer 1

Reset to default
1
$\begingroup$

In principle there's nothing wrong with that, since every instance in the test set is predicted individually. You will have a 4 x 3 confusion matrix, because the model might predict some false positives on the fourth class. Of course you won't be able to know if the model can correctly identify a true instance from the missing class.

It depends what is the goal:

  • If the model is meant to be able to predict any of the 4 classes, then it should be trained on the 4 classes and it would be preferable to also test it on the 4 classes, but testing it only on 3 should already gives a good indication of its performance.
  • If the model only needs to predict 3 classes ever, then the instances of the 4th class should be removed from the training set since they just make things more complex.
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for answering. Is it possible that this influences also accuracy? I ask this because by splitting the data set I have 92% of accuracy, while by using a test set with 3 classes I get 45%. Thanks. $\endgroup$
    – J.D.
    Commented Dec 8, 2019 at 11:26
  • 1
    $\begingroup$ @J.D. this probably means that the test set is drawn from a different distribution than the training set. it's not surprising since they don't even contain the same number of classes, but it also points to deeper differences in the two datasets. that's what would happen if the images in the test set were obtained using a different method than those in the training set, for example. for a proper ML experiment it would be preferable to mix the two datasets into one big set and then split this big set randomly, but maybe in your specific task there are external reasons why the two datasets .... $\endgroup$
    – Erwan
    Commented Dec 8, 2019 at 13:41
  • 1
    $\begingroup$ ... are different. so I'd suggest you ask whoever gave you the data this way if the difference between the two datasets is meaningful for whatever reason, in which case you have to use it as provided even if it makes the task harder. $\endgroup$
    – Erwan
    Commented Dec 8, 2019 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.