1
$\begingroup$

I used pytorch as the training framework and the official pytorch imagenet example to train the image classification model with my custom dataset.

My custom dataset has 2 different label (good and bad), and over 1 million images.

I splitted the dataset into a training set(80%), a val set(10%), and a test set(10%)

My model got average 99% training acc in training phase, and nearly 99% val acc in validation phase. In the testing phase, the model got 99% testing acc.

However, when I used my model to evaluate the whole dataset(all the images in my dataset), the acc got only 90%, which is pretty weird since my model updated its parameter in the training phase.

The model should be able to achieve higher accuracy, but it can only get 90% acc when evaluating the whole dataset.

I am wondering if it is normal or anything I can check for this problem.

$\endgroup$
1
$\begingroup$

These performance values are inconsistent, this is definitely not normal. The whole dataset is made of the training set, the validation set and the test set. Accuracy is the proportion of correctly labelled instances, so accuracy on the whole dataset is:

$$accu_{full}= 0.8 * accu_{train} + 0.1 * accu_{val} + 0.1 * accu_{test}$$

Since $0.8 * 0.99 + 0.1 * 0.99 + 0.1 * 0.99 = 0.99 \neq 0.90 $, there must be a mistake somewhere, at least one of your performance values is wrong.

$\endgroup$
1
  • 1
    $\begingroup$ Finally, I found out the root cause: I used different mean and std to normalize the dataset in training code and testing code, which cause the performance degradation. I got 99.95 accuracy in the correct version. $\endgroup$
    – Yanwei Liu
    Sep 27 at 0:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.