Trained model performs worse on the whole dataset

Question

I used pytorch as the training framework and the official pytorch imagenet example to train the image classification model with my custom dataset.

My custom dataset has 2 different label (good and bad), and over 1 million images.

I splitted the dataset into a training set(80%), a val set(10%), and a test set(10%)

My model got average 99% training acc in training phase, and nearly 99% val acc in validation phase. In the testing phase, the model got 99% testing acc.

However, when I used my model to evaluate the whole dataset(all the images in my dataset), the acc got only 90%, which is pretty weird since my model updated its parameter in the training phase.

The model should be able to achieve higher accuracy, but it can only get 90% acc when evaluating the whole dataset.

I am wondering if it is normal or anything I can check for this problem.

Answer 1

These performance values are inconsistent, this is definitely not normal. The whole dataset is made of the training set, the validation set and the test set. Accuracy is the proportion of correctly labelled instances, so accuracy on the whole dataset is:

$$accu_{full}= 0.8 * accu_{train} + 0.1 * accu_{val} + 0.1 * accu_{test}$$

Since $0.8 * 0.99 + 0.1 * 0.99 + 0.1 * 0.99 = 0.99 \neq 0.90 $, there must be a mistake somewhere, at least one of your performance values is wrong.