How reliable are model performance reportings?

Question

I had a conceptual doubt about estimating and reporting a classification model's performance. Say my model works with range of depth values and gives out different readings of test errors. We choose the model with depth having lowest test error rate as M1

Now if we want to report our model's performance on a hidden test set, would it wise to say that M1 would perform equally well on this new test set and with the same test error rate?

Answer 1

The question of the measured test error of a classification model is reliable, hence if the test error on unknown set $T_1$ is the same as on $T_2$ is hard two answer. It depends on the following factors:

• How many digits of the error are reported?
• How many samples have $T_1$ and $T_2$? The more digits are reported, the more samples you need. As a rule of thumb, make sure that any change in the reported error means at least 3 samples have changed. So if you use accuracy and report two decimal places (e.g. 12.34%), then 0.01% must be bigger than 3 => $3 < \frac{0.01}{100} \cdot |T_1| \Leftrightarrow 30000 < |T_1|$
• The distribution must be similar. The simplest part is the distribution of classes. The more difficult part is how the features look like.

For other forms of error analysis, you might want to look into my Master's thesis Analysis and Optimization of Convolutional Neural Network Architectures