How do I know that model performance improvement is significant?

Question

Say I am running a Machine Learning model that produces a certain result (say accuracy of 80%). I now change a minor detail in my model (say, in a Deep Learning model, increase the kernel size in one convolutional layer) and run the model again, leading to an accuracy of .8+x.

My question is how I would determine which in-/decrease in performance allows me to say that the new network architecture is better than my old one? I assume that x=.0001 falls within a reasonable margin of error, whereas x=-.2 is a significant decrease in performance - however, the use of "significant" here would be purely colloquial without any scientific backing.

I understand that some kind of hypothesis testing would in theory be appropriate here, but as far as I know, these require multiple samples (i.e. running the network many times), which in case of large ML models which take sometimes days to train isn't really feasible.

Answer 1

A common way to evaluate the significance of a change in performance between two models is to use a paired t-test. This test assumes that the data for both models comes from a normal distribution and tests whether the mean difference between the two models is significantly different from zero.

To apply a paired t-test, you would first need to run your models multiple times and collect the accuracy scores for each model. You can then calculate the difference between the accuracy scores for each model, and use the t-test to determine if the mean of these differences is significantly different from zero.

If you are unable to run your models multiple times, you could still use a paired t-test by randomly splitting your dataset into two halves and using one half to evaluate each model. However, this approach may not be as reliable as using multiple runs with the same data split.

In general, a significant difference in performance between two models would typically be considered to be a change of at least 0.1 in accuracy. However, the threshold for significance will depend on the specific context and the relative importance of the accuracy metric for your problem.