2
$\begingroup$

When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 score .etc, and do the same for an old optimization algorithm, or do we need to compute the average performance,i.e.,the average value of accuracy or f1 scores for these runs, to show that it is better than the old optimization algorithm? Because when I read the papers on a new optimization algorithm, I don't know how they calculate the performance and draw the train-loss vs iters curves, because it has random effects, and for different runs we may get different performance and different curves. So do we compare best performance or average performance?

$\endgroup$
2
$\begingroup$

Whenever possible, you should use the average performance when comparing different methods, and preferably even mention the standard deviation across different runs (see this question for an example why it's important sometimes). It's perfectly fine to also provide the best performance, ideally you can even present a boxplot comparison of the different methods.

What is really unacceptable is to compare a best performance for one method against a mean performance for the other method (it should go without saying but I remember a paper where the authors were happily doing just that in order to make their method look better).

$\endgroup$
  • $\begingroup$ Thanks! So when a optimization method runs 20times,and find a best eval accuracy higher than the other method, but the average eval accuracy is lower than the average accuracy of the other one, so what can we say about these two optimization methods? $\endgroup$ – abc Sep 23 '19 at 2:13
  • 1
    $\begingroup$ In general the average performance is favoured because by definition it's more representative of the expected perf of the method, whereas the best performance can be an outlier due to chance. But also that's exactly why it's important to look at all the details, including standard deviation: for example if the method with a lower avg perf also has a lower variance, this means that this method is more stable and this would be an advantage in some contexts. $\endgroup$ – Erwan Sep 23 '19 at 2:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.