I am looking for a good must-have reference about statistical analysis of classification results in machine learning.
What books do you recommend about it?
This is a thought-provoking article making a case for the advantages of using Bayesian methods to compare out-of-sample performance and even the advantages of Bayesian methods in general. The article focuses on comparing classifiers, but little is really dependent on using a particular metric.
Some of the beauty of the article, however, is that it goes through many classical (frequentist) methods for such comparisons, making it almost an encyclopedia entry on the topic.