6
$\begingroup$

I am trying to evaluate and compare several different machine learning models built with different parameters (i.e. downsampling, outlier removal) and different classifiers (i.e. Bayes Net, SVM, Decision Tree).

I am performing a type of cross validation where I randomly select 67% of the data for use in the training set and 33% of the data for use in the testing set. I perform this for several iterations, say, 20.

Now, from each iteration I am able to generate a confusion matrix and compute a kappa. My question is, what are some ways to aggregate these across the iterations? I am also interested in aggregating accuracy and expected accuracy, among other things.

For the kappa, accuracy, and expected accuracy, I have just been taking the average up to this point. One of the problems is that when I recompute kappa with the aggregated average and expected average, it is not the same with the aggregated kappa.

For the confusion matrix, I have been first normalizing the confusion matrix from each iteration and then averaging them, in an attempt to avoid an issue of confusion matrices with different numbers of total cases (which is possible with my cross validation scheme).

When I recompute the kappa from this aggregated confusion matrix, it is also different from the previous two.

Which one is most correct? Is there another way of computing an average kappa that is more correct?

Thanks, and if more concrete examples are needed in order to illustrate my question please let me know.

Improve this question
$\endgroup$
2
  • $\begingroup$ "I am able to generate a confusion matrix and compute a kappa." - what does that mean? What is $\kappa$? (So what exactly do you compute?) $\endgroup$ – Martin Thoma Dec 6 '15 at 12:18
  • 1
    $\begingroup$ @MartinThoma en.wikipedia.org/wiki/Cohen%27s_kappa $\endgroup$ – Sean Owen Feb 17 '16 at 20:20
1
$\begingroup$

I think most people would simply average a statistic like accuracy or kappa over the several runs, rather than compute one overall statistic.

While it's not necessarily true that an average (weighted by the test set size in each case) of these statistics equals the statistic computed over all of the runs at once, it happens to be true for accuracy and kappa. Hence it shouldn't matter.

I think the reason you may be getting a difference is that you're normalizing away the test set size. I don't think you should.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.