0
$\begingroup$

When I fitted a OneVsOneClassifier (or OneVsRestClassifier), I noticed I obtained different results when I used ordinary KFold and StratifiedKFold cross validation. The testing set performance is much lower when ordinary KFold is used compared to when StratifiedKFold is used.

Questions:

1) Am I right to say for OneVsOneClassifier (or OneVsRestClassifier) strategy to work, the proportion of each class needs to be preserved in training and testing sets?

2) Is it better to use ordinary KFold or StratifiedKFold when I want to use OneVsOneClassifier (or OneVsRestClassifier)?

Many thanks.

$\endgroup$
1
$\begingroup$

TL;DR

Use StratifiedKFold and the best performing model (either OneVsOneClassifier or OneVsRestClassifier).

Splitting your training and test set.

Below is are the two approaches you could use for splitting your train and test set. The KFold and the StratifiedKFold:

https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_indices.html?highlight=kfold%20comparison

https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_indices.html?highlight=kfold%20comparison

The main difference between the two is that StratifiedKFold will ensure that the class distributions are always the same in train and test. If you would use simple KFold, it might be possible that a class will appear in the test set, but never in the training set. If that happens, all the samples from that class will get wrongly predicted. So, StratifiedKFold at least gives your model a chance to see one sample from each class.

Regardless of the model you pick, use StratifiedKFold to evaluate your model.

OneVsOne or OneVsRest

It is difficult to say which model would perform best because it will depend on your data's distribution. So, I would say for you to evaluate both and pick the best performing one.

What is interesting, is that you get to choose what best performing means. It could be the mean precision/recall/F1 score in a cross-validation setting. But it could also be the one that uses less memory, or the fastest. In general it is a mix of all these things with tradeoffs between them. In general, the OneVsOneClassifier performs better in the F1 score, but it is slower than the OneVsRestClassifier, but please not the words "in general". Perform your own tests and see which one works for you.

I usually prefer the OneVsOneClassifier, because it tries to model every class against each class. This way, classes are self contained and finding boundaries around that space is more convenient. When you have a OneVsRestClassifier, you need to model the "rest" class, which can be really complex since it is made up of various classes, which can become problematic.

$\endgroup$
2
  • $\begingroup$ Ok many thanks for your answer. The key point I got from here is "Regardless of the model you pick, use StratifiedKFold to evaluate your model." $\endgroup$
    – Leockl
    Feb 29 '20 at 11:12
  • $\begingroup$ @Leockl exactly ;) $\endgroup$ Mar 2 '20 at 8:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.