StratifiedKFold and the best performing model (either
Splitting your training and test set.
Below is are the two approaches you could use for splitting your train and test set. The
KFold and the
The main difference between the two is that
StratifiedKFold will ensure that the class distributions are always the same in train and test. If you would use simple
KFold, it might be possible that a class will appear in the test set, but never in the training set. If that happens, all the samples from that class will get wrongly predicted. So,
StratifiedKFold at least gives your model a chance to see one sample from each class.
Regardless of the model you pick, use
StratifiedKFold to evaluate your model.
OneVsOne or OneVsRest
It is difficult to say which model would perform best because it will depend on your data's distribution. So, I would say for you to evaluate both and pick the best performing one.
What is interesting, is that you get to choose what best performing means. It could be
the mean precision/recall/F1 score in a cross-validation setting. But it could also be the one that uses less memory, or the fastest. In general it is a mix of all these things with tradeoffs between them. In general, the
OneVsOneClassifier performs better in the F1 score, but it is slower than the
OneVsRestClassifier, but please not the words "in general". Perform your own tests and see which one works for you.
I usually prefer the
OneVsOneClassifier, because it tries to model every class against each class. This way, classes are self contained and finding boundaries around that space is more convenient. When you have a
OneVsRestClassifier, you need to model the "rest" class, which can be really complex since it is made up of various classes, which can become problematic.