Let's say that I have a multilabel problem, where each sample can be of class A, B, C, or any combination of these.

Because of high imbalance, I've found that if I tackle the problem as 3 separate, binary problems (is it A/B/C?) I actually get a decent performance, compared to multilabel-adapted algorithms.

But I'm wondering whether it makes a difference to

A) Make a global train/test split, and then deal with each class individually for both train and test. Each classifier is trained on the same data, but with different goals. This is how e.g. skmultilearn seems to prefer it.

B) Treating the three problems entirely separately, i.e. split separately, preprocess separately, validate separately. The three classifiers never share any data, so there shouldn't be any leaking of knowledge. The final performance is also evaluated on different parts of the data, but since the classifiers are entirely independent, this shouldn't matter.

The second option should, according to my intuition, give the best representation for each binary problem, with the only downside being potentially 3x the processing. However, as my problem concerns only a few hundred samples, computing time is negligible.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.