TL;DR: I don't understand the dependency issue with binary classification (binary relevance) compared to multi-label prediction models.

I often read in papers that some kind of "dependency information" is missing when a binary classification is used for a multi-label prediction problem. I mean a multi-label problem is solved with single binary classifiers.

Let's take a simple data set with multiple target variables:

Age Gender TargetA TargetB TargetC
34 m 1 1 1
22 f 0 0 1
45 m 1 0 0

If the goal is to predict future TargetA, TargetB and TargetC, then I see these possibilities:

  • A) Train 3 independent binary classification models on the full data set. I.e. random forest model with the target "TargetA" will be trained on the input Age, Gender, TargetB and TargetC. Further separate models for the other targets.
  • B) Transform the three target columns into one "multi-label column": Targets. Train only one model to predict Targets.


  • A) Interpretability; Models can be independently distributed, different model types for each target.
  • B) Only one model to maintain / maybe faster predictions.

My questions are:

  • Are my assumptions correct?
  • Is there a difference between the two approaches in terms of predictive power and -ability?

Related links:


1 Answer 1


The approach A) you describe is not very useful since it includes the feature TargetB for the prediction of TargetA. However, for a new data point, you know neither TargetA nor B or C, so such a model cannot make any predictions.

However, let's assume your approach A) consists of three binary classifiers which have only age and gender as features. Then the 'dependency information' these models are missing are how the targets go together. For example, if TargetA is present, it might be highly (un)likely that TargetB is also present. The single models cannot learn this information, since each of them sees only TargetA or B in isolation. Depending on how strong such target dependencies are, the predictive power of three binary classifiers might be substantially worse compared to a single classifier.

  • $\begingroup$ That makes perfect sense to me. So, conversely, one can say that A) would only work for data points, which have already one more targets set (depending on how strong the dependency is). Ok, that would also make troubles for an application which uses such binary models. I will try to train both variants, compare them and share my results - such for the sake of science :) $\endgroup$
    – malisokan
    Apr 21 at 6:58
  • $\begingroup$ Sounds reasonable $\endgroup$ Apr 21 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.