Suppose I have a dataset $X$ and two different binary labels $y_{1}$ and $y_{2}$. The classes are very imbalanced - 3% of true in $y_{1}$ and 2% in $y_{2}$. Moreover, there are no pairs of (0,1), so, if the $y_{1} = 0$, then $y_{2}=0$.

I have to choose between:

  • Associating the labels with each other, so it's a 3-class classification problem
  • Build a model for $y_{1}$ scoring, then joining to $X$ and making scoring of $y_{2}$, using $X$ and $y_{1}$.

So the first idea is formalised as :

$$ y_{f} = y_{1} \bullet y_{2}; \,\,\,\, y_{f} = \{(1,1),(0,0),(1,0) \}; \,\,\, y_{f} = \Phi_{f}(X)$$

And the second road my be formalised like this: $$ y_{1} = \Phi_{1}(X); \,\,\, \, \, X=X\cup y_{1}; \,\,\,\, y_{2}=\Phi_{2}(X) $$ Taking into account all condition, what is the possible theoretical background of projecting the best algorithm? I'am trying to give estimates to an error, which is :bias + variance + noise. E.g.: For both: $\Phi_{f}$ and $\Phi_{2}$, which are RandomForests, bias and noise will be the same, variance will be higher for $\Phi_{f}$(just intuition till now). Can somebody point me the right way?


The theoretical estimation of the error depends on the data, and the fact that the labels can be predicted by the variable X. There's no way to a priori know which will be the best option regarding the two models, so just try.

However, I would bet that the second option will work better if y2 is correlated in some way to y1, and I would bet for the first option if they are independent.

However, when you train your model, I highly recommend sampling the data in a more balanced way, so that the model not learns to predict (0, 0) always regardless of the values of X (as a model that does these dummy predictions would have high accuracy). Another way to work around the imbalanced classes is to use AUC in ROC as a metric to choose among different models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.