Theoretical background for model arhitecture choosing

Question

Suppose I have a dataset $X$ and two different binary labels $y_{1}$ and $y_{2}$. The classes are very imbalanced - 3% of true in $y_{1}$ and 2% in $y_{2}$. Moreover, there are no pairs of (0,1), so, if the $y_{1} = 0$, then $y_{2}=0$.

I have to choose between:

• Associating the labels with each other, so it's a 3-class classification problem
• Build a model for $y_{1}$ scoring, then joining to $X$ and making scoring of $y_{2}$, using $X$ and $y_{1}$.

So the first idea is formalised as :

$$ y_{f} = y_{1} \bullet y_{2}; \,\,\,\, y_{f} = \{(1,1),(0,0),(1,0) \}; \,\,\, y_{f} = \Phi_{f}(X)$$

And the second road my be formalised like this: $$ y_{1} = \Phi_{1}(X); \,\,\, \, \, X=X\cup y_{1}; \,\,\,\, y_{2}=\Phi_{2}(X) $$ Taking into account all condition, what is the possible theoretical background of projecting the best algorithm? I'am trying to give estimates to an error, which is :bias + variance + noise. E.g.: For both: $\Phi_{f}$ and $\Phi_{2}$, which are RandomForests, bias and noise will be the same, variance will be higher for $\Phi_{f}$(just intuition till now). Can somebody point me the right way?

Answer 1

The theoretical estimation of the error depends on the data, and the fact that the labels can be predicted by the variable X. There's no way to a priori know which will be the best option regarding the two models, so just try.

However, I would bet that the second option will work better if y2 is correlated in some way to y1, and I would bet for the first option if they are independent.

However, when you train your model, I highly recommend sampling the data in a more balanced way, so that the model not learns to predict (0, 0) always regardless of the values of X (as a model that does these dummy predictions would have high accuracy). Another way to work around the imbalanced classes is to use AUC in ROC as a metric to choose among different models.