1
$\begingroup$

I noticed this and this questions, but my problem is more about class imbalance. So now I have, say, 1000 targets and some input samples (with some feature vectors). Each input sample can have label '1' for many targets (currently tasks), meaning they interact. Label '0' means they don't interact (for each task, it is a binary classification problem).

Unbalanced data

My current issue is: For most targets there are <1% samples (perhaps 1 or 2) that are labelled 1. Since I have to split train-val-test and calculate AUROC, there are in fact only 3 targets left that can support the classification under some threshold (say, have >5% positive labels across all samples).

Transform or not?

Someone has suggested modeling this as a multi-class problem instead of a multi-task problem, meaning I would transform the label vector of each sample into a set of label-1 targets. For example, if sample A originally has label 1 for targets 12, 232, 988 (and 0 for all others), the new label for sample A would simply be {12, 232, 988}=label_id.

But this might make the situation worse, because now a target (task) does not share labels across samples, e.g., if sample B interacts with target 12 and 232 only, originally targets (tasks) 12 and 232 would have two positively labeled data points, but now those two samples become totally different.


Would appreciate any suggestions! Side note: I'm using simple classifiers such as MLP or SVM. If there are any specific methods designed for imbalanced data (which I've never heard of), that would also be wonderful.

$\endgroup$

1 Answer 1

0
$\begingroup$

It's the opposite: if you had such a multi-class problem where a class represents a subset of labels, then it might help to transform it into a multi-label problem in order to have more instances for every label. In your case, it's very likely that the multi-class setting will make the imbalance problem worse, with very few instances for most classes. The exception would be if there are very few possible combinations of labels.

Essentially training a model requires a sufficiently large representative sample, in particular for every class/label. I'll be blunt: Every target which has only 1 or 2 positive instances is probably not usable at all. An obvious problem you'll have with those cases is that you can't test the model reliably with (at best!!) a single positive instance in the test set. And you certainly can't build a regular ROC curve.

$\endgroup$
1
  • $\begingroup$ This sounds very reasonable to me. I guess I'd need to find some other tasks to evaluate the samples (in fact, they are embeddings learned from a deep model, and I'm looking for a classification task to evaluate their usefulness). $\endgroup$
    – jasperhyp
    Feb 24 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.