General strategy for imbalanced, semi-supervised, sparse problem

Question

I am looking for some general advice on where to start with this problem. There are 350 sparse (low positive integer) features. I have 2000 positives, 1000 negatives, and infinite unlabeled data, where the estimated true positive rate in the unlabeled data is ~1% (but this isn't known for certain). The goal is to predict the probability of an example being positive.

How would you approach the problem if you had roughly 40 hours to attain the best possible model?

What about if you had more time?

Answer 1

It's a binary semi-supervised classification problem. First, establish a base-line for the supervised case. Then try if the unlabeled data helps

Supervised

1. From your labeled data: create a training, validation and test set. Don't touch the test set until the very end.
2. Try something simple, e.g. a multilayer Perceptron (MLP) with 350 input nodes and 1 output node (giving the probability of "true").
3. Try more stuff (e.g. https://github.com/MartinThoma/algorithms/blob/master/ML/mnist/many-classifiers/python.py#L92).
4. Try to combine classifiers in ensembles (see examples).
5. A (naive) bayes classifier might be worth being investigated.

Semi-supervised

• You could train an auto-encoder with a bottleneck on the unlabeled data. Then remove everything after the bottleneck. Use this network as a preprocessing step. The idea is that this network finds a more meaningful abstraction of the relevant data. However, similar as PCA can have a very low projection error and still destroy the possibility to distinguish the classes.
• SVMs can also be used in a semi-supervised setting.
• You could try to find clusters in the unlabeled data + labeled data. Then you can get an a priori probability of "true" for each cluster (ignoring the unlabeled data). You could train very small models (overfitting!) on the data in the clusters.

Usually, you can interpret features. This might help to develop new features.

Answer 2

Given your time budget and the potential challenges associated with class imbalance, I'd throw away the unlabelled data and use supervised learning on the labelled data. Try a simple classifier, e.g., logistic regression or random forests or xgBoost, and use cross-validation to see how well they perform. In advance put aside a held-out test data and don't touch it until the end. Use cross-validation on the rest of the data to try different classifiers and different approaches.

You have to deal with a severe class imbalance problem. My suspicion is that you might need to budget a large fraction of your time dealing with class imbalance. A true positive rate of 1% corresponds to a 100x class imbalance, which causes severe problems for many classifiers and might turn out to be a real headache.

Also the true underlying distribution doesn't match the distribution in your training set: the true positive rate in the wild is 1%, but 67% of your training set is positives. You'll need to adjust for this as well.

There are many ways to deal with class imbalance and to adjust for differences between the imbalance in your training set vs in the wild. I would suggest you start by setting class weights: you'll need to upweight the negatives by a factor of 198 (so that an error on a negative in your training set costs 198 times as much as an error on a positive in your positive set), since in the wild the distribution you expect to see is 2000 positives and 198000 negatives rather than 2000 positives and 1000 negatives.

The challenge is that some classifiers struggle to handle this level of class imbalance well. For instance, SVMs are known for sometimes failing badly in the presence of this kind of class imbalance; they can end up defaulting to always predict "negative", since this achieves an error rate of only 1%. In contrast, logistic regression tends to be better behaved, and provides an easy way to adjust for class imbalance.

Do some searching on class imbalance; you'll find lots written on multiple strategies you can use for handling it.

If class imbalance turns out not to be a problem and you end up with a decent classifier on your first try, and you have more time to try to improve the results, then the next thing I'd spend time on is seeing if you design better/more features, given domain knowledge about where these instances come from.