My situation is quite complicated so I will give a similar example from a simpler domain. Suppose we want to try to predict WHEN a mobile game users will make a purchase if given a sale. Almost every user is always instantaneously a non-purchaser because everybody is constantly not buying anything. Some people buy something to instantaneously become a purchaser but go back to the standard state of not purchasing. We have a time-stamp for that purchase. If we could understand the user behavior well enough, we could predict when a user will purchase before they do so. If this is the case then users with actions and game states very similar to the purchasers but who do not actually purchase would do so if given a sale. So the question is how do we turn this into a machine learning question. My current plan is to use binary classification by labeling the YES cases as the purchasers since those who purchase at regular price would have purchased if given a sale. I build my features based on actions in recent look back windows, like how many actions of a type in the last day. For all users who never purchase I choose random time-stamps, build my features and use them as the NO case. Since I can choose as many NO time-stamps as I want I have been doing 100 times my YES case. Then I can use a classifier, I like tree ensemble methods but I do not think the classifier will really matter here.
The problem is that this is not working. There are two things worth noting which I think are at the core of the issue I am having. First, we are trying to predict the "when" not just the "who". Users who play a lot or who have purchased before are much more likely to purchase in general so it is easy to make a time independent classifier and predict who is likely to purchase. This means that the features which are useful for the easy time-independant problem can sort of "contaminate" the feature set. I have taken out a number of these features and seen some improvement. The second issue has to do with labelling. We have many YES cases for the purchasers but how does one define a NO case. I described what I am doing above but I am not sure that it is really a best method. Many of the users I defined as a NO might have effectively been a YES since if they were given the sale they would have bought.
Also, I should point out two situations which this is not. This is not one class binary classification which generally uses the large class in the imbalanced case reducing the problem to unsupervised anomaly/outlier detection. This is also not the semi-supervised situation where I have both classes labelled for a subset of the data. I really only have some of the YES cases for sure and none of the NO case. Any thoughts welcome.