Time series binary classificaiton with labelling issues

Question

My situation is quite complicated so I will give a similar example from a simpler domain. Suppose we want to try to predict WHEN a mobile game users will make a purchase if given a sale. Almost every user is always instantaneously a non-purchaser because everybody is constantly not buying anything. Some people buy something to instantaneously become a purchaser but go back to the standard state of not purchasing. We have a time-stamp for that purchase. If we could understand the user behavior well enough, we could predict when a user will purchase before they do so. If this is the case then users with actions and game states very similar to the purchasers but who do not actually purchase would do so if given a sale. So the question is how do we turn this into a machine learning question. My current plan is to use binary classification by labeling the YES cases as the purchasers since those who purchase at regular price would have purchased if given a sale. I build my features based on actions in recent look back windows, like how many actions of a type in the last day. For all users who never purchase I choose random time-stamps, build my features and use them as the NO case. Since I can choose as many NO time-stamps as I want I have been doing 100 times my YES case. Then I can use a classifier, I like tree ensemble methods but I do not think the classifier will really matter here.

The problem is that this is not working. There are two things worth noting which I think are at the core of the issue I am having. First, we are trying to predict the "when" not just the "who". Users who play a lot or who have purchased before are much more likely to purchase in general so it is easy to make a time independent classifier and predict who is likely to purchase. This means that the features which are useful for the easy time-independant problem can sort of "contaminate" the feature set. I have taken out a number of these features and seen some improvement. The second issue has to do with labelling. We have many YES cases for the purchasers but how does one define a NO case. I described what I am doing above but I am not sure that it is really a best method. Many of the users I defined as a NO might have effectively been a YES since if they were given the sale they would have bought.

Also, I should point out two situations which this is not. This is not one class binary classification which generally uses the large class in the imbalanced case reducing the problem to unsupervised anomaly/outlier detection. This is also not the semi-supervised situation where I have both classes labelled for a subset of the data. I really only have some of the YES cases for sure and none of the NO case. Any thoughts welcome.

Answer 1

From your description of the data, this is not a time-series problem. Time is not a factor here, for each user you have a set of variables after you choose a time threshold. Although you seek the "when", that's probably out of your reach, based on your explanation.

If I understand correctly you're doing classification but you're unsure about your "data collection process", at least for the "NO" labeled obs.

Why not fix a time stamp, and register all the variables you think are correlated with the outcome, and keep the outcome as you defined (never buys = "NO", buys at least once = "YES).

I'd treat this as a classification problem but instead of looking at the classic metrics (from confusion matrix), I'd be more interested at the probabilities you get from each obs, compared to the truth (that you know up to that point in time).

Imagine a user, with an estimated probability of 0.90 (90%), but the labels says "NO". He's someone who the model thinks that should have bought but didn't, he's highly likely to be interested in a sale.

Maybe you could cluster on this kind of information, creating groups of users where your model is wrong, but it's really confident in it's estimation. Basically you're more interested in false positive obs.

But also false negatives can give you insight, again looking at the probabilities you get.

I know this won't probabily be possible but if you could collect more data, I'd suggest a variable (or more), that measure the relationship of the user with sales.

Also, I agree about the decision of removing the fact that the user plays for long, that's probably "too much correlated" with your outcome.

Without knowing more about the data, that's my two cents.

Answer 2

I have found a solution. My suspicions were correct and there are ways to solve both issues.

For the issue about only knowing the definite yesses and not knowing both yes and no there is a type of machine learning specifically for this called PU-learning. There are libraries for this built off scikit-learn. This type of machine learning learns the binary classification problem based off training data where the target is either labelled yes or not at all. It then attempts to find a decision boundary by classifying the unlabeled data as yes or no. This does not totally solve the issue of where the decision boundary should be for who will be a good person to offer a sale but I can do this by tuning my ROC curve under business constraints.

The second issue was about data and feature engineering. Basically it is a bad idea to do data prep differently for the different classes. Labeling my yes cases by the purchase event and then looking back at activity is not good. I need to choose random times to get the distributions when you predict correct. If you do not your probability is inaccurate. I also need to have many timestamps per user to sample different states of that user. This means that you will get some events labelled as yes for a user and some unlabelled for the same user. One would then expect that the boundary is drawn with some of the unlabelled timestamps on either side of the decision boundary for each user. This would then suppress the problem of contamination by the features which are only good from a time independent perspective. Explicitly the method is to select all (or a subset of) sign-in events. You build your features based on that event in whatever way you see fit. The labeling is done by joining on the purchase events to look forward and see if they purchase in the near future. The definition of 'near' needs to be tuned but you can likely make an educated guess from a plot. If there is a purchase event then that sign in event is labelled yes otherwise left unlabelled. The PU-learning does the rest.