Time-based over-sampling dilemma

Question

Background:

I'm working on a binary classifier that tries to predict when -- if ever -- a user goes bad, a terminal state from which a user cannot recover. This phenomenon is tricky becuase a user might start off good, go bad suddenly and only get caught after a long time. Currently I've defined my target variable at the user-month level, so for each user there is one training record for every month that the user existed.

Question: Should I only include a user's first bad month, all bad months, or something else?

Additional Context:

The problem I'm facing is that some users go bad and remain undetected for many, many months. If I include all of a user's bad months, a small percentage of the users dominate and the model focuses on them at the expense of other bad users. When I include all bad months, I have one feature in particular that performs extremely well (causes the model to have a high AUC value). When I include only a user's first bad month, this same feature has little to no importance and many of the dominant bads go undetected.

I know that today there are some users that exhibit the same signs of the dominant bads; and I suspect that they are in fact bad. If I train my model on only the first bad month and deploy it into production, I'm worried my model will likely miss the obvious. What's the best way to tackle this problem? I'm also open to using something other than client-months, e.g., "first bad occurrence in the next 6 months".

Answer 1

If you have time series for good clients and time series for bad clients, I would suggest using something along the lines of dynamic time warping to get out of a time domain, and into a dissimilarity domain.

From there you could just use just about any classifier that you can think of, I have a silly sort of toy example that detects whether or not it is a superbowl sunday based on traffic to a website. https://github.com/Rblivingstone/SuperBowl.

But the nice thing is that now that you are in dissimilarity space you can classify based on a few exemplars, say two or three columns in dissimilarity matrix, and feed that into your classifier. It should be relatively easy to implement for your particular use case.