I'm working on a binary classifier that tries to predict when -- if ever -- a user goes bad, a terminal state from which a user cannot recover. This phenomenon is tricky becuase a user might start off good, go bad suddenly and only get caught after a long time. Currently I've defined my target variable at the user-month level, so for each user there is one training record for every month that the user existed.
Question: Should I only include a user's first bad month, all bad months, or something else?
The problem I'm facing is that some users go bad and remain undetected for many, many months. If I include all of a user's bad months, a small percentage of the users dominate and the model focuses on them at the expense of other bad users. When I include all bad months, I have one feature in particular that performs extremely well (causes the model to have a high AUC value). When I include only a user's first bad month, this same feature has little to no importance and many of the dominant bads go undetected.
I know that today there are some users that exhibit the same signs of the dominant bads; and I suspect that they are in fact bad. If I train my model on only the first bad month and deploy it into production, I'm worried my model will likely miss the obvious. What's the best way to tackle this problem? I'm also open to using something other than client-months, e.g., "first bad occurrence in the next 6 months".