Classification: Dealing with unknown class balance in the prediction dataset

Question

I'm working on a classification problem that predicts if a grant application will be accepted. The data I'm training on is from 2005 to 2008. I'de like to predict any time after 2008.

The issue I'm running into is that the ratio of successful grant applications differs over time. For example, the success rate in the training set is 20%. But depending on the period I use after 2008, the success rate could be anywhere from 10 to 30%. This results in under or over predicting.

Since the success rate of grant applications is not constant, how do I account for this in my model?

Answer 1

This phenomenon is called Covariate Shift and it can affect features,target, you name it.

This happens when the test data distribution is different from the training data. Or your problem is a time-series one where time is the biggest independent variable.

First case : No model can adapt to the target shifting in its behavior. You have to either wait until you collect enough data from the current times, and retrain a fairly satisfying model, or do online learning which means that every datapoint that you predict for , is fed into the model to be trained on. That way; if changes in the target occur, your model will get updated with every new datapoint.

Second case : Lets say your target variable has a tendency to go up every 2 years, and then goes down. This is an assumption i'm making. Your task is to detect that shift in time that affects your target and model it. You create a model that takes into account that every 2 years, your target goes down. and so on. It's what we call a time-series problem where time is the biggest contributor to the target's changes.

If the second case doesn't make sense to you, and doesn't relate to your problem, then look at first case.

Answer 2

I do not think in any model I ever made was the event rate in every future time period the same as the training period. The point of the model, in this case binary classification, is to find the events and non-events (0's and 1's). The event rate(balance) of the prediction periods is irrelevant to the event rate of the trained model. Each record is scored independent in the binary classification model. The event rate of the training data effects the training.

If the model has poor performance in some periods, analyze why. It means some records are being classified wrong. It could be the threshold used, it could be there is data drift, it could be the model is weak with certain patterns that have become more prevalent.

Perhaps you need to retrain the model (and over/under-sample or not) or find new features or change the threshold or other actions, but not because of the event rate of the future periods. It will be because the model is not performing up to the problem you are trying to solve.