I am working on using machine learning to correctly predict a binary classification using an input dataset that I receive about once a month.
The idea is that I train, test and validate the classifier on one month of data, and use it to classify another month. Whether I repeatedly add the next month of data to the training dataset and upset my classifier is open for debate. I've spent a little while on this and have something that works well but it works better on some months than others. I am using either RandomForest or LinearSVC(penalty=’l1’) because I have a lot of features, around 400,000 observations and limited computing power.
I do not have much insight into the processes that produce the input dataset and based on previous analysis, I do not trust their quality control. What worries me is that one month I might receive a dataset that is very different to the data I have been training on and I wouldn’t really know unless the distribution of classes was quite different.
I am sure that I cannot be the first person to have this worry but I can’t find much information about it, so maybe I am misguided here. I’ve found the Population Stability Index, which is the right kind of idea, but I am unsure about it because it is dependent on bin size.
So my question(s):
- How is a worry like this normally dealt with?
- Can you point me to some ideas/case studies that will help me?
- Is there anything I can do on the classification side that will make my classifier more robust to population instability?