5
$\begingroup$

I am working on using machine learning to correctly predict a binary classification using an input dataset that I receive about once a month.

The idea is that I train, test and validate the classifier on one month of data, and use it to classify another month. Whether I repeatedly add the next month of data to the training dataset and upset my classifier is open for debate. I've spent a little while on this and have something that works well but it works better on some months than others. I am using either RandomForest or LinearSVC(penalty=’l1’) because I have a lot of features, around 400,000 observations and limited computing power.

I do not have much insight into the processes that produce the input dataset and based on previous analysis, I do not trust their quality control. What worries me is that one month I might receive a dataset that is very different to the data I have been training on and I wouldn’t really know unless the distribution of classes was quite different.

I am sure that I cannot be the first person to have this worry but I can’t find much information about it, so maybe I am misguided here. I’ve found the Population Stability Index, which is the right kind of idea, but I am unsure about it because it is dependent on bin size.

So my question(s):

  • How is a worry like this normally dealt with?
  • Can you point me to some ideas/case studies that will help me?
  • Is there anything I can do on the classification side that will make my classifier more robust to population instability?
Improve this question
$\endgroup$
5
  • $\begingroup$ Each month you collect 400,000 instances and you want to classify those instances, correct? And the distribution over the classes varies per month? Is that basically the problem you are describing? $\endgroup$
    – S van Balen
    Feb 20, 2018 at 8:26
  • $\begingroup$ @S van Balen it is instablity in the features that I am worried about. For example, what if the mean of Feature A for both classes increases by 20% because the data provider has made a mistake? How can I ensure that I pick this up/make a classifier that is robust to this? Obviously I could monitor the means, std devs, histograms for each feature but I was looking for other approaches. And yes ~ 400,000 instances per month. $\endgroup$
    – Stev
    Feb 20, 2018 at 9:01
  • $\begingroup$ So given that you are assuming the dataprovider makes these kinds of mistakes, which sounds suboptimal, what prevents you of just normalizing the features? $\endgroup$
    – S van Balen
    Feb 20, 2018 at 9:17
  • $\begingroup$ I agree it's suboptimal but there's not much I can do :( My understanding is that feature scaling is for reducing differences between features rather than differences between the same features within different datasets. When I do feature normalisation for the test dataset, I am using parameters calculated from the training dataset as an input (is this wrong?). If the means/distributions of the test and training datasets are different (and the difference is independent of the classes) then this could still have an adverse effect on the classification, right? $\endgroup$
    – Stev
    Feb 20, 2018 at 9:51
  • $\begingroup$ It is not wrong to normalize against the test set, but it won't help you filter out an effect that is not in the training set. You could recode your features against the average from the cohort. So you could train rules like: Feature 1 from class A is typically is one standard deviation under the average of the cohort. Those rules might also hold in a new cohort, given that there is a independent proces that influences the features per cohort. $\endgroup$
    – S van Balen
    Feb 20, 2018 at 10:11

1 Answer 1

Reset to default
1
$\begingroup$

There could be significant problems from using one time chunk in a time series to predict the next chunk. For example, if you were using data from September and October to predict retail shopping expenditure in November and December, you would be very wrong. Instead, you need to note the annual trend of increased shopping between Thanksgiving and Christmas to make correct predictions.

So, while this sort of skirts the questions you asked, I think that if you are having difficulty using one month to predict the next month, you need to start looking at longer-term trends, especially annual ones. I would investigate this first before looking into population instability.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.