3
$\begingroup$

My company has recently engaged a consultant firm to develop a predictive model to detect defective works.

I understand that there are many ways to validate the model, for example, using k-fold cross-validation and I believe that the consultant firm will carry out the validation before submitting the model to us.

However, at the employer's side, how can I check the accuracy of the model developed by the consultant firm ??

Someone suggested that I can give 2000-2015 data to the consultant firm and keep 2016 data for our own checking. However, a model with good accuracy on 2016 data does not imply that it will have good predictive power in the future. In my view, keeping 2016 data for checking seems like adding one more test set for validation, which in my view, is unnecessary since I already hv "k-fold" cross validation.

Could someone advise what the employer can do to check the consultant's model?

$\endgroup$
  • $\begingroup$ yes hold out the 16 data for testing; that is a good idea $\endgroup$ – Mast Malang Nov 30 '16 at 16:09
  • $\begingroup$ Can you withhold data for each year from 2000-2015 and still leave them enough data to work with? It would be nice to have test data for each year as this would give you an idea of year-to-year variation of the model. And then also hold onto all of the 2016 data. $\endgroup$ – Hobbes Nov 30 '16 at 16:22
  • 1
    $\begingroup$ The answer of how to test model depends greatly on what you will know about it.. How exactly is the consultant firm delivering the model results to you? Is it going to be purely a black box, and will you be able to see the complete model form and parameters? How will it be operationalized? $\endgroup$ – Paul Nov 30 '16 at 21:17
  • $\begingroup$ Basically it will be an model we will apply directly to detect defects for many years onward. We don't have in-house data scientist or statistician to modify/tune the model over time. Most likely, we will use it for quite some years until the model become too inaccurate to be used. I am learning data science that is why I have these questions in mind but I am not very good in delivering my concern. Thanks all for helping me out =] all are great advices to me. $\endgroup$ – Alex Yu Dec 4 '16 at 17:41
4
$\begingroup$

Cross validation can be used in parameter tuning or model selection, but it does not evaluate the performance of a model.

When developing a model, you divide your data between train, validation and testing. In the best case scenario, testing is only used once at the end to score the model. You should definitely keep the 2016 data.

If you give all your data, it is easy to have a model learning "by heart" your expected data but it will not generalize well to future years. This is overfiting. The only way to know is by testing it on unknown data, here, 2016

Employed for measuring model performance, cross validation can measure more than just the average accuracy and you can select your features to answer the best accuracy score

| improve this answer | |
$\endgroup$
2
$\begingroup$

I would agree with the suggestion with holding out the 2016 data to check the external agency's work. Without inspecting the code, you just can't be sure that the k-fold cross-validation process has been performed properly.

Another benefit of using the 2016 held out set is that you can see if the model trained on past years' data work well in future years data. There might be a concept drift in the new year as the true relationship between Y and Xs changes. In cross validation, each fold belongs to the same time period as the training data set.

| improve this answer | |
$\endgroup$
  • $\begingroup$ What if "time" itself is one of the parameter? I think it is reasonable to assume that "service life" of the equipment is one reasonable factor that may cause defect? So new machine should perform better (produce less defective work) than old machine. $\endgroup$ – Alex Yu Dec 4 '16 at 17:46
  • $\begingroup$ What if "time" itself is one of the parameter? I think it is reasonable to assume that "service life" of the equipment is one reasonable factor that may cause defect? So new machine should perform better (produce less defective work) than old machine. This is why I am a bit concern about holding all 2016 data, because all machine will be "one year" older? And there are new machines added in the production line in 2016 too. $\endgroup$ – Alex Yu Dec 4 '16 at 17:49
  • $\begingroup$ @AlexYu That is not an issue if you have a continuous variable named [AgeInYears]. So at the end of 2015 training set, you'd have some machines that are added in 2015, [AgeInYears] = 0. In 2016 scoring set, you could still have new machines added to it with [AgeInYears] =0. The only new value in the variable is that the oldest machine is increased by 1. Say in 2015 the oldest machine was 10 years old, in 2016 it would be 11 which is a new value. But it wouldn't be a problem if it is a continuous predictor (some algorithms wouldn't be able to predict if it is a new categorical value). $\endgroup$ – jkyh Dec 4 '16 at 20:30
1
$\begingroup$

A lot of people are suggesting holding 2016 data, but what you should hold out as a test set depends on what is being predicted. If defective works do not depend on date/time, and the date/time is not being used in the model, it may make sense to hold a random sample (at least w.r.t. date/time) for your tests.

If you have features that you want to avoid extrapolating from, then split by those features e.g. if there is some common "location" property for your items, you may want to split by identity of location, because you want to use the model to make predictions in new locations (and the location as a one-hot feature would not have any predictive value for a new location, even if your target class was correlated with existing locations).

Ideally the consultancy firm will be helping you here to identify correlations in the data set that you don't want to use in predictions, because they could affect the generalisation you want when using the model in production. If any such thing is identified, then it clearly points to using that feature to split test data by. If there is no such issue, then you may as well split randomly.

| improve this answer | |
$\endgroup$
  • $\begingroup$ What if "time" itself is one of the parameter? I think it is reasonable to assume that "service life" of the equipment is one reasonable factor that may cause defect? So new machine should perform better (produce less defective work) than old machine. This is why I am a bit concern about holding all 2016 data, because all machine will be "one year" older? And there are new machines added in the production line in 2016 too. $\endgroup$ – Alex Yu Dec 4 '16 at 17:49
  • $\begingroup$ @AlexYu If there is a possible correlation to time that you don't want including in the model, then yes holding test data by date makes sense. But you should be identifying specifically how you need your model to generalise, and holding test data that will accurately report according to that goal. Otherwise you risk getting data leaks during test and over-estimating the model performance compared to what you will see live. But holding back data that doesn't need to be held back might also reduce effectiveness of the model. $\endgroup$ – Neil Slater Dec 4 '16 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.