My company has recently engaged a consultant firm to develop a predictive model to detect defective works.
I understand that there are many ways to validate the model, for example, using k-fold cross-validation and I believe that the consultant firm will carry out the validation before submitting the model to us.
However, at the employer's side, how can I check the accuracy of the model developed by the consultant firm ??
Someone suggested that I can give 2000-2015 data to the consultant firm and keep 2016 data for our own checking. However, a model with good accuracy on 2016 data does not imply that it will have good predictive power in the future. In my view, keeping 2016 data for checking seems like adding one more test set for validation, which in my view, is unnecessary since I already hv "k-fold" cross validation.
Could someone advise what the employer can do to check the consultant's model?