I have 'practice' data set which I can split into training, validation, and test set and I will play with data to make a machine learning model. But in real situation, I will be given a very small data set which I will split into training and validation sets. There is no enough observations to make a separate test set. Then how can I estimate the 'fair' performance of the model on the real data?

The only idea I have is to find a relation between performance on the validation set and test set on the practice data (a regression line etc) and then apply the formula to the performance on the validation set of the real data to find that on the test set of the real data (which actually does not exist.)

Is there any other good idea?

  • $\begingroup$ How are you planning to train the model? by using the practice set? $\endgroup$ – Toros91 Jan 31 '18 at 2:51
  • $\begingroup$ What is the relationship between your practice data and your "real" data? Are they from the same distribution? $\endgroup$ – Imran Jan 31 '18 at 5:50

If you are working with only enough data for training and validation, consider using K-Fold Cross Validation:

One of the main reasons for using cross-validation instead of using the conventional validation (e.g. partitioning the data set into two sets of 70% for training and 30% for test) is that there is not enough data available to partition it into separate training and test sets without losing significant modelling or testing capability. In these cases, a fair way to properly estimate model prediction performance is to use cross-validation as a powerful general technique.

Assuming your "practice" data is drawn from the same distribution as your "real" data, you are on the right track by thinking about measuring the relationship between training accuracy and test accuracy in the first set to model the relationship in the second set.

However you should split your practice data into training and test only and use k-fold cross-validation on the training set. Then you should train a model on the real data with the same cross validation scheme.

For example you might get results like:

practice data:
  training accuracy: 90%
  test accuracy: 88%

real data:
  training accuracy: 89%

Since our cross-validation scheme was resistant to over-fitting, the training and test accuracy are close on the practice set, and our practice training accuracy is close to our test training accuracy. Now we can safely assume that accuracy on the unseen test set for the real data is about 2% worse than training accuracy.

However, imagine you a single validation set or no validation at all on the practice data. Now your model is more likely to overfit and you might see results like this:

practice data:
  training accuracy: 95%
  test accuracy: 75%

real data:
  training accuracy: 85%

Here we have created a model with high variance by overfitting. Training and test accuracy are far apart on the practice data, and training accuracy on the practice data does not match well with training accuracy on the real data. It is not as easy to estimate test accuracy now, because we can't really say if we have overfit by the same amount on both datasets and the second set is harder, or if we have overfit the second set by less. In the first case we might predict 65% test accuracy, and in the second case we might predict 75%.

| improve this answer | |
  • $\begingroup$ I think you are confused. Cross validation splits data into training and test, not training and validation. $\endgroup$ – user67275 Jan 31 '18 at 5:33
  • $\begingroup$ I mean, cross validation needs 'test set' but I don't have test set. $\endgroup$ – user67275 Jan 31 '18 at 5:33
  • 1
    $\begingroup$ It doesn't really matter whether you call the splits "training and validation" or "training and test" if you are not splitting into a third set. The benefit of cross validation still holds. $\endgroup$ – Imran Jan 31 '18 at 5:47
  • 1
    $\begingroup$ Of course, but cross validation is designed to give a much better estimate of test accuracy than training accuracy alone - that is the whole point! This idea is summarized in the paragraph I quoted. $\endgroup$ – Imran Jan 31 '18 at 6:10
  • 1
    $\begingroup$ I would think that extending the answer to include k-fold cross validation may help the OP. $\endgroup$ – Neil Slater Jan 31 '18 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.