how should I measure performance if there is no test data?

I have 'practice' data set which I can split into training, validation, and test set and I will play with data to make a machine learning model. But in real situation, I will be given a very small data set which I will split into training and validation sets. There is no enough observations to make a separate test set. Then how can I estimate the 'fair' performance of the model on the real data?

The only idea I have is to find a relation between performance on the validation set and test set on the practice data (a regression line etc) and then apply the formula to the performance on the validation set of the real data to find that on the test set of the real data (which actually does not exist.)

Is there any other good idea?

• How are you planning to train the model? by using the practice set? Jan 31, 2018 at 2:51
• What is the relationship between your practice data and your "real" data? Are they from the same distribution? Jan 31, 2018 at 5:50

If you are working with only enough data for training and validation, consider using K-Fold Cross Validation:

One of the main reasons for using cross-validation instead of using the conventional validation (e.g. partitioning the data set into two sets of 70% for training and 30% for test) is that there is not enough data available to partition it into separate training and test sets without losing significant modelling or testing capability. In these cases, a fair way to properly estimate model prediction performance is to use cross-validation as a powerful general technique.

Assuming your "practice" data is drawn from the same distribution as your "real" data, you are on the right track by thinking about measuring the relationship between training accuracy and test accuracy in the first set to model the relationship in the second set.

However you should split your practice data into training and test only and use k-fold cross-validation on the training set. Then you should train a model on the real data with the same cross validation scheme.

For example you might get results like:

practice data:
training accuracy: 90%
test accuracy: 88%

real data:
training accuracy: 89%


Since our cross-validation scheme was resistant to over-fitting, the training and test accuracy are close on the practice set, and our practice training accuracy is close to our test training accuracy. Now we can safely assume that accuracy on the unseen test set for the real data is about 2% worse than training accuracy.

However, imagine you a single validation set or no validation at all on the practice data. Now your model is more likely to overfit and you might see results like this:

practice data:
training accuracy: 95%
test accuracy: 75%

real data:
training accuracy: 85%


Here we have created a model with high variance by overfitting. Training and test accuracy are far apart on the practice data, and training accuracy on the practice data does not match well with training accuracy on the real data. It is not as easy to estimate test accuracy now, because we can't really say if we have overfit by the same amount on both datasets and the second set is harder, or if we have overfit the second set by less. In the first case we might predict 65% test accuracy, and in the second case we might predict 75%.

• I think you are confused. Cross validation splits data into training and test, not training and validation. Jan 31, 2018 at 5:33
• I mean, cross validation needs 'test set' but I don't have test set. Jan 31, 2018 at 5:33
• It doesn't really matter whether you call the splits "training and validation" or "training and test" if you are not splitting into a third set. The benefit of cross validation still holds. Jan 31, 2018 at 5:47
• Of course, but cross validation is designed to give a much better estimate of test accuracy than training accuracy alone - that is the whole point! This idea is summarized in the paragraph I quoted. Jan 31, 2018 at 6:10
• I would think that extending the answer to include k-fold cross validation may help the OP. Jan 31, 2018 at 8:32