If you are working with only enough data for training and validation, consider using K-Fold Cross Validation:
One of the main reasons for using cross-validation instead of using the conventional validation (e.g. partitioning the data set into two sets of 70% for training and 30% for test) is that there is not enough data available to partition it into separate training and test sets without losing significant modelling or testing capability. In these cases, a fair way to properly estimate model prediction performance is to use cross-validation as a powerful general technique.
Assuming your "practice" data is drawn from the same distribution as your "real" data, you are on the right track by thinking about measuring the relationship between training accuracy and test accuracy in the first set to model the relationship in the second set.
However you should split your practice data into training and test only and use k-fold cross-validation on the training set. Then you should train a model on the real data with the same cross validation scheme.
For example you might get results like:
training accuracy: 90%
test accuracy: 88%
training accuracy: 89%
Since our cross-validation scheme was resistant to over-fitting, the training and test accuracy are close on the practice set, and our practice training accuracy is close to our test training accuracy. Now we can safely assume that accuracy on the unseen test set for the real data is about
2% worse than training accuracy.
However, imagine you a single validation set or no validation at all on the practice data. Now your model is more likely to overfit and you might see results like this:
training accuracy: 95%
test accuracy: 75%
training accuracy: 85%
Here we have created a model with high variance by overfitting. Training and test accuracy are far apart on the practice data, and training accuracy on the practice data does not match well with training accuracy on the real data. It is not as easy to estimate test accuracy now, because we can't really say if we have overfit by the same amount on both datasets and the second set is harder, or if we have overfit the second set by less. In the first case we might predict
65% test accuracy, and in the second case we might predict