Ideal score of a model on training and cross validation data

Question

The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here.

I have trained a classifier model for binary classification task. Now I am trying to fine tune the model. With different sets of hyperparameters I am getting different sets of accuracy on my train and test set. For example:

(1) Train set: 0.99 | Cross-validation set: 0.72
(2) Train set: 0.75 | Cross-validation set: 0.70
(3) Train set: 0.69 | Cross-validation set: 0.69

These are approximate numbers. But my point is - for certain set of hyperparameters I am getting more or less similar CV accuracy, while the accuracy on training data varies from overfit to not so much overfit.

My question is - which of these models will work best on future unseen data? What is the recommendation in this scenario, shall we choose the model with higher training accuracy or lower training accuracy, given that CV accuracy is similar in all cases above (in fact CV score is better in the overfitted model)?

Answer 1

Accuracy on the training data basically doesn't count. I don't quite want to say to ignore it, because a train/test accuracy of 100/70 seems different to me than a train/test accuracy of 71/70, but you're mostly not interested in performance on the training data.

Using a test set mimics a real application of machine learning. Think about Siri or Alexa. The goal is to predict speech that it hasn't heard. There's no way to know how it will perform on such speech, so the next-best approach is to use some data where you know the answer but hide it from your model. After you train the model, assess how it performs on data where it has not seen the answer. If the model is accurate, then that's a good sign about its ability to perform on real speech recognition tasks.

Training data is like the practice questions or homework problems, while test data is the exam.