What are the “training error” and “test error” used in deep learning papers?

Question

I have heard of the terms "training" and "test error" in the context of classification quite often, but I am not sure I know what they mean. This article writes:

• Training Error: We get the by calculating the classification error of a model on the same data the model was trained on (just like the example above).

But what is the "classification error of a model"? Is it $100\% - \text{train_accuracy}$ or is it the loss? This does not get clear to me, I'm afraid.

Edit: The paper Deep Residual Learning for Image Recognition has in Figure 6 on page 8 some plots for the training and test error and in table 6 concrete values for them. How do I get them when training a ResNet, for example?

Answer 1

Training error is simply an error that occurs during model training, i.e. dataset inappropriately handle during preprocessing or in feature selection. On the other hand testing errors are slightly different, such as model overfitting and underfitting, etc.

Answer 2

In order to evaluate a model (see how well it works) you usually set aside one part of the data (often randomly chosen) which is NOT used for model training (so you split the data in train and test data).

You use the train data to train some model. Models are usually trained (or estimated) based on optimization of some function (the "loss"). In linear regression for instance, you minimize the sum of squared residuals. In logistic regression you optimize a maximum-likelihood function.

In order to get some feedback on how well your (now trained) model works, you can obtain different "metrics". In regression for instance, you can look at the mean absolute error (MAE) or the mean squared error (MSE). In classification you will usually check how well your model works in making a classification. So you get some "classification error" (how well does the model work) which is not the same as looking at a loss function.

Since most models will be able to make more or less okay predictions on data the model has seen during training, using the training data to evaluate some model is not a good idea! To get an idea of how well your model works on "new" data not seen by the model during training, you use the test set to make a prediction and to obtain the classification error.

Instead (or on top of) using a test set to check your model, you can also use cross-validation. See "Introduction to Statistical Learning" (Ch. 5.1 for more details).