0
$\begingroup$

I have heard of the terms "training" and "test error" in the context of classification quite often, but I am not sure I know what they mean. This article writes:

  • Training Error: We get the by calculating the classification error of a model on the same data the model was trained on (just like the example above).

But what is the "classification error of a model"? Is it $100\% - \text{train_accuracy}$ or is it the loss? This does not get clear to me, I'm afraid.

Edit: The paper Deep Residual Learning for Image Recognition has in Figure 6 on page 8 some plots for the training and test error and in table 6 concrete values for them. How do I get them when training a ResNet, for example?

$\endgroup$
0
$\begingroup$

Training error is simply an error that occurs during model training, i.e. dataset inappropriately handle during preprocessing or in feature selection. On the other hand testing errors are slightly different, such as model overfitting and underfitting, etc.

$\endgroup$
0
0
$\begingroup$

In order to evaluate a model (see how well it works) you usually set aside one part of the data (often randomly chosen) which is NOT used for model training (so you split the data in train and test data).

You use the train data to train some model. Models are usually trained (or estimated) based on optimization of some function (the "loss"). In linear regression for instance, you minimize the sum of squared residuals. In logistic regression you optimize a maximum-likelihood function.

In order to get some feedback on how well your (now trained) model works, you can obtain different "metrics". In regression for instance, you can look at the mean absolute error (MAE) or the mean squared error (MSE). In classification you will usually check how well your model works in making a classification. So you get some "classification error" (how well does the model work) which is not the same as looking at a loss function.

Since most models will be able to make more or less okay predictions on data the model has seen during training, using the training data to evaluate some model is not a good idea! To get an idea of how well your model works on "new" data not seen by the model during training, you use the test set to make a prediction and to obtain the classification error.

Instead (or on top of) using a test set to check your model, you can also use cross-validation. See "Introduction to Statistical Learning" (Ch. 5.1 for more details).

$\endgroup$
2
  • $\begingroup$ Thanks. I understand what you wrote about (that we use a train and test dataset) and the ResNet paper also uses a validation dataset in addition. Do I understand it correctly that the training and test error are thus 100% - accuracy? $\endgroup$
    – Hermi
    Aug 30 at 17:50
  • $\begingroup$ In neural nets, validation data is used to control success during the training process. Accuracy often is the train error, validation accuracy the error based on the validation set of the data. $\endgroup$
    – Peter
    Aug 30 at 18:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.