3
$\begingroup$

I had a small discussion with my friends on overfitting and we became confused over the two terms: "training accuracy" and "training loss (or cost)". This is the first time I've heard the term training accuracy. So far, I have only calculated accuracy on the validation and test sets.

My understanding is that training accuracy and training cost are just one thing, and more generally accuracy is only applied for classification problems.

Is that correct?

$\endgroup$
2
  • 2
    $\begingroup$ What do you mean by "training cost"? I guess you are not referring to the actual money it takes to train a model but maybe to the "loss function"? Please, clarify. $\endgroup$
    – noe
    Commented Nov 22, 2023 at 11:25
  • 1
    $\begingroup$ You're right, I missed this point actually! It's the loss reported in training the model (using Tensorflow lib indeed) but somehow my friend used the term "cost" in the log report. $\endgroup$
    – Tran Khanh
    Commented Nov 22, 2023 at 11:33

2 Answers 2

6
$\begingroup$

The concepts of "loss" and "accuracy" are NOT the same.

The loss function is what you minimize during the training of your model. There are many types of loss functions. You usually choose the loss function depending on the "task" you are facing; for instance, binary classification uses binary cross-entropy as loss function, multi-class classification uses categorical cross-entropy, regression uses mean squared-error (MSE) or mean absolute error (MAE).

Accuracy is a concept from classification (either binary classification or multiclass classification). It is defined as the percentage of correctly classified elements.

Of course, during training, when the loss decreases, we expect that the accuracy increases. This, however, is not always the case. That's why it is important to monitor both loss function and accuracy in both the training and validation data, so that we understand how our model actually performs.

Note that accuracy has its own problems representing the performance of a model. For instance, if a binary classification dataset has 95% positive elements and 5% negative elements, a classifier that ALWAYS classifies as positive will obtain a 95% accuracy. There are other measures that account for this, like the area under the ROC curve (AUC) or the F1.

$\endgroup$
5
  • $\begingroup$ "This, however, is not always the case" <- Everything is clear to me except this one. My understanding is that the accuracy (on classification tasks), if measured on the training set, is the same as the training cost (=avaraged of losses incurs on all training examples), is that correct? $\endgroup$
    – Tran Khanh
    Commented Nov 22, 2023 at 13:09
  • 2
    $\begingroup$ No, that's not correct. Note that the output of a binary classification model is a probability between 0 and 1; the loss is the binary cross-entropy between that probability and the actual target. On the other hand accuracy refers to counts of the thresholded prediction (you take a threshold and assign the probability value to either 0 or 1) against the actual target. I suggest you read this answer for a detailed explanation on the relationship of both. $\endgroup$
    – noe
    Commented Nov 22, 2023 at 13:40
  • $\begingroup$ that's true also as I recall we can use a surrogate loss for example. However, I forgot to mention that the context is for the popular 0-1 loss function, and the classification is binary. In such case I dont think there's any difference? $\endgroup$
    – Tran Khanh
    Commented Nov 22, 2023 at 14:18
  • 1
    $\begingroup$ The 0-1 loss function is indeed the same as one minus the accuracy. However, note that the 0-1 loss function is not differentiable, so you cannot optimize on it (unless you put other stuff in place, like a straight-through estimator). $\endgroup$
    – noe
    Commented Nov 22, 2023 at 14:50
  • 1
    $\begingroup$ Your above comment does remind me that the loss has to be something else since my friend using TF with GD for training. Thanks for that too! $\endgroup$
    – Tran Khanh
    Commented Nov 22, 2023 at 15:17
1
$\begingroup$

These terms are all used interchangeably (unfortunately) but they are all referring to the same concepts ("training loss" and "training accuracy", which is not to be confused with a specific metric also called by that name - "accuracy"). "Accuracy" is the evaluation estimate of some metric, usually this would be using the same cost function that was used in training the model, however sometimes people use a different metric for this purpose. "Accuracy" is just a general english word meaning the quality or state of being correct or precise, that is how well the model predicts with the cost function of choice (on whichever data set), and I believe this is what was meant in your case as well, when referring to "training accuracy".

Cost is short for "value of the cost function", while accuracy is short for "predictive accuracy". Both of these terms are general and they do not represent any specific measures. Both of these can be, and usually are, computed on all sets (training/validation/testing, although only certain results may be reported in the end, as people usually do not care about training "cost").

The cost function is the specific measure you are using, such as MSE, and the estimate of it on some data is the "cost" or "predictive accuracy". While these are synonyms it might be better to use "cost" instead of "accuracy", as that is often times confused with the metric "accuracy", which is a specific measure used in classification.

A possible reason why people use "cost" for reporting training cost and "accuracy" when evaluating it is because they use different metrics on the same data. For ex. for a classification task someone might choose to train a model using binary cross-entropy but then evaluate it using accuracy metric (because cross-entropy is not as easy to interpret as accuracy), which would give you two different values for the same model and data. This is not good practice, you should evaluate a model using the same metric which was used for training it, as that is what was actually optimized by the model.

$\endgroup$
5
  • 3
    $\begingroup$ I disagree with this, accuracy and loss/cost functions are not and should not be used interchangeably, for the reasons mentioned by @noe. $\endgroup$ Commented Nov 22, 2023 at 14:52
  • $\begingroup$ @AdamJaamour I don't know what you are disagreeing with, accuracy as a metric is not the same as cost functions, that's what I also said. I was comparing it with "forecasting accuracy" which is a much more vague and unclear concept. $\endgroup$ Commented Nov 22, 2023 at 15:20
  • $\begingroup$ @user2974951 likely the disagreement is on your first sentence: "they are all referring to the same concepts" $\endgroup$
    – justhalf
    Commented Nov 23, 2023 at 3:10
  • $\begingroup$ @justhalf fair enough, I added some more words to explain what I meant by that. $\endgroup$ Commented Nov 23, 2023 at 7:54
  • 1
    $\begingroup$ Sorry about that @user2974951, I did read that sentence and disagreed, but didn't finish reading the whole answer. My bad! Removed my downvote :) $\endgroup$ Commented Nov 23, 2023 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.