5
$\begingroup$

The title says it all: I have seen three terms for functions so far, that seem to be the same / similar:

  • error function
  • criterion function
  • cost function
  • objective function
  • loss function

I was working on classification problems

$$E(W) = \frac{1}{2} \sum_{x \in E}(t_x-o(x))^2$$

where $W$ are the weights, $E$ is the evaluation set, $t_x$ is the desired output (the class) of $x$ and $o(x)$ is the given output. This function seems to be commonly called "error function".

But while reading about this topic, I've also seen the terms "criterion function" and "objective function". Do they all mean the same for neural nets?

  • Geoffrey Hinton called cross-entropy for softmax-neurons and $E(W) = \frac{1}{2} \sum_{x \in E}(t_x-o(x))^2$ a cost function.
$\endgroup$
  • 3
    $\begingroup$ An answer provided on SE explains some differences between these similar terms. I believe criterion, error, & cost are synonyms however. $\endgroup$ – cdeterman Feb 15 '16 at 19:41
3
$\begingroup$

The error function is the function representing the difference between the values computed by your model and the real values. In the optimization field often they speak about two phases: a training phase in which the model is set, and a test phase in which the model tests its behaviour against the real values of output. In the training phase the error is necessary to improve the model, while in the test phase the error is useful to check if the model works properly.

The objective function is the function you want to maximize or minimize. When they call it "cost function" (again, it's the objective function) it's because they want to only minimize it. I see the cost function and the objective function as the same thing seen from slightly different perspectives.

The "criterion" is usually the rule for stopping the algorithm you're using. Suppose you want that your model find the minimum of an objective function, in real experiences it is often hard to find the exact minimum and the algorithm could continuing to work for a very long time. In that case you could accept to stop it "near" to the optimum with a particular stopping criterion.

I hope I gave you a correct idea of these topics.

$\endgroup$
  • 2
    $\begingroup$ Based on this definition I guess "loss function" is a synonym to "cost function"? (i.e. one that we want to minimize"). Or does it fall under a separate bucket? $\endgroup$ – Atlas7 May 11 '17 at 15:10
  • 1
    $\begingroup$ Based on my knowledge, 'loss function' is just another way to call the 'cost function' so.. same bucket! :) $\endgroup$ – Andrea Ianni ௫ May 12 '17 at 9:00
1
$\begingroup$

objective function aka criterion - a function to be minimized or maximized

  • error function - an objective function to be minimized

    • aka cost, energy, loss, penalty, regret function, where in some scenarios loss is with respect to a single example and cost is with respect to a set of examples
  • utility function - an objective function to be maximized

    • aka fitness, profit, reward function
$\endgroup$
1
$\begingroup$

When applied to machine learning (ML), these terms could all mean the same thing or not, depending on the context.

From the optimization standpoint, one would always like to have them minimized (or maximized) in order to find the solution to ML problem.

Each term came from a different field (optimization, statistics, decision theory, information theory, etc.) and brought some overlapping to the mixture:

it is quite common to have a loss function, composed of the error + some other cost term, used as the objective function in some optimization algorithm :-)

When dealing with modern neural networks, almost any error function could be eventually called a cost/loss/objective and the criterion at the same time. Therefore, it is important to distinguish between their usages:

  • functions optimized directly while training: usually referred to as loss functions, but it is quite common to see the term "cost", "objective" or simply "error" used as well. These functions can be combinations of several other loss or functions, including different error terms and regularizes (e.g., mean-squared error + L1 norm of the weights).

  • functions optimized indirectly: usually referred to as metrics. These are used as criteria for performance evaluation and for other heuristics (e.g., early stopping, cross-validation). Almost any loss function can be used as a metric, which is quite common. The opposite, however, may not work well since commonly used metric functions (such as F1, AUC, IoU and even binary accuracy) are not suitable to be optimized directly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.