What is the difference between (objective / error / criterion / cost / loss) function in the context of neural networks?

Question

The title says it all: I have seen three terms for functions so far, that seem to be the same / similar:

• error function
• criterion function
• cost function
• objective function
• loss function

I was working on classification problems

$$E(W) = \frac{1}{2} \sum_{x \in E}(t_x-o(x))^2$$

where $W$ are the weights, $E$ is the evaluation set, $t_x$ is the desired output (the class) of $x$ and $o(x)$ is the given output. This function seems to be commonly called "error function".

But while reading about this topic, I've also seen the terms "criterion function" and "objective function". Do they all mean the same for neural nets?

• Geoffrey Hinton called cross-entropy for softmax-neurons and $E(W) = \frac{1}{2} \sum_{x \in E}(t_x-o(x))^2$ a cost function.

Answer 1

The error function is the function representing the difference between the values computed by your model and the real values. In the optimization field often they speak about two phases: a training phase in which the model is set, and a test phase in which the model tests its behaviour against the real values of output. In the training phase the error is necessary to improve the model, while in the test phase the error is useful to check if the model works properly.

The objective function is the function you want to maximize or minimize. When they call it "cost function" (again, it's the objective function) it's because they want to only minimize it. I see the cost function and the objective function as the same thing seen from slightly different perspectives.

The "criterion" is usually the rule for stopping the algorithm you're using. Suppose you want that your model find the minimum of an objective function, in real experiences it is often hard to find the exact minimum and the algorithm could continuing to work for a very long time. In that case you could accept to stop it "near" to the optimum with a particular stopping criterion.

I hope I gave you a correct idea of these topics.

Answer 2

When applied to machine learning (ML), these terms could all mean the same thing or not, depending on the context.

From the optimization standpoint, one would always like to have them minimized (or maximized) in order to find the solution to ML problem.

Each term came from a different field (optimization, statistics, decision theory, information theory, etc.) and brought some overlapping to the mixture:

it is quite common to have a loss function, composed of the error + some other cost term, used as the objective function in some optimization algorithm :-)

When dealing with modern neural networks, almost any error function could be eventually called a cost/loss/objective and the criterion at the same time. Therefore, it is important to distinguish between their usages:

• functions optimized directly while training: usually referred to as loss functions, but it is quite common to see the term "cost", "objective" or simply "error" used as well. These functions can be combinations of several other loss or functions, including different error terms and regularizes (e.g., mean-squared error + L1 norm of the weights).

• functions optimized indirectly: usually referred to as metrics. These are used as criteria for performance evaluation and for other heuristics (e.g., early stopping, cross-validation). Almost any loss function can be used as a metric, which is quite common. The opposite, however, may not work well since commonly used metric functions (such as F1, AUC, IoU and even binary accuracy) are not suitable to be optimized directly.

Answer 3

objective function aka criterion - a function to be minimized or maximized

• error function - an objective function to be minimized

  • aka cost, energy, loss, penalty, regret function, where in some scenarios loss is with respect to a single example and cost is with respect to a set of examples

• utility function - an objective function to be maximized

  • aka fitness, profit, reward function

Answer 4

From my knowledge from the Deep Learning book (Ian Goodfellow), the cost function, error function, objective function and loss function are the same.

In statistics, we use the term objective function which is to be optimised(maximised or minimised).

Since the objective functions in ML almost always deals with the error generated by the model, it must be minimised only. Thus the term cost function came. Also since objective function calculates the error(equivalent term is loss-diff between actual and predicted values), it also has the names error function and loss function.