I'm reading the "Python Machine Learning" book by Sebastian Raschka, and we use different cost functions.
For Adaline model (with a linear activation function) we use the MSE error : (Phi is the activation function and y the target) $$ Err(w) = \frac{1}{2}\sum(\phi(z) - y)^2 $$ But for the Logistic Regression model we use another cost function : $$ Err(w) = \sum[-y\log(\phi(z)) - (1-y)\log(1-\phi(z))] $$

I just want to know what does the cost funciton really mean, it's a probability ? Or just something arbitrary and convex ?
Is there a way to find "the best" cost function for a specific activation function ?


2 Answers 2


The cost function is the judge for your model. It judges how well your model perfoms. By choosing a loss function you choose which properties of your model outputs the loss function will judge. Mathematical convenience usually is desired for the loss function to be applicable

The MSE will punish outputs that are further away from the desired value more severely than those that are closer because of the quadratic. Therefore, the outputs that are the furthest away from the desired value impact the cost function and thus the optimization greatly. If your dataset has many outliers, then these can influence your cost function a lot. In this case you could use techniques like dimensionality reduction or just choose another loss function like the $L_1$ loss, which is more robust to outliers. Furthermore, if you choose MSE you implicitly assume that the noise of your data is guassianly distributed, since the MSE loss will minimize the crossentropy between the empirical distribution of your outputs and the gaussian distribution. See https://stats.stackexchange.com/questions/288451/why-is-mean-squared-error-the-cross-entropy-between-the-empirical-distribution-a

The logistic regression cost function will judge the outputs of your model by other characteristics. The logistic regression intends to classify your outputs correctly into two classes. The output of your model lies between $[0, 1]$ the respective classes. Once outputs lie on the correct side of the decision boundary their impact on the lossfunction decrease rapidly with the distance to the decision boundary. Look at the logistic curve and observe where points are closest to the classes ($0$ or $1$) and then check out where the loss function is the steepest for each class. So the cost function of the logistic regression will punish those outputs severly that are missclassified, and those, although a bit less, that are correctly classified but lie very close to the decision boundary. As a result it is a decent loss function for binary classification. There are probably a few further implicit assumptions with logistic regression, however I have not dug into that in the past to be able to say what they are.

Since one usually assumes that the data is i.i.d (identically, idependently distributed: each datapoint is to be treated equally), the loss for each output is summed up.

In summary: Loss functions are chosen to judge the desired properties of the model outputs, and usually exhibit favorable mathematical qualities for the optimization. By choosing the loss function you can alter the criteria your optimization should follow and often times thus introduce prior knowledge.


To understand what is a cost function, you have to understand supervised machine learning.

In this setting, we are given n training samples $x_1,\ldots,x_n$, with $x_i \in \mathbb{R}^{f}$, for all training samples so that $f$ is the dimension of the feature space.

For each training sample $x_i$, we are given a label $y_i \in \mathbb{R}^{g}$.

Now the task of supervised machine learning is to find a function $\phi$, such that $\phi(x_i) = y_i$, for all training samples $x_i$.

To this end, a parametrized function is choosen (e.g. SVM, Neural Network, Random Forest,..), so that we consider a function $\phi_{w}$, where $w$ represents some paramters of the function.

Then, we seek for the parameters $w$, such that $\phi_w$ maps the input samples to the labels as best as possible. That means we are trying to calculate:

$\min_{w} \sum_{i=1}^{n} ||\phi_w(x_i)-y_{i}||,$

where $$|| \cdot ||$$ is a function measuring the difference between model prediction and desired outcome. It is also called cost function or loss function (I guess its called the loss as it measures the loss in accuracy when using the model's prediction instead of the true outcome).

In principle, any plausible loss function could be used, which delivers a function $\phi_w$ that has minimal loss according the chosen loss function (if the minimzation problem can be solved optimally).

Now depending on knowledge on your input data and the labels, you should select the loss function which is most approriate for the machine learning task.

A few examples,

  • If $y \in \mathbb{R}$, using the MSE loss can be considered as a least quares problem: $\min_{w} \sum_{i=1}^{n} (\phi_w(x_i)-y_{i})^2,$ as we assume to have an overdetermined system of equations ($n > f$). For least squares problems, the theory is very well-developed, e.g. it is a maximum-likelihood estimator in case of gaussian noise.

  • If your noise is not guassian distributed, you should use robust regression, e.g. using the Huber loss, which gives less weights to outliers.

  • If you want to regress a function in $[0,1]$, so $y_i \in [0,1]$ for all $i$, applying a least square fit does not make sense as it would result in a function that might have values in $\mathbb{R} \setminus [0,1]$, which might not (when certain assumptions are satisfied) make any sense if you want to regress probabilities for example. Therefore, a different loss function should be used. For the Logistic regression, the assumption is that y is Bernoulli distributed. In fact, the linear regression is fitted to the logit [$\mathrm{logit}(p) := log(\frac{p}{1-p})$], so we could compute $\min_{w} \sum_{i=1}^{n} (\mathrm{logit}(y_{i}) - \phi_{w}(x_i))^2$. This can be used to show that again, the maximum likelihood (under the assumption that all $y_i$ are independently Bernoulli distributed) is achieved, if the cross-entropy loss $\sum_{i=1}^{n} [−𝑦_{i}log(\phi_{w}(x_i))−(1−𝑦_{i})log(1−\phi_{w}(x_i))]$ is minimized.

  • The hinge loss $\sum_{i=1}^{n}\max(0,1-y_{i}\phi_{w}(x_{i}))$ is used to train a SVM in the case of a binary classification. Here the setting is $y_{i} \in \{-1,1\}$. For the SVM it can be shown that minimizing the loss is equivalent to maximizing the margin between the seperating hyperplane and the two classes.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.