What is the efficiency difference between different cost functions in case of neural networks?

I'm studying the theory behind neural nets and I wonder if there is any actual difference between using different lost/cost functions?

Let's say I could use either MAE or MSE for backpropagating loss; does this decision have any actual effect on the model efficiency?

At the end of the day both functions just calculate the difference between $y$ and $\hat y$ (albeit on a different scale). But for the optimizer, only the error tendency is what really matters, not the absolute difference.

Of course, this question is relevant for any other model evaluation.

• This has nothing to do with neural networks per se. Try it and see what difference it makes! Note that you are implicitly assuming a regression task. And how are you defining "efficiency"?
– Emre
Aug 27, 2017 at 18:05

Think of it in terms of navigating a landscape. The land you move across is created by your error function (that is, the realtion between your model and your data), and the way you move across the landscape is your training function. Especially in Neural Networks, falling in 'local minima' is a big problem. In fact, Neural Networks have been theoretially able to approximate any function for... 20 years? See Universal Approximation Theorem. But the problem has always been learning how to train a given network.

More specifically, in your case comaring MAE and MSE, the difference is in the 'square' part. The steepness of the 'fitness landscape' will be much steeper where error is greater with MSE compared to MAE. 1*1 = 1. i.e., squaring does nothing at unit-error. 10 * 10 = 100. Absolute error of 10 will become a squared error of 100. That is looking at one sample, and has huge implications around falling in local minima where the error is great.

Looking across the whole dataset, another huge implication is how the NN balances the error it sees. With MSE, samples with high error become samples with very high error. Thus, a NN trained with MSE will 'care more' about learning samples that are very wrong at the expense of small improvements to 'easier' samples.