I'm studying the theory behind [neural nets][NN] and I wonder if there is any actual difference between using different lost/cost functions?

Let's say I could use either [MAE][MAE] or [MSE][MSE] for backpropagating loss; does this decision have any actual effect on the model efficiency? 

At the end of the day both functions just calculate the difference between $y$ and $\hat y$ (albeit on a different scale). But for the optimizer, only the error tendency is what really matters, not the absolute difference.

Of course, this question is relevant for any other model evaluation.

  [NN]: https://en.wikipedia.org/wiki/Artificial_neural_network
  [MAE]: https://en.wikipedia.org/wiki/Mean_absolute_error
  [MSE]: https://en.wikipedia.org/wiki/Mean_squared_error