I'm studying the theory behind neural nets and I wonder if there is any actual difference between using different lost/cost functions? Let's say I could use either MAE or MSE for backpropagating loss; has this decision any actual effect on the model efficiency? At the end of the day both functions just calculate difference between y and y_hat (albeit on different scale), but for the optimizer only the error tendency is what really matters, not the absolute difference.

Of course, this questions is relevant for any other model evaluation.