3
$\begingroup$

Calculating absolute values is much more efficient than calculating squares. Is there any advantage, then, to using the latter as a cost function over the former? Squares are easier to treat analytically, but in practice that doesn't matter.

$\endgroup$
2
$\begingroup$

In what term "calculating absolute values is much more efficient than calculating squares"? Compared to the complexity of any estimator/model used, I don't think it is significant - but I would be interested if anyone makes me wrong.

Again, why do you think it doesn't matter in practice? Working with a smooth and convex function is more convenient (in terms of time and results) than not-convex function.

Actually, you can choose whatever function to minimize you'd like; it is just a trade-off between :

  • Which kind of value you want to penalize
  • Complexity of the function to solve (mathematically speaking : local or global solution)
  • Time consuming (related to the previous point)

1. Minimizing absolute values :

With absolute value, you penalize the distance between y and f(x) linearly. Roughly speaking, you might end up with a lot of data that will look like outliers as long as enough are well explained by your estimator f.

Then, to minimize a function, one generally looks for the root(s) of its derivative. However, the derivative of |x| is not smooth. You can work with subgradient and other more complex mathematical object that may result in a longer time process due to more calculation.

2. Minimizing square values :

In this case, the distance between y and f(x) is more penalized. You'll tend to have less outliers (relatively to f(x)).

What is interesting is that is a smooth function (i.e. a defined derivative) and convex (with a global minimum)

So I guess people believe that the square of errors is a good trade-off.

$\endgroup$
2
$\begingroup$

Both the square and the absolute value should work okay for gradient descent but the square will work better. For calculus based methods, the absolute value method may be intractable.

If you are using gradient descent methods then the square works very nicely in that it tends to form a U shape which takes big steps away from the minimum and smaller steps close to the minimum. Conversely, absolute values tend to form V shapes which take roughly the same step size. The result is that squares have better convergence properties.

Now, calculus based methods tend to take the derivative of a function and set it equal to zero. Notice that U shapes tend to have well defined derivatives equal to zero at the minima, which works perfectly. By contrast, the V shape has an undefined derivative at its minima which tend to make conjugate gradient methods un-invertible.

Hope this helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.