Fisher Scoring v/s Coordinate Descent for MLE in R

Question

R base function glm() uses Fishers Scoring for MLE, while the glmnet appears to use the coordinate descent method to solve the same equation. Coordinate descent is more time-efficient than Fisher Scoring, as Fisher Scoring calculates the second order derivative matrix, in addition to some other matrix operations. which makes expensive to perform, while coordinate descent can do the same task in O(np) time.

Why would R base function use Fisher Scoring? Does this method have an advantage over other optimization methods? How does coordinate descent and Fisher Scoring compare? I am relatively new to do this field so any help or resource will be helpful.

Answer 1

The only way to be sure is by benchmarking, but for glm Fisher scoring should be faster than coordinate descent. Fisher scoring is a special case of Newton Raphson, which has a faster rate of convergence than coordinate descent (Newton-Raphson is quadratically convergent, while coordinate descent is linearly convergent.) So while the computation of second-derivative information means each step takes more time, it can require many fewer steps than coordinate descent.

For the lasso, the special form of the penalty term makes it a very special case (and in fact absolute value isn't differentiable anyway, though sometimes you can finesse this). For this special problem, coordinate descent proves to be particularly fast. There are many other optimization problems where in practice Newton-Raphson is faster.