# Relation between MLE (Maximum Likelihood Estimation) & Gradient Descent

What are the similarities & dissimilarities between MLE (used to find the best parameters in logistic regression) & Gradient Descent?

Gradient descent is a numerical method used by a computer to calculate the minimum of a loss function. If that loss function is related to the likelihood function (such as negative log likelihood in logistic regression or a neural network), then the gradient descent is finding a maximum likelihood estimator of a parameter (the regression coefficients).

For interested readers, the rest of this answer goes into a bit more detail.

In ordinary least squares linear regression with a model matrix $$X$$ and observed dependent variables $$y$$ (the usual notation), under certain conditions, the maximum likelihood estimator of the regression coefficients is given by:

$$\hat\beta_{MLE}=(X^TX)^{-1}X^Ty$$

This is derived by calculus, and we get a closed-form solution.

In other models, such logistic regression, we are not so lucky to have a closed-form solution. There is no equation to give those estimates. However, we define them implicitly in terms of the loss function $$L(\hat\beta)$$, which is related to maximum likelihood estimator of the regression coefficients.

$$\underset{\hat\beta}{argmin}\text{ }L(\hat\beta)$$

Gradient descent is one approach to finding this value.

MLE (Maximum Likelihood Estimation) sets up the optimization problem and gradient descent is a method for finding a specific solution to the optimization problem.

MLE defines the optimization problem as finding the values of the model parameters that maximize the likelihood function over the parameter space, selecting the parameter values that make the observed data most probable.

Gradient descent estimates the values of the parameters for a model by iteratively searching for the minimum of the loss function.

Logistic regression is a specific optimization problem of predicting the log-odds of an example belonging to class 1. Gradient descent finds the parameter values for the logistic regression.