I was trying to understand the Gradient Descent algorithm from this article and the author says
Most of the data science algorithms are optimization problems
I come from software engineering background trying to get a basic understanding of data science. Can someone help me explain what this means?
Most algorithms try to minimizes some objecive functions.
For example, in linear regresssion, given $(x_i, y_i)$, we try to find $\hat{y}_i= \alpha_0 + \sum_{j=1}^d \alpha_j x_{i,j}$ and we we want it to be close to $y$. We try to minimize the mean squared error in our estimation.
That is our objective function is $$\min _\alpha\frac1n \sum_{i=1}^n (y_i-\alpha_o- \sum_{j=1}^d \alpha_j x_{i,j})^2.$$
We might have a model with unknown parameters and we can use Maximum Likelihood Estimator to find out our model, in that case, we maximize the likelihood function. Again, we get another optimization problem.
In general supervised classification error we are trying to minimize the error, which is typically formulated as a minimization of a loss function. In SVM, we are trying to find a boundary that maximizes the margin between the two classes, again, we are trying to maximize an objective function.
