I was trying to understand the Gradient Descent algorithm from this article and the author says

Most of the data science algorithms are optimization problems

I come from software engineering background trying to get a basic understanding of data science. Can someone help me explain what this means?

  • $\begingroup$ Strictly speaking, the article is wrong. An algorithm is not a problem. There are no "data science algorithms" imho of what data science is. There are algorithms that are used in data science and some solve optimization problems. "Most" is quite a statement, as it is not clear what the author means by "data science", which has no generally accepted definition. After years, I am still working on a useful one. There are machine learning algorithms, which the author might mean. Let's rephrase: "many machine learning algorithms used in data science solve optimization problems". $\endgroup$
    – Make42
    Jan 5 at 11:39

Most algorithms try to minimizes some objecive functions.

For example, in linear regresssion, given $(x_i, y_i)$, we try to find $\hat{y}_i= \alpha_0 + \sum_{j=1}^d \alpha_j x_{i,j}$ and we we want it to be close to $y$. We try to minimize the mean square error in our estimation.

That is our objective function is $$\min _\alpha\frac1n \sum_{i=1}^n (y_i-\alpha_o- \sum_{j=1}^d \alpha_j x_{i,j})^2.$$

We might have a model with unknown parameters and we can use maximum likelihood to find out our model, in that case, we maximize the likelihood function. Again, we get another optimization problem.

In general supervised classication error, we are trying to minimize the error which is typically formulated as a minimization of a loss function. In SVM, we are trying to find a boundary that maximizes the margin between the two classes, again, we are trying to maximize an objective function.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.