# Questions tagged [optimization]

In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

496 questions
Filter by
Sorted by
Tagged with
27 views

### Setting derivative to 0 when minimizing expected loss over possible data samples

Suppose, given $x\in \mathbb{R}^d$, you want $\theta$, which is the solution of $$\text{argmin}_{\theta} \mathbb{E}_a[L(\theta;x,a)]$$ where $a \sim \mathcal{N}(x,\Sigma)$ and $L$ measures a (convex)...
1 vote
24 views

### Which machine learning models are rational to use on NP-hard and NP-complete "theoretical" problems?

Time and time again I run into "surprising" NP-hard problems that seem naturally simpler than they are. I recently worked on a weighted graph theoretical problem where the point is to ...
311 views

### What do we mean by optimizer.zero_grad()

This should be a simple question. But it is vague to me. What do we mean by optimizer.zero_grad(). Consider SGD as an example: $W^{t+1}= W^{t}- \lambda g_t$. Which one becomes zero for each batch. It ...
22 views

40 views

### Quadratic approximation, second-order optimization method, Newton method

I am learning Newton's method for second-order optimization in ML. I encountered this formula, but I do not understand how we get it. I guess it is from the Taylor series, but I still cannot fully ...
43 views

### Optimization on a convex function used in a loss function

I am currently creating a deep learning model which deals with classification and regression problem together such that each class has continuous value within an interval of real numbers in common ...
21 views

### Binary crossentropy loss

When we have a binary classification problem, we use a sigmoid activation function in the output layer+ a binary crossentropy loss. We also need to one hot encode the target variable.This s a binary ...