1
$\begingroup$

I am new to ML and data science and am struggling with a simple problem. In my problem, I am given a series of datapoints $X_i$ where $X_i = (x_{i1}, x_{i2})$ with each data point having a label $y_i$ where $y_i \in [-1, 1]$.

My first task that I must complete the following: Given a weight vector $w$, write a function to compute the logistic loss (also known as the negative log likelihood) for a given dataset.

I also am tasked with building a function that computes the gradient of the logistic loss evaluated at $w$, and both of these functions in tandem will be used to run gradient descent on the given dataset. For now, I am strictly concerned with developing the first function.

I understand log likelihood to be $\sum_{i=1}^n y_i \log p(x_i) + (1 − y_i) \log (1 − p(x_i))$ for a binary classifier, but I am unsure of how to write a function that computes the negative log likelihood. Specifically, how do we calculate $p(x_i)$ and how does a given weight vector $w$ factor into things?

$\endgroup$
1
  • $\begingroup$ It looks like $p$ is the regression equation, so $p(x_i)=\hat w_0+\hat w_1x_i$. Now how does your log likelihood look? // Negative log likelihood is exactly what it sounds like. Just take $-1$ times the log likelihood. $\endgroup$
    – Dave
    Nov 23 '21 at 0:30
0
$\begingroup$

Your dataset has two features $x_{i1}$ and $x_{i2}$ per data point, $i$. Hence your weight vector would look like [$w_0$,$w_1$,$w_2$] where $w_0$ is the bias and the next two elements are weights corresponding to two features respectively.

Therefore, the predicted output would be $p(x_i) = w_0 + (w_1*x_{i1}) + (w_2*x_{i2})$

The class labels are -1 and 1. Hence it is a binary classification problem. It would be nice to have labels 0 and 1 for using standard formula for log-likelihood. So consider changing -1's to 0's. Then apply the formula you suggested to calculate log-likelihood.

Else, if you do not prefer changing labels, and keep labels [-1,1] as such, modify the formula as below:

$log{-}likelihood = \sum_{i=1}^n max(y,0)*\log p(x_i) + min(1-y,1)*(1-\log p(x_i))$

The first term in the formula should become zero when label $y$ is -1 and the second term should become zero when the label $y$ is 1.

Once log-likelihood is obtained, just negate it ($-1 * log{-}likelihood$) to obtain the negative-log-likelihood.

Hope it helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.