1
$\begingroup$

Given that probability of a matrix of features $X$ along with weights $w$ is computed:

def probability(X, w):
    z = np.dot(X,w)
    a = 1./(1+np.exp(-z))
    return np.array(a)

def loss(X, y, w):
    normalization_fator = X.shape[0] #store loss values
    features_probability = probability(X, w) #return one probability for each row in a matrix
    corss_entropy = y*np.log(features_probability) + (1-y)*np.log(1-features_probability)
    cost = -1/float(normalization_fator) * np.sum(corss_entropy)
    cost = np.squeeze(cost)  
    return cost

Question: I did it first without dividing by $normalization\_fator$, but the correct way to do it is to divide by normalization factor although in the formula I had for logistic regression loss is given by:

$$ L\left( \theta \right) =-\sum_{i=1}^n{y^{\left( i \right)}\log \left( \alpha _i \right) +\left( 1-y^{\left( i \right)} \right) \log \left( 1-\alpha _i \right)} $$

So as you can see there is no normalization facto:

$$ L\left( \theta \right) =-\frac {1}{(norm\_factor)}\sum_{i=1}^n{y^{\left( i \right)}\log \left( \alpha _i \right) +\left( 1-y^{\left( i \right)} \right) \log \left( 1-\alpha _i \right)} $$

Edit: $\alpha_i$ represent probability of each row in $X$ given by sigmoid function.

$\endgroup$
2
  • 1
    $\begingroup$ I don't see a question mark, and it is unclear to me what question(s) you have. However, I think you're trying to ask something like the question I answer here but with crossentropy loss rather than square loss. $\endgroup$
    – Dave
    Oct 4, 2021 at 20:57
  • $\begingroup$ @Dave. Similar to that question, I am asking please why we divide by $m$ in your answer before or by $normalization\_factor$ above, which is same as $m$? I did not get it when I tried to derive gradients though? $\endgroup$
    – Avv
    Oct 4, 2021 at 21:07

1 Answer 1

1
$\begingroup$

In the linked answer, it is convenient to have the $1/2$ in the loss function so it cancels when we bring down the $2$ in the derivative, and this is okay since we just want to optimize the parameters. I do not see something that should cancel out in your equation, but there could be another reason to divide through.

In your case, unless you pick a silly normalization factor like zero, your two loss functions have the same parameters that optimize them, so it does not matter which we optimize. Dividing by some factor can keep the numbers from getting too large, though, especially if you're adding up over thousands or billions of predictions. Additionally, if your normalization factor is the sample size, you get some sense of the average crossentropy loss for an observation, the same as the MSE gives some sense of the average squared deviation when we do linear regression.

$\endgroup$
2
  • $\begingroup$ Thank you. I don't know what I did wrong while deriving gradients for logistic regression it. I will double check as I did not get normalization factor $\endgroup$
    – Avv
    Oct 4, 2021 at 21:13
  • 1
    $\begingroup$ Remember that you take the derivatives of the parameters of the logistic regression (typically denoted with $\beta$ or $w$), not directly of $\alpha$. Perhaps that will help. $\endgroup$
    – Dave
    Oct 4, 2021 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.