# Normalization factor in logistic regression cross entropy

Given that probability of a matrix of features $$X$$ along with weights $$w$$ is computed:

def probability(X, w):
z = np.dot(X,w)
a = 1./(1+np.exp(-z))
return np.array(a)

def loss(X, y, w):
normalization_fator = X.shape[0] #store loss values
features_probability = probability(X, w) #return one probability for each row in a matrix
cost = np.squeeze(cost)
return cost


Question: I did it first without dividing by $$normalization\_fator$$, but the correct way to do it is to divide by normalization factor although in the formula I had for logistic regression loss is given by:

$$L\left( \theta \right) =-\sum_{i=1}^n{y^{\left( i \right)}\log \left( \alpha _i \right) +\left( 1-y^{\left( i \right)} \right) \log \left( 1-\alpha _i \right)}$$

So as you can see there is no normalization facto:

$$L\left( \theta \right) =-\frac {1}{(norm\_factor)}\sum_{i=1}^n{y^{\left( i \right)}\log \left( \alpha _i \right) +\left( 1-y^{\left( i \right)} \right) \log \left( 1-\alpha _i \right)}$$

Edit: $$\alpha_i$$ represent probability of each row in $$X$$ given by sigmoid function.

• I don't see a question mark, and it is unclear to me what question(s) you have. However, I think you're trying to ask something like the question I answer here but with crossentropy loss rather than square loss.
– Dave
Oct 4, 2021 at 20:57
• @Dave. Similar to that question, I am asking please why we divide by $m$ in your answer before or by $normalization\_factor$ above, which is same as $m$? I did not get it when I tried to derive gradients though?
– Avv
Oct 4, 2021 at 21:07

In the linked answer, it is convenient to have the $$1/2$$ in the loss function so it cancels when we bring down the $$2$$ in the derivative, and this is okay since we just want to optimize the parameters. I do not see something that should cancel out in your equation, but there could be another reason to divide through.
• Remember that you take the derivatives of the parameters of the logistic regression (typically denoted with $\beta$ or $w$), not directly of $\alpha$. Perhaps that will help.