Normalization factor in logistic regression cross entropy

Question

Given that probability of a matrix of features $X$ along with weights $w$ is computed:

def probability(X, w):
    z = np.dot(X,w)
    a = 1./(1+np.exp(-z))
    return np.array(a)

def loss(X, y, w):
    normalization_fator = X.shape[0] #store loss values
    features_probability = probability(X, w) #return one probability for each row in a matrix
    corss_entropy = y*np.log(features_probability) + (1-y)*np.log(1-features_probability)
    cost = -1/float(normalization_fator) * np.sum(corss_entropy)
    cost = np.squeeze(cost)  
    return cost

Question: I did it first without dividing by $normalization\_fator$, but the correct way to do it is to divide by normalization factor although in the formula I had for logistic regression loss is given by:

$$ L\left( \theta \right) =-\sum_{i=1}^n{y^{\left( i \right)}\log \left( \alpha _i \right) +\left( 1-y^{\left( i \right)} \right) \log \left( 1-\alpha _i \right)} $$

So as you can see there is no normalization facto:

$$ L\left( \theta \right) =-\frac {1}{(norm\_factor)}\sum_{i=1}^n{y^{\left( i \right)}\log \left( \alpha _i \right) +\left( 1-y^{\left( i \right)} \right) \log \left( 1-\alpha _i \right)} $$

Edit: $\alpha_i$ represent probability of each row in $X$ given by sigmoid function.

Answer 1

In the linked answer, it is convenient to have the $1/2$ in the loss function so it cancels when we bring down the $2$ in the derivative, and this is okay since we just want to optimize the parameters. I do not see something that should cancel out in your equation, but there could be another reason to divide through.

In your case, unless you pick a silly normalization factor like zero, your two loss functions have the same parameters that optimize them, so it does not matter which we optimize. Dividing by some factor can keep the numbers from getting too large, though, especially if you're adding up over thousands or billions of predictions. Additionally, if your normalization factor is the sample size, you get some sense of the average crossentropy loss for an observation, the same as the MSE gives some sense of the average squared deviation when we do linear regression.