2
$\begingroup$

I tried to add L2 regularization to a network class I wrote however when I train it the loss blows up even though accuracy also increases. Can someone explain where I am going wrong? (I am using the formulas from here)

The update to minibatch (The (1-eta*(lmbda/n)) coefficient to w is what I added)

def update_mini_batch(self, mini_batch, eta, lmbda, n):
    # n is the number of training samples being trained from
    # Turn the mini_batch with one dimensional samples into two matricies and then transpose them to get samples in columns
    matrix_x, matrix_y = [np.array([arr for arr in arr_list]).transpose() for arr_list in zip(*mini_batch)]
    
    gradient_b, gradient_w = self.backprop(matrix_x, matrix_y)
    
    self.bias = [b-(eta/len(mini_batch))*db for b, db in zip(self.bias, gradient_b)]
    self.weights = [(1-eta*(lmbda/n))*w-(eta/len(mini_batch))*dw for w, dw in zip(self.weights, gradient_w)]

The function that evaluates cost (I am using Quadratic Cost)

def evaluate(self, data, lmbda):
    matrix_x, matrix_y = [np.array([arr for arr in arr_list]).transpose() for arr_list in zip(*data)]
    output_matrix = self.feedforward(matrix_x)
    
    cost = self.cost_func.apply(output_matrix, matrix_y) 
    #L2 Regularization
    cost += lmbda/(2*(matrix_y.shape[1])) * sum(np.linalg.norm(w)**2 for w in self.weights)
    acc = np.sum(output_matrix.argmax(axis=0)==matrix_y.argmax(axis=0))
    
    return cost, acc

An example of my cost and accuracy during training

Epoch 0 done! Cost: 11.938649143175008. Accuracy 7397 / 10000
Epoch 1 done! Cost: 16.017232330762045. Accuracy 7381 / 10000
Epoch 2 done! Cost: 21.62351585060393. Accuracy 7431 / 10000
Epoch 3 done! Cost: 30.96422767377938. Accuracy 7498 / 10000
Epoch 4 done! Cost: 45.75409202821266. Accuracy 7669 / 10000
Epoch 5 done! Cost: 67.47752609972852. Accuracy 7691 / 10000
Epoch 6 done! Cost: 97.56030814767621. Accuracy 7574 / 10000
Epoch 7 done! Cost: 133.3273570333546. Accuracy 7503 / 10000
Epoch 8 done! Cost: 174.7085211732363. Accuracy 7341 / 10000

After running this for longer the cost still increases continually and no change in eta or lambda changes this fact. Once thing I noticed was that the individual MSE error was behaving normally and it was just the magnitude of the weights that was increasing.

Improve this question
$\endgroup$
6
  • $\begingroup$ If a full implementation is needed I can provide that here as well! $\endgroup$
    – BOSSrobot
    Jun 21, 2020 at 19:17
  • $\begingroup$ Does your "cost" ever decrease? Or does it keep increasing? $\endgroup$
    – Valentin Calomme
    Jun 22, 2020 at 12:52
  • $\begingroup$ Why don't you reduce the learning rate? $\endgroup$
    – noe
    Jun 22, 2020 at 15:46
  • $\begingroup$ I may be wrong - you are multiplying "eta" here - "eta*(lmbda/n))*w"......but not multiplied in total cost - .....np.linalg.norm(w)**2.....Either you should remove from the first equation or add into second i.e. np.linalg.norm(eta*w)**2 $\endgroup$
    – 10xAI
    Jun 23, 2020 at 2:47
  • 1
    $\begingroup$ @RoshanJha The reason I multiplied by eta is because even though the addition to the cost is lambda/(2n) * sum(weights squared). When you use the update rule w <-- w - eta * $\frac{\partial C}{\partial w}$ the eta gets multiplied with everything including the derivative of the regularization$. Once you simplify this is what comes out. Thanks for the idea though! $\endgroup$
    – BOSSrobot
    Jun 23, 2020 at 19:39

1 Answer 1

Reset to default
0
$\begingroup$

When you use regularization, your loss will be larger because you add the regularization term. So, it is normal if your best loss without regularization is lower than your best loss with regularization.

However, adding regularization should not affect convergence in the long-term. Meaning that even though your overall loss might be larger, it should decrease after some epochs.

I see that it increases for the first few epochs. I would suggest running your model for longer and see how the loss behaves. If an epoch takes long, use a small chunk of your dataset to investigate the behaviour of the loss. It may be possible that for the first few epochs, the loss doesn't start converging quite yet.

Improve this answer
$\endgroup$
3
  • $\begingroup$ @ValentinCalomme thanks for the suggestion! I reran my experiment until early stopping kicked in and the cost still continually increased. I think there is some error in the code somewhere or something. I am not sure. Also, I am still kind of new here so do you think I should edit my original post to include these new results? And should I include my full implementation? $\endgroup$
    – BOSSrobot
    Jun 23, 2020 at 19:45
  • $\begingroup$ I did a little more digging and I found out that (rather obviously but still helpful) the individual MSE error still decreases (hence the increase in accuracy) but the magnitude of the weights still grows largely. This doesn't make sense to me because the calculus is supposed to guarantee that the weights move in a direction that gives less loss... I don't know if this helps but here it is! $\endgroup$
    – BOSSrobot
    Jun 23, 2020 at 20:01
  • $\begingroup$ Please edit your answer with your new findings! This might help others help you $\endgroup$
    – Valentin Calomme
    Jun 23, 2020 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.