I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM.
# average the loss loss /= num_train # average the gradients dW /= num_train # add L2 regularization to the loss loss += reg * np.sum(W * W) # ???? dW += 2 * reg * W
I don't understand why we would add regularization loss to the gradient. My understanding of regularization is we use it to prefer certain weights, $W$, over others. But... I don't understand
- What type of regularization is occurring to
dW(L2 regularization operates on the square of all values of the weights -- this is not squaring anything)
- Why we would tweak the weights themselves, presumably you want to tweak the loss which will incentivize changing the weights in a certain direction. Why would you tweak the weights (well, their gradients) themselves?