I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM.
Relevant code:
# average the loss
loss /= num_train
# average the gradients
dW /= num_train
# add L2 regularization to the loss
loss += reg * np.sum(W * W)
# ????
dW += 2 * reg * W
I don't understand why we would add regularization loss to the gradient. My understanding of regularization is we use it to prefer certain weights, $W$, over others. But... I don't understand
- What type of regularization is occurring to
dW
(L2 regularization operates on the square of all values of the weights -- this is not squaring anything) - Why we would tweak the weights themselves, presumably you want to tweak the loss which will incentivize changing the weights in a certain direction. Why would you tweak the weights (well, their gradients) themselves?