Weight decay is a regularization technique to penalize the weights for growing up. This is done through the addition of a penalty term in the loss function. This is clear. However, consider the code :

model = Sequential()
model.add(Dense(60, input_shape=(60,), activation='relu', kernel_regularizer=l2(0.01))
model.add(Dense(30, activation='relu', kernel_regularizer=l2(0.02)))
model.add(Dense(1, activation='sigmoid'))

I need to understand this code in terms of weight decay. The code above will add 2 penalty terms to the overall loss function to be optimized, one for the first hidden layer with a regularization factor of 0.01 and the other term with a regularization factor of 0.02, is that right? Moreover, the higher the regularization factor, the more your constraining the weights to grow, so this means that there will be more "pressure" on the weights of the second hidden layer to grow, is that also right?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.