I'm training a neural network for predicting the location of a single object, so the prediction consists of 4 values: x, y, width and height. Without weight decay, the loss starts at around 0.3. If I add weight decay with a factor of 0.004, the loss jumps to around 22. Does this mean the network will give more importance to minimizing the weights over finding good coordinates? How important is the balance between the components of a loss function?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The cost function controls the algorithm completely - the new regularization from weight decay is likely responsible for the jump in loss. If you only added a small amount of regularization and the loss ballooned, you were likely very overfit.
The solution is cross validation. The only way to know the optimal amount if regularization is to see what does the best on held-out data.
